Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1505235 > unrolled thread

Oops in rapl_cpu_prepare()

Back to article view | Back to linux.kernel

Contents

  Oops in rapl_cpu_prepare() "Charles (Chas) Williams" <ciwillia@brocade.com> - 2016-10-20 22:30 +0200
    Re: [PREEMPT-RT] Oops in rapl_cpu_prepare() Sebastian Andrzej Siewior <bigeasy@linutronix.de> - 2016-10-21 13:00 +0200
      Re: [PREEMPT-RT] Oops in rapl_cpu_prepare() "Charles (Chas) Williams" <ciwillia@brocade.com> - 2016-10-21 23:10 +0200
        Re: [PREEMPT-RT] Oops in rapl_cpu_prepare() Sebastian Andrzej Siewior <bigeasy@linutronix.de> - 2016-10-25 14:30 +0200
          Re: [PREEMPT-RT] Oops in rapl_cpu_prepare() Sebastian Andrzej Siewior <bigeasy@linutronix.de> - 2016-10-25 14:50 +0200

#1505235 — Oops in rapl_cpu_prepare()

Recent 4.8 kernels have been oopsing when running under VMWare:

[    2.270203] BUG: unable to handle kernel NULL pointer dereference at 0000000000000408
[    2.270325] IP: [<ffffffff81012bb9>] rapl_cpu_online+0x59/0x70
[    2.270448] PGD 0
[    2.270570] Oops: 0002 [#1] SMP
[    2.270693] Modules linked in:
[    2.270815] CPU: 2 PID: 21 Comm: cpuhp/2 Not tainted 4.8.2-1-amd64-vyatta #1
[    2.270938] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
[    2.271060] task: ffff8802361fc2c0 task.stack: ffff880236208000
[    2.271183] RIP: 0010:[<ffffffff81012bb9>]  [<ffffffff81012bb9>] rapl_cpu_online+0x59/0x70
[    2.271306] RSP: 0000:ffff88023620be68  EFLAGS: 00010246
[    2.271428] RAX: 0000000000000004 RBX: ffff88023fd0d940 RCX: 0000000000000000
[    2.271551] RDX: 0000000000000040 RSI: 0000000000000004 RDI: 0000000000000004
[    2.271673] RBP: 0000000000000002 R08: fffffffffffffffc R09: 0000000000000000
[    2.271796] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000400
[    2.271918] R13: ffff8802361fc2c0 R14: ffff8802361fc2c0 R15: ffff8802361fc2c0
[    2.272041] FS:  0000000000000000(0000) GS:ffff88023fd00000(0000) knlGS:0000000000000000
[    2.272163] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    2.272286] CR2: 0000000000000408 CR3: 0000000001a06000 CR4: 00000000000406e0
[    2.272408] Stack:
[    2.272531]  ffff88023fd0d940 0000000000000002 ffffffff81a38240 ffffffff81061231
[    2.272654]  ffff8802361fc2c0 ffff880237002180 ffffffff8107ddcf 0000000000000000
[    2.272776]  ffff8802361a5a80 ffff880237002180 ffffffff8107dcb0 ffffffff81a6a380
[    2.272899] Call Trace:
[    2.273021]  [<ffffffff81061231>] ? cpuhp_thread_fun+0x31/0x100
[    2.273144]  [<ffffffff8107ddcf>] ? smpboot_thread_fn+0x11f/0x180
[    2.273266]  [<ffffffff8107dcb0>] ? sort_range+0x20/0x20
[    2.273389]  [<ffffffff8107b05a>] ? kthread+0xca/0xe0
[    2.273511]  [<ffffffff8157677f>] ? ret_from_fork+0x1f/0x40
[    2.273634]  [<ffffffff8107af90>] ? kthread_park+0x50/0x50
[    2.273757] Code: 00 00 48 83 c0 22 4c 8b 24 c1 48 c7 c0 30 a1 00 00 48 8b 14 10 e8 a8 61 26 00 3b 05 b6 56 ae 00 7c 0e f0 48 0f a
[    2.279445] RIP  [<ffffffff81012bb9>] rapl_cpu_online+0x59/0x70
[    2.279568]  RSP <ffff88023620be68>
[    2.279690] CR2: 0000000000000408
[    2.279813] ---[ end trace c95da920748eb432 ]---


gdb tells me:

(gdb) info line *(rapl_cpu_online+0x59)
Line 595 of "arch/x86/events/intel/rapl.c" starts at address 0xffffffff81012bb9 <rapl_cpu_online+89>
    and ends at 0xffffffff81012bbe <rapl_cpu_online+94>.

Which is:


         target = cpumask_any_and(&rapl_cpu_mask, topology_core_cpumask(cpu));
         if (target < nr_cpu_ids)
                 return 0;

         cpumask_set_cpu(cpu, &rapl_cpu_mask);
         pmu->cpu = cpu;		<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
         return 0;

This code was recently changed by commit 8b5b773d6245138c
"perf/x86/intel/rapl: Convert to hotplug state machine" and it
appears that the setup is done as a callback:

         /*
          * Install callbacks. Core will call them for each online cpu.
          */

         ret = cpuhp_setup_state(CPUHP_PERF_X86_RAPL_PREP, "PERF_X86_RAPL_PREP",
                                 rapl_cpu_prepare, NULL);
         if (ret)
                 goto out;

         ret = cpuhp_setup_state(CPUHP_AP_PERF_X86_RAPL_ONLINE,
                                 "AP_PERF_X86_RAPL_ONLINE",
                                 rapl_cpu_online, rapl_cpu_offline);

Is there a particular order guaranteed by the callbacks?  Will
rapl_cpu_prepare() always happen before online/offline?  Additionally,
rapl_cpu_prepare() can fail to allocate pmu,

	static int rapl_cpu_prepare(unsigned int cpu)
	{
		struct rapl_pmu *pmu = cpu_to_rapl_pmu(cpu);

		if (pmu)
			return 0;

		pmu = kzalloc_node(sizeof(*pmu), GFP_KERNEL, cpu_to_node(cpu));
		if (!pmu)
			return -ENOMEM;

But rapl_cpu_online() would have no idea about this.  What should be
done in this case?

[toc] | [next] | [standalone]

#1505775 — Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

On 2016-10-20 16:27:55 [-0400], Charles (Chas) Williams wrote:
> Recent 4.8 kernels have been oopsing when running under VMWare:

can you reproduce this on bare metal?

> [    2.270203] BUG: unable to handle kernel NULL pointer dereference at 0000000000000408
> [    2.270325] IP: [<ffffffff81012bb9>] rapl_cpu_online+0x59/0x70
…
> 
> gdb tells me:
> 
> (gdb) info line *(rapl_cpu_online+0x59)
> Line 595 of "arch/x86/events/intel/rapl.c" starts at address 0xffffffff81012bb9 <rapl_cpu_online+89>
>    and ends at 0xffffffff81012bbe <rapl_cpu_online+94>.
> 
> Which is:
> 
> 
>         target = cpumask_any_and(&rapl_cpu_mask, topology_core_cpumask(cpu));
>         if (target < nr_cpu_ids)
>                 return 0;
> 
>         cpumask_set_cpu(cpu, &rapl_cpu_mask);
>         pmu->cpu = cpu;		<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

can you check if pmu is NULL?

>         return 0;
…

> Is there a particular order guaranteed by the callbacks?  Will
> rapl_cpu_prepare() always happen before online/offline?  Additionally,

yes, see include/linux/cpuhotplug.h. On CPU-up the array ids are invoked
from CPUHP_OFFLINE till CPUHP_ONLINE.

> rapl_cpu_prepare() can fail to allocate pmu,

error codes callbacks are handled.

…
> But rapl_cpu_online() would have no idea about this.  What should be
> done in this case?

If a callback (such as CPUHP_PERF_X86_RAPL_PREP) fail then we rollback
to the starting point (in case of CPU up it would be CPUHP_OFFLINE.

Sebastian

[toc] | [prev] | [next] | [standalone]

#1506291 — Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

On 10/21/2016 06:56 AM, Sebastian Andrzej Siewior wrote:
> On 2016-10-20 16:27:55 [-0400], Charles (Chas) Williams wrote:
>> Recent 4.8 kernels have been oopsing when running under VMWare:
>
> can you reproduce this on bare metal?

I can't get dedicated access to the specific bare metal since it is
running as a dedicated hypervisor.  I haven't seen this issue anywhere
else though with the 4.8 kernel.

>> [    2.270203] BUG: unable to handle kernel NULL pointer dereference at 0000000000000408
>> [    2.270325] IP: [<ffffffff81012bb9>] rapl_cpu_online+0x59/0x70
>
> can you check if pmu is NULL?

It's not.  The dereference at 0x408 and pmu->cpu being fairly early in
the struct seems to indicate that pmu wasn't pointing to 0 at the time
(but fairly close).  I should have noticed that earlier.

>> Is there a particular order guaranteed by the callbacks?  Will
>> rapl_cpu_prepare() always happen before online/offline?  Additionally,
>
> yes, see include/linux/cpuhotplug.h. On CPU-up the array ids are invoked
> from CPUHP_OFFLINE till CPUHP_ONLINE.

Yes, I see that now.  Thanks for the pointer!

> If a callback (such as CPUHP_PERF_X86_RAPL_PREP) fail then we rollback
> to the starting point (in case of CPU up it would be CPUHP_OFFLINE.

You'll like this, I just did a little printk debugging because it was
easier than trying to get a debugger running:

	[    3.107126] init_rapl_pmus: maxpkg 4
	[    3.107263] rapl_cpu_prepare: pmu ffff880234faa540  cpu 0  pkgid 0
	[    3.107400] rapl_cpu_prepare: pmu ffff880234faa600  cpu 1  pkgid 2
	[    3.107537] rapl_cpu_prepare: pmu ffff880234faa6c0  cpu 2  pkgid 65535
	[    3.107662] rapl_cpu_online: pmu ffff880234faa540 cpu 0 pkgid 0
	[    3.107907] rapl_cpu_online: pmu ffff880234faa600 cpu 1 pkgid 2
	[    3.108133] rapl_cpu_online: pmu ffff880234faa6c0 cpu 2 pkgid 65535
	[    3.108333] rapl_cpu_online: pmu ffff880234faa6c0 cpu 3 pkgid 65535

where pkgid is topology_logical_package_id(cpu).

I can't understand why I don't see a cpu 3 during cpu prepare, when I
see one later.  The 65535 is a -1 from topology_phys_to_logical_pkg()
getting assigned to the logical_proc_id apparently.

So this is pretty puzzling.  Since this is a guest running under VMWare, I
don't know that there is any particular CPU pinning or emulation of RAPL.

It looks there was a proposal to not run in guests:

https://lkml.org/lkml/2015/12/3/559

[toc] | [prev] | [next] | [standalone]

#1508260 — Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

On 2016-10-21 17:03:56 [-0400], Charles (Chas) Williams wrote:
> I can't get dedicated access to the specific bare metal since it is
> running as a dedicated hypervisor.  I haven't seen this issue anywhere
> else though with the 4.8 kernel.

That is something :)

> > If a callback (such as CPUHP_PERF_X86_RAPL_PREP) fail then we rollback
> > to the starting point (in case of CPU up it would be CPUHP_OFFLINE.
> 
> You'll like this, I just did a little printk debugging because it was
> easier than trying to get a debugger running:
> 
> 	[    3.107126] init_rapl_pmus: maxpkg 4
there! vmware bug. It probably worked by chance.

> 	[    3.107263] rapl_cpu_prepare: pmu ffff880234faa540  cpu 0  pkgid 0
> 	[    3.107400] rapl_cpu_prepare: pmu ffff880234faa600  cpu 1  pkgid 2
> 	[    3.107537] rapl_cpu_prepare: pmu ffff880234faa6c0  cpu 2  pkgid 65535
> 	[    3.107662] rapl_cpu_online: pmu ffff880234faa540 cpu 0 pkgid 0
> 	[    3.107907] rapl_cpu_online: pmu ffff880234faa600 cpu 1 pkgid 2
> 	[    3.108133] rapl_cpu_online: pmu ffff880234faa6c0 cpu 2 pkgid 65535
> 	[    3.108333] rapl_cpu_online: pmu ffff880234faa6c0 cpu 3 pkgid 65535
> 
> where pkgid is topology_logical_package_id(cpu).
> 
> I can't understand why I don't see a cpu 3 during cpu prepare, when I
> see one later.  

because cpu 2 and 3 share the same package and if your printk is at the
bottom of the function, it will return early.

> The 65535 is a -1 from topology_phys_to_logical_pkg()
> getting assigned to the logical_proc_id apparently.

yes. The topology field is u16.

> So this is pretty puzzling.  Since this is a guest running under VMWare, I
> don't know that there is any particular CPU pinning or emulation of RAPL.

I assume "init_rapl_pmus: maxpkg 4" is from init_rapl_pmus() returning
topology_max_packages(). So it says 4 but then returns 65535 for CPU 2
and 3. That -1 comes probably from topology_update_package_map(). Could
you please send a complete boot log and try the following patch? This
one should fix your boot problem and disable RAPL if the info is
invalid.

diff --git a/arch/x86/events/intel/rapl.c b/arch/x86/events/intel/rapl.c
index 0a535cea8ff3..f5d85f2853d7 100644
--- a/arch/x86/events/intel/rapl.c
+++ b/arch/x86/events/intel/rapl.c
@@ -682,6 +682,15 @@ static int __init init_rapl_pmus(void)
 {
 	int maxpkg = topology_max_packages();
 	size_t size;
+	unsigned int cpu;
+
+	for_each_possible_cpu(cpu) {
+		if (topology_logical_package_id(cpu) >= maxpkg) {
+			pr_err("rapl pmu error: max package: %u but CPU%d belongs to %u\n",
+			       maxpkg, cpu, topology_logical_package_id(cpu));
+			return -EINVAL;
+		}
+	}
 
 	size = sizeof(*rapl_pmus) + maxpkg * sizeof(struct rapl_pmu *);
 	rapl_pmus = kzalloc(size, GFP_KERNEL);

Sebastian

[toc] | [prev] | [next] | [standalone]

#1508270 — Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

On 2016-10-25 14:22:05 [+0200], To Charles (Chas) Williams wrote:
> > 	[    3.107263] rapl_cpu_prepare: pmu ffff880234faa540  cpu 0  pkgid 0
> > 	[    3.107400] rapl_cpu_prepare: pmu ffff880234faa600  cpu 1  pkgid 2
> > 	[    3.107537] rapl_cpu_prepare: pmu ffff880234faa6c0  cpu 2  pkgid 65535
> > 	[    3.107662] rapl_cpu_online: pmu ffff880234faa540 cpu 0 pkgid 0
> > 	[    3.107907] rapl_cpu_online: pmu ffff880234faa600 cpu 1 pkgid 2
> > 	[    3.108133] rapl_cpu_online: pmu ffff880234faa6c0 cpu 2 pkgid 65535
> > 	[    3.108333] rapl_cpu_online: pmu ffff880234faa6c0 cpu 3 pkgid 65535

One thing I forgot to ask: Could you please check if you get the same
pkgid reported for cpu 0-3 on a pre-v4.8 kernel? (before the hotplug
rework).

Sebastian

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web