Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1638562 > unrolled thread
| Started by | Michael Ellerman <mpe@ellerman.id.au> |
|---|---|
| First post | 2017-05-10 07:10 +0200 |
| Last post | 2017-05-11 07:50 +0200 |
| Articles | 5 — 3 participants |
Back to article view | Back to linux.kernel
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
Re: [patch V2 24/24] cpu/hotplug: Convert hotplug locking to percpu rwsem Michael Ellerman <mpe@ellerman.id.au> - 2017-05-10 07:10 +0200
Re: [patch V2 24/24] cpu/hotplug: Convert hotplug locking to percpu rwsem Thomas Gleixner <tglx@linutronix.de> - 2017-05-10 10:50 +0200
Re: [patch V2 24/24] cpu/hotplug: Convert hotplug locking to percpu rwsem Steven Rostedt <rostedt@goodmis.org> - 2017-05-10 18:40 +0200
Re: [patch V2 24/24] cpu/hotplug: Convert hotplug locking to percpu rwsem Steven Rostedt <rostedt@goodmis.org> - 2017-05-10 19:20 +0200
Re: [patch V2 24/24] cpu/hotplug: Convert hotplug locking to percpu rwsem Michael Ellerman <mpe@ellerman.id.au> - 2017-05-11 07:50 +0200
| From | Michael Ellerman <mpe@ellerman.id.au> |
|---|---|
| Date | 2017-05-10 07:10 +0200 |
| Subject | Re: [patch V2 24/24] cpu/hotplug: Convert hotplug locking to percpu rwsem |
| Message-ID | <tFs2R-4Nw-3@gated-at.bofh.it> |
Thomas Gleixner <tglx@linutronix.de> writes:
> @@ -130,6 +130,7 @@ void __static_key_slow_inc(struct static
> * the all CPUs, for that to be serialized against CPU hot-plug
> * we need to avoid CPUs coming online.
> */
> + lockdep_assert_hotplug_held();
> jump_label_lock();
> if (atomic_read(&key->enabled) == 0) {
> atomic_set(&key->enabled, -1);
I seem to be hitting this assert from the ftrace event selftests,
enabled at boot with CONFIG_FTRACE_STARTUP_TEST=y, using next-20170509
(on powerpc).
[ 842.691191] Testing event rpc_call_status:
[ 842.691209] ------------[ cut here ]------------
[ 842.691399] WARNING: CPU: 6 PID: 1 at ../kernel/cpu.c:234 lockdep_assert_hotplug_held+0x5c/0x70
[ 842.691575] Modules linked in:
[ 842.691675] CPU: 6 PID: 1 Comm: swapper/0 Tainted: G W 4.11.0-gcc-5.4.1-next-20170509 #218
[ 842.691865] task: c0000001fe780000 task.stack: c0000001fe800000
[ 842.692003] NIP: c0000000000ff3dc LR: c0000000000ff3d0 CTR: c000000000218650
[ 842.692166] REGS: c0000001fe8036e0 TRAP: 0700 Tainted: G W (4.11.0-gcc-5.4.1-next-20170509)
[ 842.692343] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE>
[ 842.692491] CR: 28002222 XER: 20000000
[ 842.692689] CFAR: c000000000171530 SOFTE: 1
GPR00: c0000000000ff3d0 c0000001fe803960 c0000000012b7600 0000000000000000
GPR04: ffffffffffffffff 0000000000000000 c0000000fc10c0e8 0000000000000000
GPR08: 0000000000000000 0000000000000000 0000000000000000 c0000000f8180008
GPR12: 0000000000002200 c00000000fd42100 c00000000000e218 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR20: 0000000000000000 0000000000000000 0000000000000000 c0000000f9341610
GPR24: c00000000127ee48 c000000000aa49d0 000000000000000a c0000000fc3c0000
GPR28: c00000000117b148 c000000001264230 0000000000000000 c00000000127ee48
[ 842.694287] NIP [c0000000000ff3dc] lockdep_assert_hotplug_held+0x5c/0x70
[ 842.694434] LR [c0000000000ff3d0] lockdep_assert_hotplug_held+0x50/0x70
[ 842.694577] Call Trace:
[ 842.694658] [c0000001fe803960] [c0000000000ff3d0] lockdep_assert_hotplug_held+0x50/0x70 (unreliable)
[ 842.694876] [c0000001fe803980] [c0000000002a3754] __static_key_slow_inc+0x104/0x170
[ 842.695054] [c0000001fe8039f0] [c0000000002176ac] tracepoint_probe_register_prio+0x2dc/0x390
[ 842.695258] [c0000001fe803a60] [c00000000024cf50] trace_event_reg+0xe0/0x130
[ 842.695434] [c0000001fe803a80] [c00000000024d5f0] __ftrace_event_enable_disable+0x270/0x3e0
[ 842.695601] [c0000001fe803b10] [c000000000e20328] event_trace_self_tests+0x14c/0x350
[ 842.695778] [c0000001fe803bc0] [c000000000e20774] event_trace_self_tests_init+0xc8/0xf4
[ 842.695944] [c0000001fe803c30] [c00000000000d87c] do_one_initcall+0x6c/0x1d0
[ 842.696113] [c0000001fe803cf0] [c000000000df462c] kernel_init_freeable+0x304/0x3e4
[ 842.696282] [c0000001fe803dc0] [c00000000000e23c] kernel_init+0x2c/0x170
[ 842.696460] [c0000001fe803e30] [c00000000000bdec] ret_from_kernel_thread+0x5c/0x70
[ 842.696662] Instruction dump:
[ 842.696763] 409e0014 38210020 e8010010 7c0803a6 4e800020 3c62ffe6 3880ffff 38634808
[ 842.697009] 480720ed 60000000 2fa30000 409effd8 <0fe00000> 38210020 e8010010 7c0803a6
[ 842.697271] ---[ end trace f68728a0d30544a1 ]---
The stupidly obvious (or perhaps obviously stupid) patch below fixes it:
diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c
index daefdee9411a..5531f7ce8fa6 100644
--- a/kernel/trace/trace_events.c
+++ b/kernel/trace/trace_events.c
@@ -3241,9 +3241,19 @@ static __init void event_trace_self_tests(void)
continue;
}
+ get_online_cpus();
+ mutex_lock(&event_mutex);
ftrace_event_enable_disable(file, 1);
+ mutex_unlock(&event_mutex);
+ put_online_cpus();
+
event_test_stuff();
+
+ get_online_cpus();
+ mutex_lock(&event_mutex);
ftrace_event_enable_disable(file, 0);
+ mutex_unlock(&event_mutex);
+ put_online_cpus();
pr_cont("OK\n");
}
cheers
[toc] | [next] | [standalone]
| From | Thomas Gleixner <tglx@linutronix.de> |
|---|---|
| Date | 2017-05-10 10:50 +0200 |
| Subject | Re: [patch V2 24/24] cpu/hotplug: Convert hotplug locking to percpu rwsem |
| Message-ID | <tFvtL-6TH-1@gated-at.bofh.it> |
| In reply to | #1638562 |
On Wed, 10 May 2017, Michael Ellerman wrote:
> Thomas Gleixner <tglx@linutronix.de> writes:
>
> > @@ -130,6 +130,7 @@ void __static_key_slow_inc(struct static
> > * the all CPUs, for that to be serialized against CPU hot-plug
> > * we need to avoid CPUs coming online.
> > */
> > + lockdep_assert_hotplug_held();
> > jump_label_lock();
> > if (atomic_read(&key->enabled) == 0) {
> > atomic_set(&key->enabled, -1);
>
> I seem to be hitting this assert from the ftrace event selftests,
> enabled at boot with CONFIG_FTRACE_STARTUP_TEST=y, using next-20170509
> (on powerpc).
>
> The stupidly obvious (or perhaps obviously stupid) patch below fixes it:
Kinda. There is more horror in that area lurking and I'm still trying to
figure out all the convoluted call pathes.
Thanks,
tglx
> diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c
> index daefdee9411a..5531f7ce8fa6 100644
> --- a/kernel/trace/trace_events.c
> +++ b/kernel/trace/trace_events.c
> @@ -3241,9 +3241,19 @@ static __init void event_trace_self_tests(void)
> continue;
> }
>
> + get_online_cpus();
> + mutex_lock(&event_mutex);
> ftrace_event_enable_disable(file, 1);
> + mutex_unlock(&event_mutex);
> + put_online_cpus();
> +
> event_test_stuff();
> +
> + get_online_cpus();
> + mutex_lock(&event_mutex);
> ftrace_event_enable_disable(file, 0);
> + mutex_unlock(&event_mutex);
> + put_online_cpus();
>
> pr_cont("OK\n");
> }
>
> cheers
>
[toc] | [prev] | [next] | [standalone]
| From | Steven Rostedt <rostedt@goodmis.org> |
|---|---|
| Date | 2017-05-10 18:40 +0200 |
| Subject | Re: [patch V2 24/24] cpu/hotplug: Convert hotplug locking to percpu rwsem |
| Message-ID | <tFCOC-31A-15@gated-at.bofh.it> |
| In reply to | #1638657 |
On Wed, 10 May 2017 10:49:09 +0200 (CEST)
Thomas Gleixner <tglx@linutronix.de> wrote:
> On Wed, 10 May 2017, Michael Ellerman wrote:
>
> > Thomas Gleixner <tglx@linutronix.de> writes:
> >
> > > @@ -130,6 +130,7 @@ void __static_key_slow_inc(struct static
> > > * the all CPUs, for that to be serialized against CPU hot-plug
> > > * we need to avoid CPUs coming online.
> > > */
> > > + lockdep_assert_hotplug_held();
> > > jump_label_lock();
> > > if (atomic_read(&key->enabled) == 0) {
> > > atomic_set(&key->enabled, -1);
> >
> > I seem to be hitting this assert from the ftrace event selftests,
> > enabled at boot with CONFIG_FTRACE_STARTUP_TEST=y, using next-20170509
> > (on powerpc).
> >
> > The stupidly obvious (or perhaps obviously stupid) patch below fixes it:
>
> Kinda. There is more horror in that area lurking and I'm still trying to
> figure out all the convoluted call pathes.
I finally got some time to look at this. I'm looking at your commit:
commit b53e5129c4c7ab47ec4f709fd8f5784ca45fb46d
Author: Thomas Gleixner <tglx@linutronix.de>
Date: Sun Apr 23 12:17:13 2017 +0200
trace/perf: Cure hotplug lock ordering issues
What were the circular locking dependencies that were uncovered.
event_mutex could possibly be broken up, if that helps. It sorta became
a catch all for various modifications to tracing.
-- Steve
>
> Thanks,
>
> tglx
>
> > diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c
> > index daefdee9411a..5531f7ce8fa6 100644
> > --- a/kernel/trace/trace_events.c
> > +++ b/kernel/trace/trace_events.c
> > @@ -3241,9 +3241,19 @@ static __init void event_trace_self_tests(void)
> > continue;
> > }
> >
> > + get_online_cpus();
> > + mutex_lock(&event_mutex);
> > ftrace_event_enable_disable(file, 1);
> > + mutex_unlock(&event_mutex);
> > + put_online_cpus();
> > +
> > event_test_stuff();
> > +
> > + get_online_cpus();
> > + mutex_lock(&event_mutex);
> > ftrace_event_enable_disable(file, 0);
> > + mutex_unlock(&event_mutex);
> > + put_online_cpus();
> >
> > pr_cont("OK\n");
> > }
> >
> > cheers
> >
[toc] | [prev] | [next] | [standalone]
| From | Steven Rostedt <rostedt@goodmis.org> |
|---|---|
| Date | 2017-05-10 19:20 +0200 |
| Subject | Re: [patch V2 24/24] cpu/hotplug: Convert hotplug locking to percpu rwsem |
| Message-ID | <tFDrk-3t7-7@gated-at.bofh.it> |
| In reply to | #1638961 |
On Wed, 10 May 2017 12:30:57 -0400 Steven Rostedt <rostedt@goodmis.org> wrote: > I finally got some time to look at this. I'm looking at your commit: > > commit b53e5129c4c7ab47ec4f709fd8f5784ca45fb46d > Author: Thomas Gleixner <tglx@linutronix.de> > Date: Sun Apr 23 12:17:13 2017 +0200 > > trace/perf: Cure hotplug lock ordering issues > > What were the circular locking dependencies that were uncovered. > > event_mutex could possibly be broken up, if that helps. It sorta became > a catch all for various modifications to tracing. > I checked out the commit before this and started playing with it. I see some of the issues now. I'll look more into it. -- Steve
[toc] | [prev] | [next] | [standalone]
| From | Michael Ellerman <mpe@ellerman.id.au> |
|---|---|
| Date | 2017-05-11 07:50 +0200 |
| Message-ID | <tFP97-2ex-1@gated-at.bofh.it> |
| In reply to | #1638657 |
Thomas Gleixner <tglx@linutronix.de> writes:
> On Wed, 10 May 2017, Michael Ellerman wrote:
>
>> Thomas Gleixner <tglx@linutronix.de> writes:
>>
>> > @@ -130,6 +130,7 @@ void __static_key_slow_inc(struct static
>> > * the all CPUs, for that to be serialized against CPU hot-plug
>> > * we need to avoid CPUs coming online.
>> > */
>> > + lockdep_assert_hotplug_held();
>> > jump_label_lock();
>> > if (atomic_read(&key->enabled) == 0) {
>> > atomic_set(&key->enabled, -1);
>>
>> I seem to be hitting this assert from the ftrace event selftests,
>> enabled at boot with CONFIG_FTRACE_STARTUP_TEST=y, using next-20170509
>> (on powerpc).
>>
>> The stupidly obvious (or perhaps obviously stupid) patch below fixes it:
>
> Kinda. There is more horror in that area lurking and I'm still trying to
> figure out all the convoluted call pathes.
OK thanks.
cheers
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web