Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1168575 > unrolled thread

[-next] !irqd_can_balance() WARNINGs at irq_move_masked_irq()

Started bySergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
First post2015-06-19 09:20 +0200
Last post2015-06-21 16:50 +0200
Articles 8 — 4 participants

Back to article view | Back to linux.kernel

Contents

  [-next] !irqd_can_balance() WARNINGs at irq_move_masked_irq() Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2015-06-19 09:20 +0200
    Re: [-next] !irqd_can_balance() WARNINGs at irq_move_masked_irq() Thomas Gleixner <tglx@linutronix.de> - 2015-06-19 11:30 +0200
      Re: [-next] !irqd_can_balance() WARNINGs at irq_move_masked_irq() Thomas Gleixner <tglx@linutronix.de> - 2015-06-19 14:30 +0200
        Re: [-next] !irqd_can_balance() WARNINGs at irq_move_masked_irq() Sergey Senozhatsky <sergey.senozhatsky@gmail.com> - 2015-06-19 15:20 +0200
        Re: [-next] !irqd_can_balance() WARNINGs at irq_move_masked_irq() Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2015-06-20 06:40 +0200
          Re: [-next] !irqd_can_balance() WARNINGs at irq_move_masked_irq() Sergey Senozhatsky <sergey.senozhatsky@gmail.com> - 2015-06-20 20:00 +0200
          Re: [-next] !irqd_can_balance() WARNINGs at irq_move_masked_irq() Thomas Gleixner <tglx@linutronix.de> - 2015-06-21 16:40 +0200
            [tip:x86/apic] x86/hpet:   Use proper hpet device number for MSI allocation tip-bot for Thomas Gleixner <tipbot@zytor.com> - 2015-06-21 16:50 +0200

#1168575 — [-next] !irqd_can_balance() WARNINGs at irq_move_masked_irq()

FromSergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
Date2015-06-19 09:20 +0200
Subject[-next] !irqd_can_balance() WARNINGs at irq_move_masked_irq()
Message-ID<pCYKJ-oj-1@gated-at.bofh.it>
Hi,

next-20150619 is un-bootable (it `drowns` in warnings).

I modified irq_move_masked_irq() to do WARN_ONCE() instead of WARN_ON
every !irqd_can_balance(), and here it is:

[    0.412227] ------------[ cut here ]------------
[    0.412291] WARNING: CPU: 0 PID: 0 at kernel/irq/migration.c:21 irq_move_masked_irq+0x57/0xc4()
[    0.412371] Can't balance irq 0 [edge]
[    0.412428] Modules linked in:
[    0.412524] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.1.0-rc8-next-20150619-dbg-00010-g841f367-dirty #228
[    0.412606] Hardware name: SAMSUNG ELECTRONICS CO.,LTD Samsung DeskTop System/Samsung DeskTop System, BIOS 05CC                   04/09/2010
[    0.412693]  0000000000000009 ffff880137c03e18 ffffffff814f2acd ffffffff810851a4
[    0.412920]  ffff880137c03e68 ffff880137c03e58 ffffffff8103d196 0000000000000000
[    0.413146]  ffffffff8108bfe8 ffff880133007c00 0000000000000000 0000000000000000
[    0.413373] Call Trace:
[    0.413428]  <IRQ>  [<ffffffff814f2acd>] dump_stack+0x4c/0x6e
[    0.413532]  [<ffffffff810851a4>] ? console_unlock+0x400/0x42f
[    0.413596]  [<ffffffff8103d196>] warn_slowpath_common+0x9b/0xb5
[    0.413659]  [<ffffffff8108bfe8>] ? irq_move_masked_irq+0x57/0xc4
[    0.413722]  [<ffffffff8103d1f6>] warn_slowpath_fmt+0x46/0x48
[    0.413785]  [<ffffffff8108bfe8>] irq_move_masked_irq+0x57/0xc4
[    0.413847]  [<ffffffff8108c097>] irq_move_irq+0x42/0x56
[    0.413910]  [<ffffffff8102c410>] apic_ack_edge+0x23/0x3b
[    0.413972]  [<ffffffff81089b79>] irq_chip_ack_parent+0x14/0x16
[    0.414034]  [<ffffffff810894cd>] handle_edge_irq+0x90/0xe1
[    0.414096]  [<ffffffff81004aa9>] handle_irq+0x24/0x2a
[    0.414158]  [<ffffffff814fb2ec>] do_IRQ+0x4c/0xcd
[    0.414219]  [<ffffffff814f9c6a>] common_interrupt+0x6a/0x6a
[    0.414280]  <EOI>  [<ffffffff81403266>] ? cpuidle_enter_state+0x195/0x29d
[    0.414386]  [<ffffffff81403262>] ? cpuidle_enter_state+0x191/0x29d
[    0.414450]  [<ffffffff8140339a>] cpuidle_enter+0x17/0x19
[    0.414512]  [<ffffffff81075c30>] call_cpuidle+0x55/0x57
[    0.414573]  [<ffffffff81403381>] ? cpuidle_select+0x13/0x15
[    0.414636]  [<ffffffff81075eac>] cpu_startup_entry+0x27a/0x399
[    0.414700]  [<ffffffff814ef8bd>] rest_init+0x134/0x13a
[    0.414762]  [<ffffffff81b0df2c>] start_kernel+0x41a/0x427
[    0.414825]  [<ffffffff81b0d2fe>] x86_64_start_reservations+0x2a/0x2c
[    0.414889]  [<ffffffff81b0d46b>] x86_64_start_kernel+0x16b/0x17a
[    0.414987] ---[ end trace 082909e0a7e1599a ]---

Do you guys want to replace WAN_ON() with WARN_ONCE(), perhaps? This, of course,
doesn't fix anything; but at least one can boot the system. (not really a patch,
just an idea).

Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>

---

 kernel/irq/migration.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/kernel/irq/migration.c b/kernel/irq/migration.c
index dd203e2..0352b04 100644
--- a/kernel/irq/migration.c
+++ b/kernel/irq/migration.c
@@ -16,7 +16,9 @@ void irq_move_masked_irq(struct irq_data *idata)
 	 * Paranoia: cpu-local interrupts shouldn't be calling in here anyway.
 	 */
 	if (!irqd_can_balance(&desc->irq_data)) {
-		WARN_ON(1);
+		WARN_ONCE(1, "Can't balance irq %d [%s]\n",
+				desc->irq_data.irq,
+				desc->name);
 		return;
 	}
 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1168652

FromThomas Gleixner <tglx@linutronix.de>
Date2015-06-19 11:30 +0200
Message-ID<pD0Mx-3fQ-5@gated-at.bofh.it>
In reply to#1168575
On Fri, 19 Jun 2015, Sergey Senozhatsky wrote:
> [    0.412291] WARNING: CPU: 0 PID: 0 at kernel/irq/migration.c:21 irq_move_masked_irq+0x57/0xc4()
> [    0.412371] Can't balance irq 0 [edge]

Yuck.

> Do you guys want to replace WAN_ON() with WARN_ONCE(), perhaps? This, of course,
> doesn't fix anything; but at least one can boot the system. (not really a patch,
> just an idea).

Indeed. We really want to clear the move pending bit before the can
balance check. Patch below. But that does not explain why this happens
in the first place.

Can you please send me a full dmesg, kernel config and output of
/proc/interrupts ? (Private mail is fine, or upload it to some place)

Thanks,

	tglx
---
diff --git a/kernel/irq/migration.c b/kernel/irq/migration.c
index dd203e276b07..8cf8c24afaab 100644
--- a/kernel/irq/migration.c
+++ b/kernel/irq/migration.c
@@ -12,16 +12,19 @@ void irq_move_masked_irq(struct irq_data *idata)
 	if (likely(!irqd_is_setaffinity_pending(&desc->irq_data)))
 		return;
 
+	irqd_clr_move_pending(&desc->irq_data);
+
 	/*
 	 * Paranoia: cpu-local interrupts shouldn't be calling in here anyway.
 	 */
 	if (!irqd_can_balance(&desc->irq_data)) {
-		WARN_ON(1);
+		pr_warn("Cannot move irq %d: %s\n",
+			desc->irq_data.irq,
+			irqd_is_per_cpu(&desc->irq_data) ?
+			"per cpu" : "balancing disabled");
 		return;
 	}
 
-	irqd_clr_move_pending(&desc->irq_data);
-
 	if (unlikely(cpumask_empty(desc->pending_mask)))
 		return;
 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1168739

FromThomas Gleixner <tglx@linutronix.de>
Date2015-06-19 14:30 +0200
Message-ID<pD3AK-7hF-19@gated-at.bofh.it>
In reply to#1168652
On Fri, 19 Jun 2015, Thomas Gleixner wrote:
> On Fri, 19 Jun 2015, Sergey Senozhatsky wrote:
> > [    0.412291] WARNING: CPU: 0 PID: 0 at kernel/irq/migration.c:21 irq_move_masked_irq+0x57/0xc4()
> > [    0.412371] Can't balance irq 0 [edge]
> 
> Yuck.
> 
> > Do you guys want to replace WAN_ON() with WARN_ONCE(), perhaps? This, of course,
> > doesn't fix anything; but at least one can boot the system. (not really a patch,
> > just an idea).
> 
> Indeed. We really want to clear the move pending bit before the can
> balance check. Patch below. But that does not explain why this happens
> in the first place.
> 
> Can you please send me a full dmesg, kernel config and output of
> /proc/interrupts ? (Private mail is fine, or upload it to some place)

Thanks for providing the data. I think I know what happens.

Something in the kernel (not yet clear what) tries to move the hpet
irq 0 by calling irq_set_affinity(). That's an kernel internal
interface which does not check whether the NO BALANCE flag is set for
the irq. So the call runs and triggers the move from next interrupt
machinery which ends up calling irq_move_masked_irq() and that trips
over the flag and yells.

That's why I changed the WARN to a pr_warn() because we already know
the call stack.

So the core behaviour is inconsistent. We let the caller of
irq_set_affinity() succeed and yell later because we think it's wrong.

I'm pretty sure that we must drop the check for NO BALANCE in
irq_move_masked_irq() and only check for the per_cpu bit, but at the
same time I really want to know where that call to irq_set_affinity(irq0)
is coming from.

Can you please collect the output of /proc/timer_list for the previous
patch and then replace the previous patch with the one below and
gather all the data again?

Thanks,

	tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1168767

FromSergey Senozhatsky <sergey.senozhatsky@gmail.com>
Date2015-06-19 15:20 +0200
Message-ID<pD4n7-8sb-3@gated-at.bofh.it>
In reply to#1168739
On (06/19/15 14:21), Thomas Gleixner wrote:
> On Fri, 19 Jun 2015, Thomas Gleixner wrote:
> > On Fri, 19 Jun 2015, Sergey Senozhatsky wrote:
> > > [    0.412291] WARNING: CPU: 0 PID: 0 at kernel/irq/migration.c:21 irq_move_masked_irq+0x57/0xc4()
> > > [    0.412371] Can't balance irq 0 [edge]
> > 
> > Yuck.
> > 
> > > Do you guys want to replace WAN_ON() with WARN_ONCE(), perhaps? This, of course,
> > > doesn't fix anything; but at least one can boot the system. (not really a patch,
> > > just an idea).
> > 
> > Indeed. We really want to clear the move pending bit before the can
> > balance check. Patch below. But that does not explain why this happens
> > in the first place.
> > 
> > Can you please send me a full dmesg, kernel config and output of
> > /proc/interrupts ? (Private mail is fine, or upload it to some place)
> 
> Thanks for providing the data. I think I know what happens.
> 
> Something in the kernel (not yet clear what) tries to move the hpet
> irq 0 by calling irq_set_affinity(). That's an kernel internal
> interface which does not check whether the NO BALANCE flag is set for
> the irq. So the call runs and triggers the move from next interrupt
> machinery which ends up calling irq_move_masked_irq() and that trips
> over the flag and yells.
> 
> That's why I changed the WARN to a pr_warn() because we already know
> the call stack.
> 
> So the core behaviour is inconsistent. We let the caller of
> irq_set_affinity() succeed and yell later because we think it's wrong.
> 
> I'm pretty sure that we must drop the check for NO BALANCE in
> irq_move_masked_irq() and only check for the per_cpu bit, but at the
> same time I really want to know where that call to irq_set_affinity(irq0)
> is coming from.
> 
> Can you please collect the output of /proc/timer_list for the previous
> patch and then replace the previous patch with the one below and
> gather all the data again?
> 

It's 10pm here in Korea and I'm out of office already. I'll try
to collect the data tomorrow (or on Monday in the worst case).

Thank you.

	-ss
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1169286

FromSergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
Date2015-06-20 06:40 +0200
Message-ID<pDiJr-42v-3@gated-at.bofh.it>
In reply to#1168739

[Multipart message — attachments visible in raw view] — view raw

On (06/19/15 14:21), Thomas Gleixner wrote:
[..]
> Can you please collect the output of /proc/timer_list for the previous
> patch and then replace the previous patch with the one below and
> gather all the data again?
> 

attached.

	-ss

[toc] | [prev] | [next] | [standalone]

#1169418

FromSergey Senozhatsky <sergey.senozhatsky@gmail.com>
Date2015-06-20 20:00 +0200
Message-ID<pDvdD-5ou-3@gated-at.bofh.it>
In reply to#1169286
On (06/20/15 11:12), Thomas Gleixner wrote:
[..]
> > [    0.198654]  [<ffffffff8108795c>] irq_set_affinity_locked+0x5b/0xf9
> > [    0.198717]  [<ffffffff81087a46>] __irq_set_affinity+0x4c/0x67
> > [    0.198781]  [<ffffffff81b1d48c>] setup_ioapic_dest+0xa7/0xbf
> > [    0.198843]  [<ffffffff810851a4>] ? console_unlock+0x400/0x42f
> > [    0.198906]  [<ffffffff81b1b1ef>] native_smp_cpus_done+0xa4/0xae
> > [    0.198970]  [<ffffffff81b2408a>] smp_init+0x75/0x7a
> 
> OK. It's setup_ioapic_dest() and I understand now where this all comes
> from.
> 
> One last request. Can you please provide /proc/timer_list and dmesg
> from a mainline kernel for comparison?
> 

sorry, was offline. sure, will send out tomorrow.

	-ss
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1169597

FromThomas Gleixner <tglx@linutronix.de>
Date2015-06-21 16:40 +0200
Message-ID<pDOzE-7Qm-19@gated-at.bofh.it>
In reply to#1169286
On Sun, 21 Jun 2015, Sergey Senozhatsky wrote:
> On (06/20/15 11:12), Thomas Gleixner wrote:
> > One last request. Can you please provide /proc/timer_list and dmesg
> > from a mainline kernel for comparison?
> 
> attached.

Thanks for providing that. As I feared already, there is a regression
as well. The new code fails to allocated more than one hpet MSI
interrupt. Fix below.

Thanks,

	tglx
---
Subject: x86/hpet: Use proper hpet device number for MSI allocation
From: Thomas Gleixner <tglx@linutronix.de>
Date: Sun, 21 Jun 2015 16:21:50 +0200

hpet_assign_irq() is called with hpet_device->num as "hardware
interrupt number", but hpet_device->num is initialized after the
interrupt has been assigned, so it's always 0. As a consequence only
the first MSI allocation succeeds, the following ones fail because the
"hardware interrupt number" already exists.

Move the initialization of dev->num and other fields before the call
to hpet_assign_irq(), which is the ordering before the offending
commit which introduced that regression.

Fixes: "3cb96f0c9733 x86/hpet: Enhance HPET IRQ to support hierarchical irqdomains"
Reported-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: Borislav Petkov <bp@alien8.de>
---
 arch/x86/kernel/hpet.c |   11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

Index: tip/arch/x86/kernel/hpet.c
===================================================================
--- tip.orig/arch/x86/kernel/hpet.c
+++ tip/arch/x86/kernel/hpet.c
@@ -577,16 +577,17 @@ static void hpet_msi_capability_lookup(u
 		if (!(cfg & HPET_TN_FSB_CAP))
 			continue;
 
+		hdev->flags = 0;
+		if (cfg & HPET_TN_PERIODIC_CAP)
+			hdev->flags |= HPET_DEV_PERI_CAP;
+		sprintf(hdev->name, "hpet%d", i);
+		hdev->num = i;
+
 		irq = hpet_assign_irq(hpet_domain, hdev, hdev->num);
 		if (irq <= 0)
 			continue;
 
-		sprintf(hdev->name, "hpet%d", i);
-		hdev->num = i;
 		hdev->irq = irq;
-		hdev->flags = 0;
-		if (cfg & HPET_TN_PERIODIC_CAP)
-			hdev->flags |= HPET_DEV_PERI_CAP;
 		hdev->flags |= HPET_DEV_FSB_CAP;
 		hdev->flags |= HPET_DEV_VALID;
 		num_timers_used++;
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1169598 — [tip:x86/apic] x86/hpet: Use proper hpet device number for MSI allocation

Fromtip-bot for Thomas Gleixner <tipbot@zytor.com>
Date2015-06-21 16:50 +0200
Subject[tip:x86/apic] x86/hpet: Use proper hpet device number for MSI allocation
Message-ID<pDOJk-81G-3@gated-at.bofh.it>
In reply to#1169597
Commit-ID:  cb17b2a674f2059343f997599b4b001e64eec516
Gitweb:     http://git.kernel.org/tip/cb17b2a674f2059343f997599b4b001e64eec516
Author:     Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Sun, 21 Jun 2015 16:21:50 +0200
Committer:  Thomas Gleixner <tglx@linutronix.de>
CommitDate: Sun, 21 Jun 2015 16:38:40 +0200

x86/hpet: Use proper hpet device number for MSI allocation

hpet_assign_irq() is called with hpet_device->num as "hardware
interrupt number", but hpet_device->num is initialized after the
interrupt has been assigned, so it's always 0. As a consequence only
the first MSI allocation succeeds, the following ones fail because the
"hardware interrupt number" already exists.

Move the initialization of dev->num and other fields before the call
to hpet_assign_irq(), which is the ordering before the offending
commit which introduced that regression.

Fixes: "3cb96f0c9733 x86/hpet: Enhance HPET IRQ to support hierarchical irqdomains"
Reported-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/alpine.DEB.2.11.1506211635010.4107@nanos
Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: Borislav Petkov <bp@alien8.de>
---
 arch/x86/kernel/hpet.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/arch/x86/kernel/hpet.c b/arch/x86/kernel/hpet.c
index c47aab3..10757d0 100644
--- a/arch/x86/kernel/hpet.c
+++ b/arch/x86/kernel/hpet.c
@@ -577,16 +577,17 @@ static void hpet_msi_capability_lookup(unsigned int start_timer)
 		if (!(cfg & HPET_TN_FSB_CAP))
 			continue;
 
+		hdev->flags = 0;
+		if (cfg & HPET_TN_PERIODIC_CAP)
+			hdev->flags |= HPET_DEV_PERI_CAP;
+		sprintf(hdev->name, "hpet%d", i);
+		hdev->num = i;
+
 		irq = hpet_assign_irq(hpet_domain, hdev, hdev->num);
 		if (irq <= 0)
 			continue;
 
-		sprintf(hdev->name, "hpet%d", i);
-		hdev->num = i;
 		hdev->irq = irq;
-		hdev->flags = 0;
-		if (cfg & HPET_TN_PERIODIC_CAP)
-			hdev->flags |= HPET_DEV_PERI_CAP;
 		hdev->flags |= HPET_DEV_FSB_CAP;
 		hdev->flags |= HPET_DEV_VALID;
 		num_timers_used++;
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web