Re: system hung up when offlining CPUs

From	Thomas Gleixner <tglx@linutronix.de>
Newsgroups	linux.kernel
Subject	Re: system hung up when offlining CPUs
Date	2017-09-16 12:20 +0200
Message-ID	<uqimB-78-7@gated-at.bofh.it> (permalink)
References	(7 earlier) <uoXWW-4LY-11@gated-at.bofh.it> <updS1-6GT-13@gated-at.bofh.it> <upebn-6N5-1@gated-at.bofh.it> <upg3w-7ZJ-3@gated-at.bofh.it> <upFbA-7nG-21@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

On Thu, 14 Sep 2017, YASUAKI ISHIMATSU wrote:
> On 09/13/2017 09:33 AM, Thomas Gleixner wrote:
> >> Question - "what happens once __cpu_disable is called and some of the queued
> >> interrupt has affinity to that particular CPU ?"
> >> I assume ideally those pending/queued Interrupt should be migrated to
> >> remaining online CPUs. It should not be unhandled if we want to avoid such
> >> IO timeout.
> > 
> > Can you please provide the following information, before and after
> > offlining the last CPU in the affinity set:
> > 
> > # cat /proc/irq/$IRQNUM/smp_affinity_list
> > # cat /proc/irq/$IRQNUM/effective_affinity
> > # cat /sys/kernel/debug/irq/irqs/$IRQNUM
> > 
> > The last one requires: CONFIG_GENERIC_IRQ_DEBUGFS=y
> 
> Here are one irq's info of megasas:
> 
> - Before offline CPU
> /proc/irq/70/smp_affinity_list
> 24-29
> 
> /proc/irq/70/effective_affinity
> 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,3f000000
> 
> /sys/kernel/debug/irq/irqs/70
> handler:  handle_edge_irq
> status:   0x00004000
> istate:   0x00000000
> ddepth:   0
> wdepth:   0
> dstate:   0x00609200
>             IRQD_ACTIVATED
>             IRQD_IRQ_STARTED
>             IRQD_MOVE_PCNTXT
>             IRQD_AFFINITY_SET
>             IRQD_AFFINITY_MANAGED

So this uses managed affinity, which means that once the last CPU in the
affinity mask goes offline, the interrupt is shut down by the irq core
code, which is the case:

> dstate:   0x00a39000
>             IRQD_IRQ_DISABLED
>             IRQD_IRQ_MASKED
>             IRQD_MOVE_PCNTXT
>             IRQD_AFFINITY_SET
>             IRQD_AFFINITY_MANAGED
>             IRQD_MANAGED_SHUTDOWN  <---------------

So the irq core code works as expected, but something in the
driver/scsi/block stack seems to fiddle with that shut down queue.

I only can tell about the inner workings of the irq code, but I have no
clue about the rest.

Thanks,

	tglx

Thread

Re: system hung up when offlining CPUs YASUAKI ISHIMATSU <yasu.isimatu@gmail.com> - 2017-09-12 20:20 +0200
  Re: system hung up when offlining CPUs Hannes Reinecke <hare@suse.de> - 2017-09-13 13:20 +0200
    RE: system hung up when offlining CPUs Kashyap Desai <kashyap.desai@broadcom.com> - 2017-09-13 13:40 +0200
      RE: system hung up when offlining CPUs Thomas Gleixner <tglx@linutronix.de> - 2017-09-13 15:40 +0200
        Re: system hung up when offlining CPUs YASUAKI ISHIMATSU <yasu.isimatu@gmail.com> - 2017-09-14 18:30 +0200
          Re: system hung up when offlining CPUs Thomas Gleixner <tglx@linutronix.de> - 2017-09-16 12:20 +0200
            Re: system hung up when offlining CPUs Thomas Gleixner <tglx@linutronix.de> - 2017-09-16 17:10 +0200

csiph-web