Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #63824 > unrolled thread

Handling irqbalance in virtual environments

Started byBastian Blank <waldi@debian.org>
First post2019-04-12 11:10 +0200
Last post2019-04-23 21:50 +0200
Articles 7 — 4 participants

Back to article view | Back to linux.debian.kernel

Contents

  Handling irqbalance in virtual environments Bastian Blank <waldi@debian.org> - 2019-04-12 11:10 +0200
    Re: Handling irqbalance in virtual environments Bastian Blank <waldi@debian.org> - 2019-04-12 11:40 +0200
    Re: Handling irqbalance in virtual environments Ben Hutchings <ben@decadent.org.uk> - 2019-04-12 22:20 +0200
      Bug#926967: Don't recommend irqbalance (was: Re: Handling irqbalance in virtual environments) Bastian Blank <waldi@debian.org> - 2019-04-12 23:10 +0200
        Bug#926967: Don't recommend irqbalance (was: Re: Handling irqbalance in virtual environments) Nicholas D Steeves <nsteeves@gmail.com> - 2019-04-15 01:00 +0200
        Bug#926967: Don't recommend irqbalance (was: Re: Handling irqbalance in virtual environments) Bastian Blank <waldi@debian.org> - 2019-04-23 21:50 +0200
        Processed: Re: Bug#926967: Don't recommend irqbalance (was: Re:  Handling irqbalance in virtual environments) "Debian Bug Tracking System" <owner@bugs.debian.org> - 2019-04-23 21:50 +0200

#63824 — Handling irqbalance in virtual environments

FromBastian Blank <waldi@debian.org>
Date2019-04-12 11:10 +0200
SubjectHandling irqbalance in virtual environments
Message-ID<xM0m6-2fa-5@gated-at.bofh.it>
Moin

It turns out we got again problems with irqbalance.

It was added as recommends of the main image in 3.16, as it was reported
that older kernels move all interrupts to CPU 0 without help.[1]

In the meantime the kernel can do balancing on it's own.  In 4.9, I've
seen it working with aacraid, each queue gets hard pinned to it's own
CPU from 0 to $NRCPUS.  In 4.19 I've seen the same working properly with
virtio-net.

With 4.19, even on real hardware, where interrupts have an affinity for
all cpus, each interrupt is actually delivered to different cpu.

Random example for this, it even selects only one thread of each core:

|  26:    0    0    0    0   92    0    0    0  IR-PCI-MSI 3670017-edge      eno1-TxRx-0
|  27:    0    0    0    0    0  167    0    0  IR-PCI-MSI 3670018-edge      eno1-TxRx-1
|  28:    0    0    0    0    0    0  467    0  IR-PCI-MSI 3670019-edge      eno1-TxRx-2
|  29:    0    0    0    0    0    0    0  454  IR-PCI-MSI 3670020-edge      eno1-TxRx-3

Now irqbalance comes to re-do the existing pinning, and the result is not
longer correct but $RANDOM for the hard queue-to-cpu case of virtio.

At least Google considers the work irqbalance does to "correct" the existing
balancing a large problem.

I'm not sure how to go forward.  I have a workaround pending for our
cloud images to hard exclude the installation of irqbalance.[2]

Regards,
Bastian

[1]: https://bugs.debian.org/577788
[2]: https://salsa.debian.org/cloud-team/debian-cloud-images/merge_requests/81
-- 
Youth doesn't excuse everything.
		-- Dr. Janice Lester (in Kirk's body), "Turnabout Intruder",
		   stardate 5928.5.

[toc] | [next] | [standalone]

#63825

FromBastian Blank <waldi@debian.org>
Date2019-04-12 11:40 +0200
Message-ID<xM0P8-2pe-21@gated-at.bofh.it>
In reply to#63824
On Fri, Apr 12, 2019 at 10:53:47AM +0200, Bastian Blank wrote:
> With 4.19, even on real hardware, where interrupts have an affinity for
> all cpus, each interrupt is actually delivered to different cpu.

It seems a lot of this comes from
https://lore.kernel.org/patchwork/cover/801590/

Regards,
Bastian

-- 
Violence in reality is quite different from theory.
		-- Spock, "The Cloud Minders", stardate 5818.4

[toc] | [prev] | [next] | [standalone]

#63833

FromBen Hutchings <ben@decadent.org.uk>
Date2019-04-12 22:20 +0200
Message-ID<xMaOt-ce-5@gated-at.bofh.it>
In reply to#63824

[Multipart message — attachments visible in raw view] — view raw

On Fri, 2019-04-12 at 10:53 +0200, Bastian Blank wrote:
> Moin
> 
> It turns out we got again problems with irqbalance.
> 
> It was added as recommends of the main image in 3.16, as it was reported
> that older kernels move all interrupts to CPU 0 without help.[1]
> 
> In the meantime the kernel can do balancing on it's own.  In 4.9, I've
> seen it working with aacraid, each queue gets hard pinned to it's own
> CPU from 0 to $NRCPUS.  In 4.19 I've seen the same working properly with
> virtio-net.
> 
> With 4.19, even on real hardware, where interrupts have an affinity for
> all cpus, each interrupt is actually delivered to different cpu.
> 
> Random example for this, it even selects only one thread of each core:
> 
> >  26:    0    0    0    0   92    0    0    0  IR-PCI-MSI 3670017-edge      eno1-TxRx-0
> >  27:    0    0    0    0    0  167    0    0  IR-PCI-MSI 3670018-edge      eno1-TxRx-1
> >  28:    0    0    0    0    0    0  467    0  IR-PCI-MSI 3670019-edge      eno1-TxRx-2
> >  29:    0    0    0    0    0    0    0  454  IR-PCI-MSI 3670020-edge      eno1-TxRx-3
> 
> Now irqbalance comes to re-do the existing pinning, and the result is not
> longer correct but $RANDOM for the hard queue-to-cpu case of virtio.

Then let's drop the recommendation.

Ben.

> At least Google considers the work irqbalance does to "correct" the existing
> balancing a large problem.
> 
> I'm not sure how to go forward.  I have a workaround pending for our
> cloud images to hard exclude the installation of irqbalance.[2]
> 
> Regards,
> Bastian
> 
> [1]: https://bugs.debian.org/577788
> [2]: https://salsa.debian.org/cloud-team/debian-cloud-images/merge_requests/81
-- 
Ben Hutchings
Hoare's Law of Large Problems:
   Inside every large problem is a small problem struggling to get out.

[toc] | [prev] | [next] | [standalone]

#63838 — Bug#926967: Don't recommend irqbalance (was: Re: Handling irqbalance in virtual environments)

FromBastian Blank <waldi@debian.org>
Date2019-04-12 23:10 +0200
SubjectBug#926967: Don't recommend irqbalance (was: Re: Handling irqbalance in virtual environments)
Message-ID<xMbAS-KD-11@gated-at.bofh.it>
In reply to#63833
Package: src:linux
Version: 4.19.28-2
Severity: important

On Fri, Apr 12, 2019 at 09:10:32PM +0100, Ben Hutchings wrote:
> On Fri, 2019-04-12 at 10:53 +0200, Bastian Blank wrote:
> > It turns out we got again problems with irqbalance.
> > 
> > It was added as recommends of the main image in 3.16, as it was reported
> > that older kernels move all interrupts to CPU 0 without help.[1]
> > 
> > In the meantime the kernel can do balancing on it's own.  In 4.9, I've
> > seen it working with aacraid, each queue gets hard pinned to it's own
> > CPU from 0 to $NRCPUS.  In 4.19 I've seen the same working properly with
> > virtio-net.
> > 
> > With 4.19, even on real hardware, where interrupts have an affinity for
> > all cpus, each interrupt is actually delivered to different cpu.
> > 
> > Random example for this, it even selects only one thread of each core:
> > 
> > >  26:    0    0    0    0   92    0    0    0  IR-PCI-MSI 3670017-edge      eno1-TxRx-0
> > >  27:    0    0    0    0    0  167    0    0  IR-PCI-MSI 3670018-edge      eno1-TxRx-1
> > >  28:    0    0    0    0    0    0  467    0  IR-PCI-MSI 3670019-edge      eno1-TxRx-2
> > >  29:    0    0    0    0    0    0    0  454  IR-PCI-MSI 3670020-edge      eno1-TxRx-3
> > 
> > Now irqbalance comes to re-do the existing pinning, and the result is not
> > longer correct but $RANDOM for the hard queue-to-cpu case of virtio.
> 
> Then let's drop the recommendation.

Okay.

Regards,
Bastian

-- 
I'm a soldier, not a diplomat.  I can only tell the truth.
		-- Kirk, "Errand of Mercy", stardate 3198.9

[toc] | [prev] | [next] | [standalone]

#63865 — Bug#926967: Don't recommend irqbalance (was: Re: Handling irqbalance in virtual environments)

FromNicholas D Steeves <nsteeves@gmail.com>
Date2019-04-15 01:00 +0200
SubjectBug#926967: Don't recommend irqbalance (was: Re: Handling irqbalance in virtual environments)
Message-ID<xMWgp-4f8-3@gated-at.bofh.it>
In reply to#63838

[Multipart message — attachments visible in raw view] — view raw

On Fri, Apr 12, 2019 at 10:56:29PM +0200, Bastian Blank wrote:
> > On Fri, 2019-04-12 at 10:53 +0200, Bastian Blank wrote:
> > > It turns out we got again problems with irqbalance.
> > > 
> > > It was added as recommends of the main image in 3.16, as it was reported
> > > that older kernels move all interrupts to CPU 0 without help.[1]
> > > 
> > > In the meantime the kernel can do balancing on it's own.  In 4.9, I've
> > > seen it working with aacraid, each queue gets hard pinned to it's own
> > > CPU from 0 to $NRCPUS.  In 4.19 I've seen the same working properly with
> > > virtio-net.
> > > 
> > > With 4.19, even on real hardware, where interrupts have an affinity for
> > > all cpus, each interrupt is actually delivered to different cpu.
> > > 
> > > Random example for this, it even selects only one thread of each core:
> > > 
> > > >  26:    0    0    0    0   92    0    0    0  IR-PCI-MSI 3670017-edge      eno1-TxRx-0
> > > >  27:    0    0    0    0    0  167    0    0  IR-PCI-MSI 3670018-edge      eno1-TxRx-1
> > > >  28:    0    0    0    0    0    0  467    0  IR-PCI-MSI 3670019-edge      eno1-TxRx-2
> > > >  29:    0    0    0    0    0    0    0  454  IR-PCI-MSI 3670020-edge      eno1-TxRx-3
> > > 
> > > Now irqbalance comes to re-do the existing pinning, and the result is not
> > > longer correct but $RANDOM for the hard queue-to-cpu case of virtio.
> > 
> > Then let's drop the recommendation.
> 
> Okay.

+1 !  On what appears to be a related note, irqbalance + linux-4.19
appears to cause a 1.5-to-2.5w power regression on my Thinkpad X220 vs
irqbalance + linux-4.9, even when no VMs are running.

Cheers,
Nicholas

[toc] | [prev] | [next] | [standalone]

#63906 — Bug#926967: Don't recommend irqbalance (was: Re: Handling irqbalance in virtual environments)

FromBastian Blank <waldi@debian.org>
Date2019-04-23 21:50 +0200
SubjectBug#926967: Don't recommend irqbalance (was: Re: Handling irqbalance in virtual environments)
Message-ID<xQ9At-81x-9@gated-at.bofh.it>
In reply to#63838
Control: severity -1 serious

On Fri, Apr 12, 2019 at 10:56:29PM +0200, Bastian Blank wrote:
> > Then let's drop the recommendation.
> Okay.

After re-evaluating, I'll consider that a blocker.

Regards,
Bastian

-- 
The more complex the mind, the greater the need for the simplicity of play.
		-- Kirk, "Shore Leave", stardate 3025.8

[toc] | [prev] | [next] | [standalone]

#63907 — Processed: Re: Bug#926967: Don't recommend irqbalance (was: Re: Handling irqbalance in virtual environments)

From"Debian Bug Tracking System" <owner@bugs.debian.org>
Date2019-04-23 21:50 +0200
SubjectProcessed: Re: Bug#926967: Don't recommend irqbalance (was: Re: Handling irqbalance in virtual environments)
Message-ID<xQ9At-81x-15@gated-at.bofh.it>
In reply to#63838
Processing control commands:

> severity -1 serious
Bug #926967 [src:linux] Don't recommend irqbalance (was: Re: Handling irqbalance in virtual environments)
Severity set to 'serious' from 'important'

-- 
926967: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=926967
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web