Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #52381 > unrolled thread

Bug#784688: Thousands of "xen:balloon: Cannot add additional memory (-17) messages" despite dom0 ballooning disabled

Started byKSB <ksb@inbox.lv>
First post2016-01-22 21:00 +0100
Last post2016-02-01 14:20 +0100
Articles 11 — 4 participants

Back to article view | Back to linux.debian.kernel

Contents

  Bug#784688: Thousands of "xen:balloon: Cannot add additional memory (-17) messages" despite dom0 ballooning disabled KSB <ksb@inbox.lv> - 2016-01-22 21:00 +0100
    Bug#784688: Thousands of "xen:balloon: Cannot add additional memory (-17) messages" despite dom0 ballooning disabled Ian Campbell <ijc@debian.org> - 2016-01-25 15:10 +0100
      Bug#784688: Thousands of "xen:balloon: Cannot add additional memory (-17) messages" despite dom0 ballooning disabled KSB <ksb@inbox.lv> - 2016-01-25 19:40 +0100
        Bug#784688: Thousands of "xen:balloon: Cannot add additional memory (-17) messages" despite dom0 ballooning disabled Ian Campbell <ijc@debian.org> - 2016-01-26 11:00 +0100
        Bug#784688: Thousands of "xen:balloon: Cannot add additional memory (-17) messages" despite dom0 ballooning disabled Ian Campbell <ijc@debian.org> - 2016-01-26 13:50 +0100
          Bug#784688: Thousands of "xen:balloon: Cannot add additional memory (-17) messages" despite dom0 ballooning disabled KSB <ksb@inbox.lv> - 2016-01-26 19:00 +0100
            Bug#784688: Thousands of "xen:balloon: Cannot add additional memory (-17) messages" despite dom0 ballooning disabled Ian Campbell <ijc@debian.org> - 2016-01-27 12:10 +0100
              Bug#784688: Thousands of "xen:balloon: Cannot add additional memory (-17) messages" despite dom0 ballooning disabled Chad Dougherty <crd@andrew.cmu.edu> - 2016-01-27 15:40 +0100
              Bug#784688: Thousands of "xen:balloon: Cannot add additional memory (-17) messages" despite dom0 ballooning disabled Ian Campbell <ijc@debian.org> - 2016-01-29 16:10 +0100
                Bug#784688: Thousands of "xen:balloon: Cannot add additional memory (-17) messages" despite dom0 ballooning disabled Andy Smith <andy@strugglers.net> - 2016-01-29 17:10 +0100
                  Bug#784688: Thousands of "xen:balloon: Cannot add additional memory (-17) messages" despite dom0 ballooning disabled Ian Campbell <ijc@debian.org> - 2016-02-01 14:20 +0100

#52381 — Bug#784688: Thousands of "xen:balloon: Cannot add additional memory (-17) messages" despite dom0 ballooning disabled

FromKSB <ksb@inbox.lv>
Date2016-01-22 21:00 +0100
SubjectBug#784688: Thousands of "xen:balloon: Cannot add additional memory (-17) messages" despite dom0 ballooning disabled
Message-ID<qTQ2e-cO-7@gated-at.bofh.it>
Seen this behavior on earlier kernels (i.e. 3.14-2-amd64 pkg 3.14.15-2.) 
and seems to be gone at least in 4.3

[toc] | [next] | [standalone]

#52417

FromIan Campbell <ijc@debian.org>
Date2016-01-25 15:10 +0100
Message-ID<qUQ0b-54G-45@gated-at.bofh.it>
In reply to#52381
On Fri, 2016-01-22 at 21:38 +0200, KSB wrote:
> Seen this behavior on earlier kernels (i.e. 3.14-2-amd64 pkg 3.14.15-2.) 
> and seems to be gone at least in 4.3

That's useful info thanks, I've been unable to pinpoint a culprit for this
for ages now.

Do you have a package version which you know to be good? How confident are
you that it is ok (sometimes the problem is intermittent)?

Lastly, is there any chance you upgraded the Xen packages at the same time?
I'm starting to wonder if maybe this is not a kernel issue.

Ian.

[toc] | [prev] | [next] | [standalone]

#52437

FromKSB <ksb@inbox.lv>
Date2016-01-25 19:40 +0100
Message-ID<qUUds-88K-21@gated-at.bofh.it>
In reply to#52417
> Do you have a package version which you know to be good? How confident are
> you that it is ok (sometimes the problem is intermittent)?
>
> Lastly, is there any chance you upgraded the Xen packages at the same time?
> I'm starting to wonder if maybe this is not a kernel issue.
>
Sorry, but there is chance, sadly.

But I checked logs more thoroughly and found it even on more recent kernels:
1) Lot of messages on 3.14-2-amd64 with xen-4.6, 13 domU's.
2) 4.3.0-1-amd64 xen-4.6, only two messages shortly after boot, only 1 
domU running:
[   12.473778] xen:balloon: Cannot add additional memory (-17)
[   21.673298] xen:balloon: Cannot add additional memory (-17)
uptime 17 days.

Previous on same machine was 4.2.0-1-amd64 with more (-17)'s

3) 4.3.0-1-amd64, one month, several reboots, average 4 domU's, and no 
messages

4) 3.16.0-4-amd64, xen-4.1, 22 domU's, uptime 188 days, in last month I 
see only
Jan 7 14:12:08
Jan 7 14:12:08
Jan 7 14:12:08
Jan 7 14:12:08
Jan 7 14:27:47
Jan 7 14:27:47
Jan 7 14:27:47
Jan 7 14:27:48
and this is roughly the time last machine was created(started).

[toc] | [prev] | [next] | [standalone]

#52450

FromIan Campbell <ijc@debian.org>
Date2016-01-26 11:00 +0100
Message-ID<qV8zM-2fZ-25@gated-at.bofh.it>
In reply to#52437
On Mon, 2016-01-25 at 20:36 +0200, KSB wrote:
> > Do you have a package version which you know to be good? How confident
> > are
> > you that it is ok (sometimes the problem is intermittent)?
> > 
> > Lastly, is there any chance you upgraded the Xen packages at the same
> > time?
> > I'm starting to wonder if maybe this is not a kernel issue.
> > 
> Sorry, but there is chance, sadly.

This is actually useful, because it shows that the issue occurs even with
Xen 4.6, which I think rules out a Xen side issue (otherwise we'd have had
lots more reports from 4.4 through to 4.6) and points to a kernel side
issue.

> But I checked logs more thoroughly and found it even on more recent
> kernels:
> 1) Lot of messages on 3.14-2-amd64 with xen-4.6, 13 domU's.

Just to be clear, "Lots" here means "hundreds or thousands"? I think it is
expected to see one or two around the time a VM is started or stopped, so
with 13 domUs a couple of dozen messages wouldn't seem out of line to me.

> 2) 4.3.0-1-amd64 xen-4.6, only two messages shortly after boot, only 1 
> domU running:
> [   12.473778] xen:balloon: Cannot add additional memory (-17)
> [   21.673298] xen:balloon: Cannot add additional memory (-17)
> uptime 17 days.
> 
> Previous on same machine was 4.2.0-1-amd64 with more (-17)'s

Was it running xen-4.6 when it was running 4.2.0 or was that also older?

Also 4.2.0-1-amd64 here (and all the other numbers you gave) is the ABI,
not the package version. The package versions is either in dpkg or you can
find it in /proc/version:

Linux version 4.1.0-2-amd64 (debian-kernel@lists.debian.org) (gcc version 4.9.3 (Debian 4.9.3-3) ) #1 SMP Debian 4.1.6-1 (2015-08-23)
              ^^^ABI^^^^^^^                                                                               ^^^VERSION^^^^

> 3) 4.3.0-1-amd64, one month, several reboots, average 4 domU's, and no 
> messages

Any idea which Xen?

> 4) 3.16.0-4-amd64, xen-4.1, 22 domU's, uptime 188 days, in last month I 
> see only
> Jan 7 14:12:08
> Jan 7 14:12:08
> Jan 7 14:12:08
> Jan 7 14:12:08
> Jan 7 14:27:47
> Jan 7 14:27:47
> Jan 7 14:27:47
> Jan 7 14:27:48
> and this is roughly the time last machine was created(started).
> 
> 
>

[toc] | [prev] | [next] | [standalone]

#52453

FromIan Campbell <ijc@debian.org>
Date2016-01-26 13:50 +0100
Message-ID<qVbeh-4b6-5@gated-at.bofh.it>
In reply to#52437
On Mon, 2016-01-25 at 20:36 +0200, KSB wrote:
> > Do you have a package version which you know to be good? How confident
> > are
> > you that it is ok (sometimes the problem is intermittent)?
> > 
> > Lastly, is there any chance you upgraded the Xen packages at the same
> > time?
> > I'm starting to wonder if maybe this is not a kernel issue.
> > 
> Sorry, but there is chance, sadly.

This is actually useful, because it shows that the issue occurs even with
Xen 4.6, which I think rules out a Xen side issue (otherwise we'd have had
lots more reports from 4.4 through to 4.6) and points to a kernel side
issue somewhere.

> But I checked logs more thoroughly and found it even on more recent
> kernels:
> 1) Lot of messages on 3.14-2-amd64 with xen-4.6, 13 domU's.

Just to be clear, "Lots" here means "hundreds or thousands"? I think it is
expected to see one or two around the time a VM is started or stopped, so
with 13 domUs a couple of dozen messages wouldn't seem out of line to me.

> 2) 4.3.0-1-amd64 xen-4.6, only two messages shortly after boot, only 1 
> domU running:
> [   12.473778] xen:balloon: Cannot add additional memory (-17)
> [   21.673298] xen:balloon: Cannot add additional memory (-17)
> uptime 17 days.
> 
> Previous on same machine was 4.2.0-1-amd64 with more (-17)'s

Was it running xen-4.6 when it was running 4.2.0 or was that also older?

Also 4.2.0-1-amd64 is the ABI, not the package version. The package
versions is either in dpkg or you can find it in /proc/version:

Linux version 4.1.0-2-amd64 (debian-kernel@lists.debian.org) (gcc version 4.9.3 (Debian 4.9.3-3) ) #1 SMP Debian 4.1.6-1 (2015-08-23)
              ^^^ABI^^^^^^^                                                                               ^^^VERSION^^^^

> 3) 4.3.0-1-amd64, one month, several reboots, average 4 domU's, and no 
> messages

Any idea which Xen?

> 4) 3.16.0-4-amd64, xen-4.1, 22 domU's, uptime 188 days, in last month I 
> see only
> Jan 7 14:12:08
> Jan 7 14:12:08
> Jan 7 14:12:08
> Jan 7 14:12:08
> Jan 7 14:27:47
> Jan 7 14:27:47
> Jan 7 14:27:47
> Jan 7 14:27:48
> and this is roughly the time last machine was created(started).
> 
> 
>

[toc] | [prev] | [next] | [standalone]

#52459

FromKSB <ksb@inbox.lv>
Date2016-01-26 19:00 +0100
Message-ID<qVg4i-7AF-1@gated-at.bofh.it>
In reply to#52453
> This is actually useful, because it shows that the issue occurs even with
> Xen 4.6, which I think rules out a Xen side issue (otherwise we'd have had
> lots more reports from 4.4 through to 4.6) and points to a kernel side
> issue somewhere.
>
>> But I checked logs more thoroughly and found it even on more recent
>> kernels:
>> 1) Lot of messages on 3.14-2-amd64 with xen-4.6, 13 domU's.
>
> Just to be clear, "Lots" here means "hundreds or thousands"? I think it is
> expected to see one or two around the time a VM is started or stopped, so
> with 13 domUs a couple of dozen messages wouldn't seem out of line to me.
>
pkg 3.14.15-2
~1600 from last dmesg cleanup which was 23h ago, but all of them 
distributed in last 15h


>> 2) 4.3.0-1-amd64 xen-4.6, only two messages shortly after boot, only 1
>> domU running:
>> [   12.473778] xen:balloon: Cannot add additional memory (-17)
>> [   21.673298] xen:balloon: Cannot add additional memory (-17)
>> uptime 17 days.
>>
>> Previous on same machine was 4.2.0-1-amd64 with more (-17)'s
>
> Was it running xen-4.6 when it was running 4.2.0 or was that also older?

4.3.3-5 xen-4.6.0 and previous 4.2.6-1 xen-4.4.1

>
> Also 4.2.0-1-amd64 is the ABI, not the package version. The package
> versions is either in dpkg or you can find it in /proc/version:
>
> Linux version 4.1.0-2-amd64 (debian-kernel@lists.debian.org) (gcc version 4.9.3 (Debian 4.9.3-3) ) #1 SMP Debian 4.1.6-1 (2015-08-23)
>                ^^^ABI^^^^^^^                                                                               ^^^VERSION^^^^

Ok, if pkg versions is more important, then I updated all data with pkg 
versions in this post


>> 3) 4.3.0-1-amd64, one month, several reboots, average 4 domU's, and no
>> messages
>
> Any idea which Xen?

kernel pkg 4.3.3-5 and 4.3.3-7 and xen-4.6.0

>
>> 4) 3.16.0-4-amd64, xen-4.1, 22 domU's, uptime 188 days, in last month I
>> see only
>> Jan 7 14:12:08
>> Jan 7 14:12:08
>> Jan 7 14:12:08
>> Jan 7 14:12:08
>> Jan 7 14:27:47
>> Jan 7 14:27:47
>> Jan 7 14:27:47
>> Jan 7 14:27:48
>> and this is roughly the time last machine was created(started).

pkg 3.16.7-ckt7-1

[toc] | [prev] | [next] | [standalone]

#52467

FromIan Campbell <ijc@debian.org>
Date2016-01-27 12:10 +0100
Message-ID<qVw94-2C5-1@gated-at.bofh.it>
In reply to#52459
On Tue, 2016-01-26 at 19:46 +0200, KSB wrote:
> > This is actually useful, because it shows that the issue occurs even
> > with
> > Xen 4.6, which I think rules out a Xen side issue (otherwise we'd have
> > had
> > lots more reports from 4.4 through to 4.6) and points to a kernel side
> > issue somewhere.
> > 
> > > But I checked logs more thoroughly and found it even on more recent
> > > kernels:
> > > 1) Lot of messages on 3.14-2-amd64 with xen-4.6, 13 domU's.
> > 
> > Just to be clear, "Lots" here means "hundreds or thousands"? I think it
> > is
> > expected to see one or two around the time a VM is started or stopped,
> > so
> > with 13 domUs a couple of dozen messages wouldn't seem out of line to
> > me.
> > 
> pkg 3.14.15-2
> ~1600 from last dmesg cleanup which was 23h ago, but all of them 
> distributed in last 15h
> 
> 
> > > 2) 4.3.0-1-amd64 xen-4.6, only two messages shortly after boot, only
> > > 1
> > > domU running:
> > > [   12.473778] xen:balloon: Cannot add additional memory (-17)
> > > [   21.673298] xen:balloon: Cannot add additional memory (-17)
> > > uptime 17 days.
> > > 
> > > Previous on same machine was 4.2.0-1-amd64 with more (-17)'s
> > 
> > Was it running xen-4.6 when it was running 4.2.0 or was that also
> > older?
> 
> 4.3.3-5 xen-4.6.0 and previous 4.2.6-1 xen-4.4.1

Thanks. And just to clarify, with Linux 4.2.6-1 xen-4.4.1 you were or were
not seeing this issue?

To summarise what I can tell from this bug log the following combinations
are/are not prone to this issue:

                        xen-??? xen-4.1 xen-4.4.1 4.4.1-9+deb8u3 xen-4.6.0
3.14.15-2                                                        Y[1]

3.16.7-ckt7-1                   N[1]
3.16.7-ckt9-3~deb8u1    Y[2]
3.16.7-ckt20-1+deb8u2                             Y[3]

4.2.6-1                                 ?[1]
4.3.3-5                                                          NN[1]
4.3.3-7                                                          N[1]

[1] KSV
[2] ML (original report, Xen version unknown)
[3] AS (with dom0_mem=1024M,max:1024M, but not dom0_mem=1024M)

The N for xen-4.1 + linux-3.16.7-ckt7-1 (KSV's #4) seems anomalous. Perhaps that version is susceptible but not exhibiting it during the span of the logs.

The ? for xen-4.4.1 + linux-4.2.6-1 is the "just to clarify" above.

In any case it does appear to correlate with the Linux version and not the Xen version, and it does appear to be fixed in 4.3.3-5, or possibly even 4.2.6-1.

I'm still unable to spot what might have changed between 3.16.7-ckt20-1+deb8u2 and 4.3.3-5 though to explain it going away, which I'd still quite liketo get to the bottom of in order to fix in Jessie.

Thanks,


Ian.

[toc] | [prev] | [next] | [standalone]

#52469

FromChad Dougherty <crd@andrew.cmu.edu>
Date2016-01-27 15:40 +0100
Message-ID<qVzqh-4RW-1@gated-at.bofh.it>
In reply to#52467
On 2016-01-27 05:57, Ian Campbell wrote:
> To summarise what I can tell from this bug log the following combinations
> are/are not prone to this issue:
>
>                          xen-??? xen-4.1 xen-4.4.1 4.4.1-9+deb8u3 xen-4.6.0
> 3.14.15-2                                                        Y[1]
>
> 3.16.7-ckt7-1                   N[1]
> 3.16.7-ckt9-3~deb8u1    Y[2]
> 3.16.7-ckt20-1+deb8u2                             Y[3]
>
> 4.2.6-1                                 ?[1]
> 4.3.3-5                                                          NN[1]
> 4.3.3-7                                                          N[1]
>
> [1] KSV
> [2] ML (original report, Xen version unknown)
> [3] AS (with dom0_mem=1024M,max:1024M, but not dom0_mem=1024M)
>

Although it may not add a lot to the situation at this point, you can 
add my configuration as being affected:
- kernel 3.16.7-ckt11-1+deb8u5
- Xen 4.4.1-9+deb8u2
- autoballoon="off" in xl.conf
- GRUB_CMDLINE_XEN_DEFAULT="dom0_mem=2048M,max:2056M" (and also just 
"dom0_mem=2048M").

-- 
     -Chad

[toc] | [prev] | [next] | [standalone]

#52499

FromIan Campbell <ijc@debian.org>
Date2016-01-29 16:10 +0100
Message-ID<qWiQr-4Xa-43@gated-at.bofh.it>
In reply to#52467
On Wed, 2016-01-27 at 10:57 +0000, Ian Campbell wrote:
> 
> I'm still unable to spot what might have changed between 3.16.7-ckt20-
> 1+deb8u2 and 4.3.3-5 though to explain it going away, which I'd still
> quite liketo get to the bottom of in order to fix in Jessie.

I spent a bit of time investigating this, but sadly I'm not able to
reproduce the basis failure.

I've tried the combinations below and all are OK. Some of them produce 1 or
2 of the "-17" messages (I should have noted which but didn't, I think it
was most) but in no case did I see thousands of them.

Ian.

for i in $(seq 1 15) ; do xl reboot debian.guest.osstest ; sleep 10s; done

dom0_mem=2048M,max:2056M
L: 3.16.7-ckt20-1+deb8u3
X: 4.6.0-1+nmu1

=> OK

dom0_mem=2048M,max:2048M
L: 3.16.7-ckt20-1+deb8u3
X: 4.6.0-1+nmu1

=> OK

dom0_mem=2048M,max:2048M
L: 3.16.7-ckt9-3~deb8u1
X: 4.6.0-1+nmu1

=> OK

---------------------------------------------------------------------

for i in $(seq 1 15) ; do xl create /etc/xen/debian.guest.osstest.cfg ; sleep 10s ;  xl shutdown -w debian.guest.osstest; sleep 5s; done

dom0_mem=2048M,max:2048M
L: 3.16.7-ckt9-3~deb8u1
X: 4.6.0-1+nmu1

=> OK

dom0_mem=2048M,max:2048M
L: 3.16.7-ckt9-3~deb8u1
X: 4.4.1-9+deb8u3

=> OK

[toc] | [prev] | [next] | [standalone]

#52501

FromAndy Smith <andy@strugglers.net>
Date2016-01-29 17:10 +0100
Message-ID<qWjMu-5G8-29@gated-at.bofh.it>
In reply to#52499
Hi Ian,

On Fri, Jan 29, 2016 at 02:57:23PM +0000, Ian Campbell wrote:
> I spent a bit of time investigating this, but sadly I'm not able to
> reproduce the basis failure.

FWIW it was me who reported this with the packages in Debian stable
(linux-image-3.16.0-4-amd64 3.16.7-ckt20-1+deb8u3,
xen-hypervisor-4.4-amd64 4.4.1-9+deb8u3) when using
"dom0_mem=1024M,max:1024M" on the hypervisor command line.

I must admit I found this bug when searching for the error message,
and have only been seeing it printed a couple of times at guest
shutdown, not thousands of times.

So if having it printed a couple of times isn't considered a bug,
I'm sorry if I've led you astray here. Might be worth finding a way
to remove it anyway though; anyone having a problem is going to keep
searching for it thinking it is relevant to their case.

At the moment I am avoiding seeing the message at all by running with
only "dom0_mem=1024M" on the command line. What's the disadvantage
of not having the "max:1024M" there?

Cheers,
Andy

[toc] | [prev] | [next] | [standalone]

#52537

FromIan Campbell <ijc@debian.org>
Date2016-02-01 14:20 +0100
Message-ID<qXmyC-3iQ-5@gated-at.bofh.it>
In reply to#52501
On Fri, 2016-01-29 at 15:59 +0000, Andy Smith wrote:
> Hi Ian,
> 
> On Fri, Jan 29, 2016 at 02:57:23PM +0000, Ian Campbell wrote:
> > I spent a bit of time investigating this, but sadly I'm not able to
> > reproduce the basis failure.
> 
> FWIW it was me who reported this with the packages in Debian stable
> (linux-image-3.16.0-4-amd64 3.16.7-ckt20-1+deb8u3,
> xen-hypervisor-4.4-amd64 4.4.1-9+deb8u3) when using
> "dom0_mem=1024M,max:1024M" on the hypervisor command line.
> 
> I must admit I found this bug when searching for the error message,
> and have only been seeing it printed a couple of times at guest
> shutdown, not thousands of times.
> 
> So if having it printed a couple of times isn't considered a bug,
> I'm sorry if I've led you astray here.

No worries, thanks for letting me know.

>  Might be worth finding a way
> to remove it anyway though; anyone having a problem is going to keep
> searching for it thinking it is relevant to their case.

Indeed. That might be tricky to arrange 100% reliably due to the way these
things work out in practice wrt ballooning and maxes and dynamic changes
due to the use by backends, an upstream thing in any case.

> At the moment I am avoiding seeing the message at all by running with
> only "dom0_mem=1024M" on the command line. What's the disadvantage
> of not having the "max:1024M" there?

I'm not 100% sure. It looks like it causes no max to be set (LONG_MAX is
the default), which I suppose would allow dom0 (from Xen's PoV, the kernel
might have its own limitations) to balloon to more than 1024M if it tried
to (which would explain it working around this issue without).

Ian.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web