Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.bugs.dist > #1130810 > unrolled thread

Bug#1027337: mariadb-server-10.5: Huge increase in memory consumption (leak?) since upgrade from 10.5.15 to 10.5.18

Back to article view | Back to linux.debian.bugs.dist

Contents

  Bug#1027337: mariadb-server-10.5: Huge increase in memory consumption (leak?) since upgrade from 10.5.15 to 10.5.18 Xan Charbonnet <xan@charbonnet.com> - 2022-12-30 17:20 +0100
    Bug#1027337: Update Xan Charbonnet <xan@charbonnet.com> - 2022-12-31 23:10 +0100
    Bug#1027337: Update 2 Xan Charbonnet <xan@charbonnet.com> - 2023-01-02 20:10 +0100
    Bug#1027337: second that bug Andreas Laut <andreas.laut@spark5.de> - 2023-01-05 15:20 +0100
      Bug#1027337: second that bug Julien Banchet <jbanchet@cusae.com> - 2023-01-05 16:20 +0100
        Bug#1027337: second that bug Andreas Laut <andreas.laut@spark5.de> - 2023-01-06 09:10 +0100
    Bug#1027337: "Starting cleanup" Xan Charbonnet <xan@charbonnet.com> - 2023-01-06 15:30 +0100
    Bug#1027337: [debian-mysql] Bug#1027337: mariadb-server-10.5: Huge increase in memory consumption (leak?) since upgrade from 10.5.15 to 10.5.18 Faustin Lammler <faustin@fala.red> - 2023-01-10 13:40 +0100
    Bug#1027337: [debian-mysql] Bug#1027337: mariadb-server-10.5: Huge increase in memory consumption (leak?) since upgrade from 10.5.15 to 10.5.18 Xan Charbonnet <xan@charbonnet.com> - 2023-01-10 17:10 +0100
      Bug#1027337: [debian-mysql] Bug#1027337: Bug#1027337: mariadb-server-10.5: Huge increase in memory consumption (leak?) since upgrade from 10.5.15 to 10.5.18 Faustin Lammler <faustin@fala.red> - 2023-01-10 20:00 +0100
    Bug#1027337: [debian-mysql] Bug#1027337: mariadb-server-10.5: Huge increase in memory consumption (leak?) since upgrade from 10.5.15 to 10.5.18 fmaesen <fmaesen@gmail.com> - 2023-02-07 16:40 +0100
    Bug#1027337: [debian-mysql] Bug#1027337: Bug#1027337: mariadb-server-10.5: Huge increase in memory consumption (leak?) since upgrade from 10.5.15 to 10.5.18 Faustin Lammler <faustin@fala.red> - 2023-02-28 18:30 +0100
      Bug#1027337: [debian-mysql] Bug#1027337: Bug#1027337: mariadb-server-10.5: Huge increase in memory consumption (leak?) since upgrade from 10.5.15 to 10.5.18 Otto Kekäläinen <otto@debian.org> - 2023-03-01 07:40 +0100

#1130810 — Bug#1027337: mariadb-server-10.5: Huge increase in memory consumption (leak?) since upgrade from 10.5.15 to 10.5.18

Package: mariadb-server-10.5
Version: 1:10.5.18-0+deb11u1
Severity: important

Dear Maintainer,

I run a pair of nameservers, and after a recent round of minor software
updates (see below for the complete list), mariadbd was OOM'd on both
machines.

These machines are minimal VMs: one has 512MB RAM, the other 1GB RAM.  They've
been running various versions of Debian flawlessly for many years.  I
temporarily upgraded the hardware to 4GB RAM on both in order to remove some
urgency from the situation.  It does appear that mariadbd's memory consumption
is far higher than it should be, and slowly growing.

There isn't too much being demanded of this database.  Its only client is
PowerDNS which runs locally.  PowerDNS was not changed in this recent update.

Please let me know what additional information I can provide.  Thank you!


Packages updated when the problem began:
base-files:amd64 from 11.1+deb11u5 to 11.1+deb11u6
nftables:amd64 from 0.9.8-3.1 to 0.9.8-3.1+deb11u1
libnftables1:amd64 from 0.9.8-3.1 to 0.9.8-3.1+deb11u1
mariadb-common:all from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
libmariadb3:amd64 from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
mariadb-client-core-10.5:amd64 from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
mariadb-client-10.5:amd64 from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
mariadb-server-core-10.5:amd64 from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
mariadb-server-10.5:amd64 from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
libtasn1-6:amd64 from 4.16.0-2 to 4.16.0-2+deb11u1
nano:amd64 from 5.4-2+deb11u1 to 5.4-2+deb11u2
distro-info-data:all from 0.51+deb11u2 to 0.51+deb11u3
grub2-common:amd64 from 2.06-3~deb11u4 to 2.06-3~deb11u5
grub-pc:amd64 from 2.06-3~deb11u4 to 2.06-3~deb11u5
grub-pc-bin:amd64 from 2.06-3~deb11u4 to 2.06-3~deb11u5
grub-common:amd64 from 2.06-3~deb11u4 to 2.06-3~deb11u5
libksba8:amd64 from 1.5.0-3+deb11u1 to 1.5.0-3+deb11u2
linux-image-amd64:amd64 from 5.10.149-2 to 5.10.158-2
linux-libc-dev:amd64 from 5.10.149-2 to 5.10.158-2
mariadb-server:all from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1




-- System Information:
Debian Release: 11.6
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-20-amd64 (SMP w/2 CPU threads)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages mariadb-server-10.5 depends on:
ii  adduser                   3.118
ii  debconf [debconf-2.0]     1.5.77
ii  galera-4                  26.4.11-0+deb11u1
ii  gawk                      1:5.1.0-1
ii  iproute2                  5.10.0-4
ii  libc6                     2.31-13+deb11u5
ii  libdbi-perl               1.643-3+b1
ii  libpam0g                  1.4.0-9+deb11u1
ii  libssl1.1                 1.1.1n-0+deb11u3
ii  libstdc++6                10.2.1-6
ii  lsb-base                  11.1.0
ii  lsof                      4.93.2+dfsg-1.1
ii  mariadb-client-10.5       1:10.5.18-0+deb11u1
ii  mariadb-common            1:10.5.18-0+deb11u1
ii  mariadb-server-core-10.5  1:10.5.18-0+deb11u1
ii  passwd                    1:4.8.1-1
ii  perl                      5.32.1-4+deb11u2
ii  procps                    2:3.3.17-5
ii  psmisc                    23.4-2
ii  rsync                     3.2.3-4+deb11u1
ii  socat                     1.7.4.1-3
ii  zlib1g                    1:1.2.11.dfsg-2+deb11u2

Versions of packages mariadb-server-10.5 recommends:
ii  libhtml-template-perl  2.97-1.1

Versions of packages mariadb-server-10.5 suggests:
ii  mailutils [mailx]  1:3.10-3+b1
pn  mariadb-test       <none>
pn  netcat-openbsd     <none>

-- debconf information excluded

[toc] | [next] | [standalone]

#1130995 — Bug#1027337: Update

Some memory statistics.  Uptime, resident memory size, and % of memory 
in the system for mariadbd:

30 hours -- 1.0g -- 26.7%
36 hours -- 1.2g -- 30.3%
54 hours -- 1.6g -- 41.5%

Looks like a pretty linear progression of ~33MB/hour being leaked.  This 
is on a system that previously had 512MB of total RAM.  Now with 4GB it 
appears it'll be able to stay up about 5 days total.  There were no 
configuration changes other than upgrading the packages.

[toc] | [prev] | [next] | [standalone]

#1131332 — Bug#1027337: Update 2

At the 96 hour mark it hit 2.5GB, 64.6%.  I reverted the seven MariaDB 
packages back to 10.5.15, and the problem has disappeared.  mariadbd is 
holding steady at 135MB, 3.4%.

I'm happy to help participate in figuring out what's going wrong.

[toc] | [prev] | [next] | [standalone]

#1131802 — Bug#1027337: second that bug

Hi all,

we'd like to say: We have hit the same bug in the same way and we have 
no idea what happens here.

We are also running this mariadb server as powerdns backend (means small 
dataset) and since the upgrade to 10.5.18 the ressources usage increase 
constantly. Not only RSS of mariadbd. It's also the cpu load which 
increases the same way. Until the memory limit is reached and the 
database process gets oom killed.

[toc] | [prev] | [next] | [standalone]

#1131814 — Bug#1027337: second that bug

> On 5 Jan 2023, at 15:05, Andreas Laut <andreas.laut@spark5.de> wrote:
> 
> Hi all,
> 
> we'd like to say: We have hit the same bug in the same way and we have no idea what happens here.
> 
> We are also running this mariadb server as powerdns backend (means small dataset) and since the upgrade to 10.5.18 the ressources usage increase constantly. Not only RSS of mariadbd. It's also the cpu load which increases the same way. Until the memory limit is reached and the database process gets oom killed.


Same here,
Though, oddly, it doesn't seem to effect each and every instance of 10.5.18 (most were automatically upgraded)

For instance in one of the master-master setups we have for a powerdns+radius backend, it's mainly the second node that has little queries that hogs memory, while the first, much more solicited, is keeping in check

https://www.evernote.com/l/ACgw53fJdg5C_qBaMsAkst14aXRnI1tJjhg

Cordialement,
Julien Banchet
--
Cusae - 6 Allée Cardinal de Givry 21000 Dijon
Tél : +33951717101 / +33651886877

[toc] | [prev] | [next] | [standalone]

#1131919 — Bug#1027337: second that bug

[Multipart message — attachments visible in raw view] — view raw

What we tested yesterday:

dump the pdns database, drop it and push all data again into 
mariadb-server. To be sure that the data doesn't get messed up) After 
that we restarted pdns prozess.

|mysqldump pdns > /tmp/pdns.sql; mysql -e "DROP DATABASE pdns"; mysql -e 
"CREATE DATABASE pdns";cat /tmp/pdns.sql | mysql pdns; service pdns restart|

(command not tested so don't use this directly!!, just a sum up of the 
commands used)


You can see the effect on this graph. At point x it was done. CPU load 
of mariadb-server process goes down but it started again to increase. On 
the memory usage it has no effect, increase constantly.

https://iili.io/HRvj4wP.png

[toc] | [prev] | [next] | [standalone]

#1131976 — Bug#1027337: "Starting cleanup"

Every few seconds (at least in my configuration), PDNS runs database 
queries to see which domains need refreshing.

After running for a while with this problem, I can "catch" those queries 
(at least one of them) with SHOW PROCESSLIST.
Excerpt:
| 117 | pdns | localhost | pdns | Execute |    0 | Starting cleanup | 
SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records 
WHERE disabled=0 and type=? and name=? |    0.000 |

It seems to me that the longer this problem has been going on, the 
longer this query takes, at least the "starting cleanup" phase of it.  I 
think that's why CPU usage climbs: the longer it goes, the longer this 
query takes and the more CPU is used to execute it.

Caught a different one just now, also in phase "starting cleanup":
| 128 | pdns | localhost | pdns | Execute |    0 | Starting cleanup | 
SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records 
WHERE disabled=0 and name=? and domain_id=? |    0.000 |

Yes, I can regularly "catch" many queries and they're always in state 
"starting cleanup".

I went over to the machine that is not having this problem (because it's 
still on 10.5.15) and was able to catch one such query in "starting 
cleanup", and another in "commit".  But it took maybe 25x more attempts 
to catch such a query than on the one with the problem.  The queries on 
the "good" box finish very quickly, and they're slow on the "bad" box.

[toc] | [prev] | [next] | [standalone]

#1132623 — Bug#1027337: [debian-mysql] Bug#1027337: mariadb-server-10.5: Huge increase in memory consumption (leak?) since upgrade from 10.5.15 to 10.5.18

[Multipart message — attachments visible in raw view] — view raw

Hi,
first of all, thank you all for the excellent (and detailed) report!

I don't see anything that could be related with Debian packaging of
MariaDB and so I suggest you check directly upstream.

I have searched into https://jira.mariadb.org and the closest report
that I can come with is https://jira.mariadb.org/browse/MDEV-29988. I do
not forward this bug report to it because I am not sure that this is the
same problem.

There is also https://jira.mariadb.org/browse/MDEV-29097 that may give
some ideas on how to find a workaround until the next release happen.

Cheers!

-- 
Faustin
GPG: F652 BCD1 1AA8 8975 F010 48A5 390A 2F27 832A 5C79

[toc] | [prev] | [next] | [standalone]

#1132641 — Bug#1027337: [debian-mysql] Bug#1027337: mariadb-server-10.5: Huge increase in memory consumption (leak?) since upgrade from 10.5.15 to 10.5.18

Faustin,

Thanks for looking into this, and for your work packaging MariaDB for 
Debian.

It looks to me like https://jira.mariadb.org/browse/MDEV-29988 is very 
likely to be the issue we're seeing.  Fortunately (if I'm interpreting 
upstream's JIRA correctly), it looks like a fix exists and will be 
included in 10.5.19.

 From their release history, 10.5.19 can be expected in early February.

Probably what I will do is downgrade all machines to 10.5.15 and look 
forward to the Debian package of 10.5.19, which hopefully will not be 
long after the upstream release.

[toc] | [prev] | [next] | [standalone]

#1132667 — Bug#1027337: [debian-mysql] Bug#1027337: Bug#1027337: mariadb-server-10.5: Huge increase in memory consumption (leak?) since upgrade from 10.5.15 to 10.5.18

[Multipart message — attachments visible in raw view] — view raw

Hi!

Xan Charbonnet <xan@charbonnet.com>,
10/01/2023 – 10:07:02 (-0600):

> It looks to me like https://jira.mariadb.org/browse/MDEV-29988 is very
> likely to be the issue we're seeing.  Fortunately (if I'm interpreting
> upstream's JIRA correctly), it looks like a fix exists and will be included
> in 10.5.19.

Indeed, MDEV-29988 is fixed in the next release.

> From their release history, 10.5.19 can be expected in early February.

Exact, maybe even by the end of January.

> Probably what I will do is downgrade all machines to 10.5.15 and look
> forward to the Debian package of 10.5.19, which hopefully will not be long
> after the upstream release.

I already sent an email to Otto about this critical issue and hopefully,
10.5.19 will hit Debian soon after it's GA upstream.

Cheers!

-- 
Faustin

[toc] | [prev] | [next] | [standalone]

#1136837 — Bug#1027337: [debian-mysql] Bug#1027337: mariadb-server-10.5: Huge increase in memory consumption (leak?) since upgrade from 10.5.15 to 10.5.18

[Multipart message — attachments visible in raw view] — view raw

Hi all, there is an upstream release 10.5.19 with hopefully a fix:

https://mariadb.com/kb/en/mariadb-10-5-19-release-notes/


On Thu, 12 Jan 2023 08:55:06 +0000 Matteo Valsasna <MValsasna@pdxeng.ch>
wrote:
> Thanks a lot Faustin
>
>
>
> I found an useful description of MDEV-29988 in a comment to one of its
duplicates, and it matches quite well our application scenario for the
database connection that cause the memory leak according to
information_schema.PROCESSLIST.memory_used
> (prepared statements executed maaany times, string comparisons)
>
> we reproduced the issue on a test machine, and we will try to downgrade
to 10.5.15
>
>
>
> MAtteo
>
>
>
>
https://jira.mariadb.org/browse/MDEV-30096?focusedCommentId=243814&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-243814
>
> [serg]Sergei Golubchik<
https://jira.mariadb.org/secure/ViewProfile.jspa?name=serg> added a comment
- 2022-11-28 22:20
>
> Yes, <https://jira.mariadb.org/browse/MDEV-29988>  was in 10.6.11 too.
But let's first check the preconditions:
>
>   *   you use prepared statements
>   *   you prepare once and then execute many thousands of times before
deallocating the statement
>   *   your statements contains a string comparison with strings in
different (but compatible) character sets (e.g. utf8 and latin1).
>
> if all that matches your use case you could be affected by <
https://jira.mariadb.org/browse/MDEV-29988> . If you do, then
>
>   *   every execution of such a prepared statements will allocate more
memory
>   *   all memory will be freed when a statements is deallocated
(explicitly or when the client disconnects)
>   *   time to deallocate a prepared statement is proportional to the
number of times it was executed
>
> and workarounds could be
>
>   *   deallocate prepared statements more often
>   *   rewrite statements to avoid comparing strings in different
character sets
>
> or wait for the next release that will have it fixed
>
>
>
> On Tue, 10 Jan 2023 13:36:02 +0100 Faustin Lammler <faustin@fala.red>
wrote:
> > Hi,
> > first of all, thank you all for the excellent (and detailed) report!
> >
> > I don't see anything that could be related with Debian packaging of
> > MariaDB and so I suggest you check directly upstream.
> >
> > I have searched into https://jira.mariadb.org and the closest report
> > that I can come with is https://jira.mariadb.org/browse/MDEV-29988. I do
> > not forward this bug report to it because I am not sure that this is the
> > same problem.
> >
> > There is also https://jira.mariadb.org/browse/MDEV-29097 that may give
> > some ideas on how to find a workaround until the next release happen.
> >
> > Cheers!
> >
> > --
> > Faustin
> > GPG: F652 BCD1 1AA8 8975 F010 48A5 390A 2F27 832A 5C79

[toc] | [prev] | [next] | [standalone]

#1139628 — Bug#1027337: [debian-mysql] Bug#1027337: Bug#1027337: mariadb-server-10.5: Huge increase in memory consumption (leak?) since upgrade from 10.5.15 to 10.5.18

[Multipart message — attachments visible in raw view] — view raw

Hi Matteo!
I am not in charge of doing uploads to Debian FTP so, I don't know.

@Otto do you know when 10.5.19 will be uploaded?

Regards,

-- 
Faustin
GPG: F652 BCD1 1AA8 8975 F010 48A5 390A 2F27 832A 5C79

[toc] | [prev] | [next] | [standalone]

#1139701 — Bug#1027337: [debian-mysql] Bug#1027337: Bug#1027337: mariadb-server-10.5: Huge increase in memory consumption (leak?) since upgrade from 10.5.15 to 10.5.18

On Tue, 28 Feb 2023 at 09:25, Faustin Lammler <faustin@fala.red> wrote:
>
> Hi Matteo!
> I am not in charge of doing uploads to Debian FTP so, I don't know.
>
> @Otto do you know when 10.5.19 will be uploaded?

Only Debian release team knows the exact date. Page
https://release.debian.org/ says mid-February.

You can +1 on https://bugs.debian.org/1031042 if you want or you can
already now download binaries from the CI pipeline from commit
https://salsa.debian.org/mariadb-team/mariadb-10.5/-/commits/debian/1%2510.5.19-0+deb11u1
= https://salsa.debian.org/mariadb-team/mariadb-10.5/-/pipelines/498296
= https://salsa.debian.org/mariadb-team/mariadb-10.5/-/jobs/3929573/artifacts/browse/debian/output/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.bugs.dist

csiph-web