Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #85233 > unrolled thread

Bug#1093243: linux-image-6.1.0-29-amd64 causes mariadb hangs

Back to article view | Back to linux.debian.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#1093243: linux-image-6.1.0-29-amd64 causes mariadb hangs Volker Maibaum <maibaum@dfn.de> - 2025-01-20 14:40 +0100
    Bug#1093243: linux-image-6.1.0-29-amd64 causes mariadb hangs Bernhard Schmidt <berni@debian.org> - 2025-01-21 20:10 +0100
      Bug#1093243: linux-image-6.1.0-29-amd64 causes mariadb hangs Max Jakub Ried <Max.Ried@hhu.de> - 2025-01-22 07:40 +0100
      Bug#1093243: linux-image-6.1.0-29-amd64 causes mariadb hangs Salvatore Bonaccorso <carnil@debian.org> - 2025-01-22 21:00 +0100
        Processed: Re: Bug#1093243: linux-image-6.1.0-29-amd64 causes  mariadb hangs "Debian Bug Tracking System" <owner@bugs.debian.org> - 2025-01-22 21:00 +0100

#85233 — Bug#1093243: linux-image-6.1.0-29-amd64 causes mariadb hangs

[Multipart message — attachments visible in raw view] — view raw

Hi,

we can confirm this issue on our systems (Debian 12) as well.

It appeared after updating the Kernel from 6.1.0-28-amd64 to 6.1.0-30-amd64.

We experienced issues with the db requests that were not processed. Connecting to 
mariadb was still possible but some updates on tables got hung resulting in a locked 
tables.
The issue is fully reproducible on multiple identical systems when updating to kernel 
6.1.0.30 and it vanishes after booting with the old kernel 6.1.0.28.

-- 
E-Mail: maibaum@dfn.de | Fon: +49 711 63314-219 | Fax: +49 30884299-370
__________________________________________________________________________________

DFN - Deutsches Forschungsnetz | German National Research and Education Network
Verein zur Förderung eines Deutschen Forschungsnetzes e.V.
Alexanderplatz 1 | 10178 Berlin
https://www.dfn.de

Vorstand: Prof. Dr.-Ing. Stefan Wesner | Prof. Dr. Helmut Reiser | Christian Zens
Geschäftsführung: Dr. Christian Grimm | Jochem Pattloch
VR AG Charlottenburg 7729B | USt.-ID. DE 136623822

[toc] | [next] | [standalone]

#85242

Control: affects -1 src:mariadb
Control: tags -1 + confirmed
Control: severity -1 critical

Seeing this too. We have two standalone systems running the stock
bookworm MariaDB and the opensource network management system LibreNMS,
which is quite write-heavy. After some time (sometimes a couple of
hours, sometimes 1-2 days) all connection slots to the database are
full.

When you kill one client process you can connect and issue "show
processlist", you see all slots busy with easy update/select queries
that have been running for hours. You need to SIGKILL mariadbd to
recover.

The last two days our colleagues running a Galera cluster (unsure about
the version, inquiring) have been affected by this as well. They found
an mariadb bug report about this.

https://jira.mariadb.org/projects/MDEV/issues/MDEV-35886?filter=allopenissues

Since there have been reports about data loss I think it warrants
increasing the severity to critical.

I'm not 100% sure about -30 though, we have been downgrading the
production system to -28 and upgraded the test system to -30, and both
are working fine. The test system has less load though, and I trust the
reports here that -30 is still broken.

[toc] | [prev] | [next] | [standalone]

#85247

[Multipart message — attachments visible in raw view] — view raw

Dear all,

I have found that at least in my case sending a SIGSTOP followed by a SIGCONT to the MariaDB process, i.e.,
kill -STOP $(pgrep -f mariadb) ; kill -CONT $(pgrep -f mariadb)

is sufficient to bring it back to life. I believe this approach has fewer side effects compared to using SIGKILL. Also attaching gdb and detaching it works, too, this is how I found out about this.


Best regards,
Max




On Tue, 21 Jan 2025 20:06:18 +0100 Bernhard Schmidt <berni@debian.org> wrote:

> When you kill one client process you can connect and issue "show
> processlist", you see all slots busy with easy update/select queries
> that have been running for hours. You need to SIGKILL mariadbd to
> recover.

[toc] | [prev] | [next] | [standalone]

#85255

Control: forwarded -1 https://jira.mariadb.org/projects/MDEV/issues/MDEV-35886
Hi,

On Tue, Jan 21, 2025 at 08:06:18PM +0100, Bernhard Schmidt wrote:
> Control: affects -1 src:mariadb
> Control: tags -1 + confirmed
> Control: severity -1 critical
> 
> Seeing this too. We have two standalone systems running the stock
> bookworm MariaDB and the opensource network management system LibreNMS,
> which is quite write-heavy. After some time (sometimes a couple of
> hours, sometimes 1-2 days) all connection slots to the database are
> full.
> 
> When you kill one client process you can connect and issue "show
> processlist", you see all slots busy with easy update/select queries
> that have been running for hours. You need to SIGKILL mariadbd to
> recover.
> 
> The last two days our colleagues running a Galera cluster (unsure about
> the version, inquiring) have been affected by this as well. They found
> an mariadb bug report about this.
> 
> https://jira.mariadb.org/projects/MDEV/issues/MDEV-35886?filter=allopenissues
> 
> Since there have been reports about data loss I think it warrants
> increasing the severity to critical.
> 
> I'm not 100% sure about -30 though, we have been downgrading the
> production system to -28 and upgraded the test system to -30, and both
> are working fine. The test system has less load though, and I trust the
> reports here that -30 is still broken.

I would be interested to know if someone is able to reproduce the
issue more in under lab conditions, which would enable us to bisect
the issue.

As a start I set the above issue as a forward, to have the issues
linked (and we later on can update it to the linux upstream report).

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#85257 — Processed: Re: Bug#1093243: linux-image-6.1.0-29-amd64 causes mariadb hangs

Processing control commands:

> forwarded -1 https://jira.mariadb.org/projects/MDEV/issues/MDEV-35886
Bug #1093243 [src:linux] linux-image-6.1.0-29-amd64 causes mariadb hangs
Set Bug forwarded-to-address to 'https://jira.mariadb.org/projects/MDEV/issues/MDEV-35886'.

-- 
1093243: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1093243
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web