Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #84547 > unrolled thread

Bug#1087602: linux: io_uring deadlock when building chromium (6.1.115 regression)

Back to article view | Back to linux.debian.kernel

Contents

  Bug#1087602: linux: io_uring deadlock when building chromium (6.1.115 regression) Aurelien Jarno <aurel32@debian.org> - 2024-11-15 22:40 +0100
    Bug#1087602: linux: io_uring deadlock when building chromium (6.1.115 regression) Aurelien Jarno <aurel32@debian.org> - 2024-11-19 06:40 +0100
    Bug#1087602: linux: io_uring deadlock... Salvatore Bonaccorso <carnil@debian.org> - 2024-11-19 15:20 +0100

#84547 — Bug#1087602: linux: io_uring deadlock when building chromium (6.1.115 regression)

Source: linux
Version: 6.1.115-1
Severity: important
Tags: upstream
X-Debbugs-Cc: dsa@debian.org

Dear maintainers,

Since the last Bookworm point release, the build daemons often run into
a io_uring deadlock when building chromium:

[123733.312356] INFO: task iou-sqp-2625111:2625119 blocked for more than 120 seconds.
[123733.313547]       Not tainted 6.1.0-27-amd64 #1 Debian 6.1.115-1
[123733.314418] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[123733.315543] task:iou-sqp-2625111 state:D stack:0     pid:2625119 ppid:2625110 flags:0x00004000
[123733.315548] Call Trace:
[123733.315550]  <TASK>
[123733.315556]  __schedule+0x34d/0x9e0
[123733.315583]  schedule+0x5a/0xd0
[123733.315586]  schedule_preempt_disabled+0x11/0x20
[123733.315589]  __mutex_lock.constprop.0+0x399/0x700
[123733.315594]  io_sq_thread+0x26e/0x4e0
[123733.315600]  ? cpuusage_read+0x10/0x10
[123733.315604]  ? io_sqd_handle_event+0xd0/0xd0
[123733.315607]  ret_from_fork+0x22/0x30
[123733.315613]  </TASK>

We have observed that 3 times on amd64, and once on arm64, so it's not a
random issue, although so far we only observed this problem when
building chromium. There was no such issue with kernel 6.1.112-1.

Looking at the queued patches for 6.1.118, I noticed that this one
is a possible candidate for fixing the issue:
https://patchwork.kernel.org/project/io-uring/patch/20240604130527.3597-1-hagarhem@amazon.com/

That said I haven't tested it.

Regards
Aurelien

[toc] | [next] | [standalone]

#84590

[Multipart message — attachments visible in raw view] — view raw

Hi,

On 2024-11-18 23:23, Andres Salomon wrote:
> On Fri, 15 Nov 2024 22:31:22 +0100 Aurelien Jarno <aurel32@debian.org>
> wrote:
> > Source: linux
> > Version: 6.1.115-1
> > Severity: important
> > Tags: upstream
> > X-Debbugs-Cc: dsa@debian.org
> > 
> > Dear maintainers,
> > 
> > Since the last Bookworm point release, the build daemons often run into
> > a io_uring deadlock when building chromium:
> 
> 
> Does it happen at the same place of the chromium build, or is it randomly
> during the build?  I'm trying to reproduce this over here (on my bookworm
> build machine) and failing.

Yes the logs ends up in the same location for the all x86 builds:

[36013/36026] ACTION //chrome:packed_resources_locales_uk(//build/toolchain/linux/unbundle:default)
[36014/36026] ACTION //chrome:packed_resources_locales_vi(//build/toolchain/linux/unbundle:default)
[36015/36026] ACTION //chrome:packed_resources_locales_tr(//build/toolchain/linux/unbundle:default)
[36016/36026] ACTION //chrome:packed_resources_locales_ur(//build/toolchain/linux/unbundle:default)
[36017/36026] ACTION //chrome:packed_resources_locales_zh-TW(//build/toolchain/linux/unbundle:default)
[36018/36026] ACTION //chrome:packed_resources_locales_zh-CN(//build/toolchain/linux/unbundle:default)

The arm64 one ends at:
[51547/52050] CXX obj/services/screen_ai/screen_ai/screen_ai_service_impl.o
[51548/52050] CXX obj/ui/linux/display_server_utils/display_server_utils.o
[51549/52050] CXX obj/services/screen_ai/proto/proto_convertors/visual_annotator_proto_convertor.o
[51550/52050] LINK ./chromedriver.unstripped
[51551/52050] ACTION //chrome/test/chromedriver:chromedriver_server(//build/toolchain/linux/unbundle:default)
[51552/52050] LINK ./chrome_sandbox

Regards
Aurelien

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                     http://aurel32.net

[toc] | [prev] | [next] | [standalone]

#84591 — Bug#1087602: linux: io_uring deadlock...

Hi,

On Tue, Nov 19, 2024 at 11:50:43AM +0000, Mikko Nahkola (Trivore) wrote:
> Hi,
> 
> This looks a lot similar to what we saw here, on Gitlab runner Docker hosts,
> building various internal things.
> 
> The build jobs would *sometimes* have the containerized runner deadlock on
> io_uring, timeout, and the build fail. Then another build would start a
> fresh container, leaving the old one still hanging. Almost like old-school
> NFS unkillable deadlocks - these too were unable to receive even a SIGKILL,
> and the system "load" numbers gradually increased.
> 
> This only happened on linux-image-6.1.0-27 (Built-Using: linux (=
> 6.1.115-1). No such problem on 6.1.0-26.
> 
> We currently no longer have a runner host on 6.1.0-27.
> 
> Would this be fixed by commit 9e8debb8e51354b201db494689198078ec2c1e75 in
> upstream https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.116 ?

If you are able to easily reproduce the issue, would you be able to
test the test packages at

https://people.debian.org/~carnil/tmp/linux/1086447/

which do rebae to 6.1.118?

Regards,
Salvatore

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web