Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #74458 > unrolled thread

Bug#1001001: closed by Salvatore Bonaccorso <carnil@debian.org> (Re: linux-image-5.10.0-9-arm64: kernel BUG at include/linux/swapops.h:204!)

Back to article view | Back to linux.debian.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#1001001: closed by Salvatore Bonaccorso <carnil@debian.org> (Re: linux-image-5.10.0-9-arm64: kernel BUG at include/linux/swapops.h:204!) Paul Gevers <elbrus@debian.org> - 2022-02-20 13:50 +0100
    Bug#1001001: closed by Salvatore Bonaccorso <carnil@debian.org> (Re: linux-image-5.10.0-9-arm64: kernel BUG at include/linux/swapops.h:204!) Paul Gevers <elbrus@debian.org> - 2022-03-29 21:10 +0200
      Bug#1001001: linux-image-5.10.0-9-arm64: kernel BUG at include/linux/swapops.h:204! Diederik de Haas <didi.debian@cknow.org> - 2022-06-21 22:10 +0200
        Bug#1001001: linux-image-5.10.0-9-arm64: kernel BUG at include/linux/swapops.h:204! Paul Gevers <elbrus@debian.org> - 2022-06-21 22:40 +0200
          Bug#1001001: linux-image-5.10.0-9-arm64: kernel BUG at include/linux/swapops.h:204! Paul Gevers <elbrus@debian.org> - 2022-06-22 22:10 +0200
            Bug#1001001: linux-image-5.10.0-9-arm64: kernel BUG at include/linux/swapops.h:204! Diederik de Haas <didi.debian@cknow.org> - 2022-06-22 23:20 +0200
              Bug#1001001: linux-image-5.10.0-9-arm64: kernel BUG at include/linux/swapops.h:204! Diederik de Haas <didi.debian@cknow.org> - 2022-06-22 23:30 +0200
              Bug#1001001: linux-image-5.10.0-9-arm64: kernel BUG at include/linux/swapops.h:204! Paul Gevers <elbrus@debian.org> - 2022-06-23 10:50 +0200

#74458 — Bug#1001001: closed by Salvatore Bonaccorso <carnil@debian.org> (Re: linux-image-5.10.0-9-arm64: kernel BUG at include/linux/swapops.h:204!)

[Multipart message — attachments visible in raw view] — view raw

Control: reopen -1
Control: found -1 5.10.84-1

Hi,

Sad to say, but this week we had two hangs again.

The one on ci-worker-arm64-06 had this:

Feb 15 08:51:12 ci-worker-arm64-06 kernel: kernel BUG at 
include/linux/swapops.h:204!
Feb 15 08:51:12 ci-worker-arm64-06 kernel: Internal error: Oops - BUG: 0 
[#1] SMP

root@ci-worker-arm64-06:~# uname -a
Linux ci-worker-arm64-06 5.10.0-10-arm64 #1 SMP Debian 5.10.84-1 
(2021-12-08) aarch64 GNU/Linux

I'm upgrading the workers to the latest kernel now.

Paul

[toc] | [next] | [standalone]

#74903

[Multipart message — attachments visible in raw view] — view raw

Hi all,

On 20-02-2022 13:44, Paul Gevers wrote:
> Sad to say, but this week we had two hangs again.

And this week another two.

============ ci-worker-arm64-07 ==============

Mar 26 10:15:55 ci-worker-arm64-07 kernel: kernel BUG at 
include/linux/swapops.h:204!
Mar 26 10:15:55 ci-worker-arm64-07 kernel: Internal error: Oops - BUG: 0 
[#1] SMP

Linux kernel from before the last point release:
Linux version 5.10.0-12-arm64 (debian-kernel@lists.debian.org) (gcc-10 
(Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2>

============ ci-worker-arm64-08 ==============
Mar 25 22:13:44 ci-worker-arm64-08 kernel: kernel BUG at 
include/linux/swapops.h:204!
Mar 25 22:13:44 ci-worker-arm64-08 kernel: Internal error: Oops - BUG: 0 
[#1] SMP

Paul

[toc] | [prev] | [next] | [standalone]

#75715 — Bug#1001001: linux-image-5.10.0-9-arm64: kernel BUG at include/linux/swapops.h:204!

[Multipart message — attachments visible in raw view] — view raw

Control: found -1 linux/5.10.103-1

Hi Paul,

On Tuesday, 29 March 2022 20:58:59 CEST Paul Gevers wrote:
> On 20-02-2022 13:44, Paul Gevers wrote:
> 
> > Sad to say, but this week we had two hangs again.
> 
> And this week another two.
> 
> ============ ci-worker-arm64-07 ==============
> 
> Mar 26 10:15:55 ci-worker-arm64-07 kernel: kernel BUG at 
> include/linux/swapops.h:204!
> Mar 26 10:15:55 ci-worker-arm64-07 kernel: Internal error: Oops - BUG: 0 
> [#1] SMP
> 
> Linux kernel from before the last point release:
> Linux version 5.10.0-12-arm64 (debian-kernel@lists.debian.org) (gcc-10 
> (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2>
> 
> ============ ci-worker-arm64-08 ==============
> Mar 25 22:13:44 ci-worker-arm64-08 kernel: kernel BUG at 
> include/linux/swapops.h:204!
> Mar 25 22:13:44 ci-worker-arm64-08 kernel: Internal error: Oops - BUG: 0 
> [#1] SMP

Do these errors still occur? Still with 5.10.103-1 or a later one?
Is it only on arm64 machines? Or is this just an example which also occurs
on other arches?
Is it possible to try newer kernel versions from Stable-backports to see
whether the issue occurs there too?

If it still occurs, then the likely only way to get a possible resolve is 
reporting it to upstream. For 'swapops.h' that should be this:

~/dev/kernel.org/linux$ scripts/get_maintainer.pl include/linux/swapops.h
Andrew Morton <akpm@linux-foundation.org>
Peter Xu <peterx@redhat.com>
David Hildenbrand <david@redhat.com>
Alistair Popple <apopple@nvidia.com>
Miaohe Lin <linmiaohe@huawei.com>
Naoya Horiguchi <naoya.horiguchi@nec.com>
linux-kernel@vger.kernel.org (open list)

But I'm not sure that's the right list as it is from the include directory,
so the actual problem may be somewhere else.
But I guess it would be a good start?

Cheers,
  Diederik

[toc] | [prev] | [next] | [standalone]

#75716 — Bug#1001001: linux-image-5.10.0-9-arm64: kernel BUG at include/linux/swapops.h:204!

[Multipart message — attachments visible in raw view] — view raw

Hi Diederik,

On 21-06-2022 22:07, Diederik de Haas wrote:
> Do these errors still occur? Still with 5.10.103-1 or a later one?

The last occurrence of a machine hang I had is from 5 May 2022, but I'm 
not sure if I checked if it was this same issue. Normally our kernels 
are up-to-date, but I don't recall what we had at the time. We have 
recommissioned our arm64 hosts, so the install logs are lost by now.

> Is it only on arm64 machines? Or is this just an example which also occurs
> on other arches?

I'm pretty sure I haven't seen this on other arches, otherwise I'm sure 
I would have reported it to this bug.

> If it still occurs, then the likely only way to get a possible resolve is
> reporting it to upstream.

1.5 months is quite long for it to be gone, although, before that it was 
2.5 months.

Paul

[toc] | [prev] | [next] | [standalone]

#75730 — Bug#1001001: linux-image-5.10.0-9-arm64: kernel BUG at include/linux/swapops.h:204!

[Multipart message — attachments visible in raw view] — view raw

Hi Diederik,

On 21-06-2022 23:19, Diederik de Haas wrote:
> I think that the install logs aren't that important (anymore) as the issue/
> symptoms appear to be the same:
> - some swap action resulting in some failure
> - CPU gets stuck
> - watchdog triggers a reboot

If the reboot would actually happen/finish, I wouldn't have problems of 
the hanging host. The issues I spotted required a manual reboot (and 
that's why I spotted them).

> How is swap configured on these devices?

https://salsa.debian.org/ci-team/debian-ci-config/-/blob/master/cookbooks/basics/default.rb#L3 
until line 11

> Yeah, I _assumed_ as such, but assumptions can be dangerous ;-)

Total ACK.

> Normally I scroll (hard) by the hardware listings as that rarely says anything
> to me. And I did that before too, but just now I made an important discovery.
> 
> I *assumed* it was running on arm64 (native) hardware and was about to ask
> specifics about it and then I noticed this:
> Host bridge [0600]: Red Hat, Inc. QEMU PCIe Host bridge [1b36:0008]
> 
> Qemu. Quite likely unrelated, but a while back I had an issue with qemu in
> building arm64 images: https://bugs.debian.org/988174

hmm, OK, right (I forgot that I knew this).

> I think it would be useful to know which qemu version(s) were used.

Is there any way to know from inside the VM?

> If the issue does occur again, I think it would be useful to bring 'upstream'
> into the conversation. They likely can bring much more useful input into this
> then (f.e.) I could. Also, if upstream is made aware there is an issue (even
> infrequent), then they can make the most informed choice what to do with it.

Ack.

Paul

[toc] | [prev] | [next] | [standalone]

#75731 — Bug#1001001: linux-image-5.10.0-9-arm64: kernel BUG at include/linux/swapops.h:204!

[Multipart message — attachments visible in raw view] — view raw

Hi Paul,

On Wednesday, 22 June 2022 21:57:06 CEST Paul Gevers wrote:
> On 21-06-2022 23:19, Diederik de Haas wrote:
> 
> > I think that the install logs aren't that important (anymore) as the
> > issue/symptoms appear to be the same:
> > - some swap action resulting in some failure
> > - CPU gets stuck
> > - watchdog triggers a reboot
> 
> If the reboot would actually happen/finish, I wouldn't have problems of 
> the hanging host. The issues I spotted required a manual reboot (and 
> that's why I spotted them).

Hmm ...interesting. AFAIK that is a watchdog's task.
And I was certain I saw sth about it as I've seen (a yet to be reported) an 
issue related to watchdog myself, hence why I remembered it.

On Saturday, 4 December 2021 22:44:38 CEST Paul Gevers wrote:
> I noticed in the logs that *after* the reported kernel bug but before
> the actual hang, I see multiple instances of:
> watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [apt-get:2204621]
> and
> watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kcompactd0:40]
> on ci-worker-arm64-07.

And here is where I saw it. (My watchdog issue doesn't cause a hang btw)

> > How is swap configured on these devices?
> 
> https://salsa.debian.org/ci-team/debian-ci-config/-/blob/master/cookbooks/ba
> sics/default.rb#L3 until line 11

Not familiar with Ruby, but IIUC a swap file get created half the size of RAM.
I _think_ the swapon command isn't technically needed as it will be done on 
bootup through fstab, but shouldn't hurt either. Seems fine :)

> > I *assumed* it was running on arm64 (native) hardware and was about to
> > ask specifics about it and then I noticed this:
> > Host bridge [0600]: Red Hat, Inc. QEMU PCIe Host bridge [1b36:0008]
> > 
> > Qemu. Quite likely unrelated, but a while back I had an issue with qemu
> > in building arm64 images: https://bugs.debian.org/988174
> 
> hmm, OK, right (I forgot that I knew this).
> 
> > I think it would be useful to know which qemu version(s) were used.
> 
> Is there any way to know from inside the VM?

If you have access to the host, APT should be able to tell you.

Via sources.list.erb I found that "<%= node['debian_release'] %>-backports" 
gets enabled, which I assume results in Stable-backports.
It appears that various tools get installed (but I don't see qemu mentioned 
(explicitly), but I do see 'virt-what' and the package description seems to 
indicate it may be useful to figure out detail of the VM.

@mjt, I have two questions for you:
1) do you know if/how the qemu version can be queried from within the VM?
2) Are you aware of potential issues wrt hangs in arm64 VM created with Qemu?
Or IOW, could you take a look at this bug and can you give tips which could 
help in tracking down the cause and subsequently the solution?

TIA,
  Diederik

[toc] | [prev] | [next] | [standalone]

#75732 — Bug#1001001: linux-image-5.10.0-9-arm64: kernel BUG at include/linux/swapops.h:204!

[Multipart message — attachments visible in raw view] — view raw

On Wednesday, 22 June 2022 23:15:46 CEST Diederik de Haas wrote:
> Via sources.list.erb I found that "<%= node['debian_release'] %>-backports"
> gets enabled, which I assume results in Stable-backports.
> It appears that various tools get installed (but I don't see qemu mentioned
> (explicitly)), but I do see 'virt-what' and the package description seems to
> indicate it may be useful to figure out detail of the VM.

Forgot to add: Backports seems available but it needs to be explicitly 
specified to install packages from it, which _I_ didn't see, but I'm not 
familiar with your (build) systems.
I don't know if it's an option, but stable-bpo has 1:7.0+dfsg-2~bpo11+2

[toc] | [prev] | [next] | [standalone]

#75739 — Bug#1001001: linux-image-5.10.0-9-arm64: kernel BUG at include/linux/swapops.h:204!

[Multipart message — attachments visible in raw view] — view raw

Hi Diederik,

On 22-06-2022 23:15, Diederik de Haas wrote:
> Hmm ...interesting. AFAIK that is a watchdog's task.
> And I was certain I saw sth about it as I've seen (a yet to be reported) an
> issue related to watchdog myself, hence why I remembered it.
> 
> On Saturday, 4 December 2021 22:44:38 CEST Paul Gevers wrote:
>> I noticed in the logs that *after* the reported kernel bug but before
>> the actual hang, I see multiple instances of:
>> watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [apt-get:2204621]
>> and
>> watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kcompactd0:40]
>> on ci-worker-arm64-07.
> 
> And here is where I saw it. (My watchdog issue doesn't cause a hang btw)

That might be, but this doesn't result in a successful reboot (of the 
system, maybe you meant a reboot of the core?).

> If you have access to the host, APT should be able to tell you.

Depends on what you mean with "the host". Our VM (our host) is 
provisioned by Huawei (their host). I have access to our host.

root@ci-worker-arm64-02:~# apt list *qemu* --installed
Listing... Done
qemu-utils/stable-security,now 1:5.2+dfsg-11+deb11u2 arm64 
[installed,automatic]
N: There is 1 additional version. Please use the '-a' switch to see it

> Via sources.list.erb I found that "<	%= node['debian_release'] %>-backports"
> gets enabled, which I assume results in Stable-backports.

Correct, but currently we don't install anything from there.

> It appears that various tools get installed (but I don't see qemu mentioned
> (explicitly), but I do see 'virt-what' and the package description seems to
> indicate it may be useful to figure out detail of the VM.

root@ci-worker-arm64-02:~# virt-what
qemu
root@ci-worker-arm64-02:~# virt-what --version
1.19

Paul

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web