Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #84499 > unrolled thread

Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start

Back to article view | Back to linux.debian.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start Uwe Kleine-König <ukleinek@debian.org> - 2024-11-11 11:30 +0100
    Processed: Re: Bug#1086520: linux-image-6.11.2-amd64: makes  opensm fail to start "Debian Bug Tracking System" <owner@bugs.debian.org> - 2024-11-11 11:30 +0100
    Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start Francesco Poli <invernomuto@paranoici.org> - 2024-11-13 23:20 +0100
      Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start Uwe Kleine-König <ukleinek@debian.org> - 2024-11-18 10:10 +0100
        Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start Francesco Poli <invernomuto@paranoici.org> - 2024-11-18 20:10 +0100
          Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start Uwe Kleine-König <ukleinek@debian.org> - 2024-11-21 11:10 +0100
            Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start Francesco Poli <invernomuto@paranoici.org> - 2024-11-25 20:00 +0100
              Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start Francesco Poli <invernomuto@paranoici.org> - 2024-11-26 08:30 +0100
              Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start Francesco Poli <invernomuto@paranoici.org> - 2024-11-27 19:00 +0100

#84499 — Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start

[Multipart message — attachments visible in raw view] — view raw

Control: tag -1 + moreinfo
Control: forwarded -1 https://github.com/linux-rdma/opensm/issues/37

Hello,

On Thu, Oct 31, 2024 at 07:53:52PM +0100, Francesco Poli (wintermute) wrote:
> Package: src:linux
> Version: 6.11.2-1
> Severity: important
> X-Debbugs-Cc: invernomuto@paranoici.org
> 
> Hello,
> I encountered a major issue on an HPC cluster head node, as soon as
> I upgraded the Linux kernel from version 6.10.11-1 to version 6.11.2-1 .
> 
> The issue is that the head node runs OpenSM (InfiniBand subnet manager),
> which is needed for the Infiniband network to work.
> As soon as I reboot the head node with kernel 6.11.2-1 (or 6.11.4-1),
> OpenSM fails to start.
> If I reboot with the previous kernel version 6.10.11-1, everything
> works fine.
> 
> The symptoms are described in bug [#1085300], filed against package
> opensm.
> 
> [#1085300]: <https://bugs.debian.org/1085300>
> 
> Now I am not sure what's going on.
> 
> Is there any important change in the Linux kernel that OpenSM needs
> to adapt for?
> Or is this a bug in the newer Linux kernel version (that needs to
> be fixed there)?
> 
> I filed this bug report against the Debian Linux kernel, in order
> to warn other users about this issue, and in order to ask the Debian
> Kernel Team to investigate the issue and/or to forward the bug report
> to the relevant upstream Linux kernel maintainers.
> 
> Please do not reassign to package opensm with the intention of
> merging with bug [#1085300], unless you know for sure that the
> issue is in opensm and you know how to fix it.

Please do not report multiple bugs for the same issue. The right(er)
thing to do is to make use of "affects". Now there are three bug reports
(2 for Debian and one upstream) and someone being aware of only one (or
two) of them, might miss some action which results in duplicate work.
 
> Please help, I would very much like to run the head node with
> an up-to-date kernel!

This is hard to act on without further input. Some questions to debug
this:

I guess the kernel provides a directory "/sys/class/infiniband_mad". Do
its contents look different on 6.10.x and 6.11.x?

Can you please bisect the problem? There are a few kernel versions that
were packaged for Debian (i.e. 6.11-1~exp1, 6.11~rc5-1~exp1,
6.11~rc4-1~exp1, 6.10.12-1). I would expect that 6.11~rc4-1~exp1 is the
oldest failing one. It would be great if you could bisect this further.
Something like the following on the working kernel:

	git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
	cd linux
	git checkout v6.10
	cp /boot/config-6.10.11-1-$(uname -r) .config
	make localmodconfig
	cp .config arch/x86/configs/my_defconfig
	make bindeb-pkg

This creates a debian kernel package that you can test. I would hope
this one to be "good".

(The following steps don't need to be done on the working kernel, this
is only critical for the localmodconfig step above.)

Then test 6.11:

	git checkout v6.11
	make my_defconfig bindeb-pkg

I would expect this one to produce a broken kernel package.

If you can confirm that (i.e. vanilla 6.10 works and 6.11 doesn't), do
the actual bisection:

	git bisect start v6.11 v6.10

and in each to-be-tested version do:

	make my_defconfig bindeb-pkg

and test the resulting kernel package. Depending on if that is good or
bad do:

	git bisect good

or

	git bisect bad

Note you don't need to test the versions that are suggested there. To
speed up, it might be beneficial to test v6.11-rc1~117 and v6.11-rc1~116
first. To do so just don't test the version that git-bisect proposes but
do:

	git checkout v6.11-rc1~117
	make my_defconfig bindeb-pkg
	... test ...
	git bisect ...

and then the same for v6.11-rc1~116.

Then report back the found first bad commit. If you have difficulties
following this instruction, feel free to contact me, e.g. in the
#debian-kernel irc channel.

Best regards
Uwe

[toc] | [next] | [standalone]

#84502 — Processed: Re: Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start

Processing control commands:

> tag -1 + moreinfo
Bug #1086520 [src:linux] linux-image-6.11.2-amd64: makes opensm fail to start
Added tag(s) moreinfo.
> forwarded -1 https://github.com/linux-rdma/opensm/issues/37
Bug #1086520 [src:linux] linux-image-6.11.2-amd64: makes opensm fail to start
Set Bug forwarded-to-address to 'https://github.com/linux-rdma/opensm/issues/37'.

-- 
1086520: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1086520
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [next] | [standalone]

#84529

[Multipart message — attachments visible in raw view] — view raw

On Mon, 11 Nov 2024 11:22:26 +0100 Uwe Kleine-König wrote:

[...]
> Hello,

Hi Uwe, thanks for your followup.

> 
> On Thu, Oct 31, 2024 at 07:53:52PM +0100, Francesco Poli (wintermute) wrote:
[...]
> > I filed this bug report against the Debian Linux kernel, in order
> > to warn other users about this issue, and in order to ask the Debian
> > Kernel Team to investigate the issue and/or to forward the bug report
> > to the relevant upstream Linux kernel maintainers.
> > 
> > Please do not reassign to package opensm with the intention of
> > merging with bug [#1085300], unless you know for sure that the
> > issue is in opensm and you know how to fix it.
> 
> Please do not report multiple bugs for the same issue. The right(er)
> thing to do is to make use of "affects". Now there are three bug reports
> (2 for Debian and one upstream) and someone being aware of only one (or
> two) of them, might miss some action which results in duplicate work.

You are right, the "affects" field is the most appropriate means to
show that a bug report against a given package also affects other
packages.

However, in this case, the lack of replies from opensm maintainers made
me doubtful about the best possible course of action. Sorry about that.

>  
> > Please help, I would very much like to run the head node with
> > an up-to-date kernel!
> 
> This is hard to act on without further input. Some questions to debug
> this:
> 
> I guess the kernel provides a directory "/sys/class/infiniband_mad". Do
> its contents look different on 6.10.x and 6.11.x?

I will look into this as soon as I can reboot the cluster head node.

> 
> Can you please bisect the problem?
[...]

I have to find a time window where I can perform multiple reboots,
which can result in a non-working InfiniBand network... It won't be
easy, since the cluster has entered production and users keep launching
jobs.

Anyway, what I have done so far is: I have tried and rebuilt a Linux
kernel image Debian package, following your instructions.
After some failed attempts (due to missing dependencies and/or required
tools), I think I succeeded, but I had to reply to a number of
questions during the procedure: I have always replied with the default
answer (by hitting [Enter]), I hope that was the right thing to do!

Before I go on and try to install the resulting Debian package, could
you please review the transcript of what I did (see the attached file)?

Please bear with me, some of the questions were really obscure to me
and I am not really familiar with the procedure: I think that the last
time I rebuilt a Linux kernel image Debian package was some 15 years
ago (I was still using the now-obsolete [kernel-package]!).

[kernel-package]: <https://tracker.debian.org/pkg/kernel-package>

Thanks for your time and for the help you are providing.


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE

[toc] | [prev] | [next] | [standalone]

#84573

[Multipart message — attachments visible in raw view] — view raw

Hello Francesco,

On Wed, Nov 13, 2024 at 11:15:03PM +0100, Francesco Poli wrote:
> On Mon, 11 Nov 2024 11:22:26 +0100 Uwe Kleine-König wrote:
> 
> [...]
> > Hello,
> 
> Hi Uwe, thanks for your followup.
> 
> > 
> > On Thu, Oct 31, 2024 at 07:53:52PM +0100, Francesco Poli (wintermute) wrote:
> [...]
> > > I filed this bug report against the Debian Linux kernel, in order
> > > to warn other users about this issue, and in order to ask the Debian
> > > Kernel Team to investigate the issue and/or to forward the bug report
> > > to the relevant upstream Linux kernel maintainers.
> > > 
> > > Please do not reassign to package opensm with the intention of
> > > merging with bug [#1085300], unless you know for sure that the
> > > issue is in opensm and you know how to fix it.
> > 
> > Please do not report multiple bugs for the same issue. The right(er)
> > thing to do is to make use of "affects". Now there are three bug reports
> > (2 for Debian and one upstream) and someone being aware of only one (or
> > two) of them, might miss some action which results in duplicate work.
> 
> You are right, the "affects" field is the most appropriate means to
> show that a bug report against a given package also affects other
> packages.
> 
> However, in this case, the lack of replies from opensm maintainers made
> me doubtful about the best possible course of action. Sorry about that.
> 
> >  
> > > Please help, I would very much like to run the head node with
> > > an up-to-date kernel!
> > 
> > This is hard to act on without further input. Some questions to debug
> > this:
> > 
> > I guess the kernel provides a directory "/sys/class/infiniband_mad". Do
> > its contents look different on 6.10.x and 6.11.x?
> 
> I will look into this as soon as I can reboot the cluster head node.
> 
> > 
> > Can you please bisect the problem?
> [...]
> 
> I have to find a time window where I can perform multiple reboots,
> which can result in a non-working InfiniBand network... It won't be
> easy, since the cluster has entered production and users keep launching
> jobs.
> 
> Anyway, what I have done so far is: I have tried and rebuilt a Linux
> kernel image Debian package, following your instructions.
> After some failed attempts (due to missing dependencies and/or required
> tools), I think I succeeded, but I had to reply to a number of
> questions during the procedure: I have always replied with the default
> answer (by hitting [Enter]), I hope that was the right thing to do!

Yes, that sounds right. I wouldn't have expected that questions are
asked, but that's a problem with my expectations, not your following of
my instructions.

	yes '' | make localmodconfig

would be the better recommendation it seems.

> Before I go on and try to install the resulting Debian package, could
> you please review the transcript of what I did (see the attached file)?

Looks good. Probably the individual answers don't matter much and the
default should be fine. Just continue with my instructions and if the
resulting kernels boots and behave as the respective versions packaged
by Debian, everything is fine. Iff that fails, a more detailed review is
needed.

Best regards
Uwe

[toc] | [prev] | [next] | [standalone]

#84584

[Multipart message — attachments visible in raw view] — view raw

On Mon, 18 Nov 2024 09:58:03 +0100 Uwe Kleine-König wrote:

[...]
> On Wed, Nov 13, 2024 at 11:15:03PM +0100, Francesco Poli wrote:
> > On Mon, 11 Nov 2024 11:22:26 +0100 Uwe Kleine-König wrote:
[...]
> > > I guess the kernel provides a directory "/sys/class/infiniband_mad". Do
> > > its contents look different on 6.10.x and 6.11.x?
> > 
> > I will look into this as soon as I can reboot the cluster head node.

I looked into this, while testing the new Debian Linux kernel that has just migrated to testing (which, once again, makes opensm fail to start, just like other 6.11.x versions).

With a working kernel:

  $ uname -v
  #1 SMP PREEMPT_DYNAMIC Debian 6.10.11-1 (2024-09-22)
  $ ls -altrF /sys/class/infiniband_mad/
  total 0
  lrwxrwxrwx  1 root root    0 Nov  4 15:58 umad0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/umad0/
  lrwxrwxrwx  1 root root    0 Nov  4 15:58 umad1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/umad1/
  lrwxrwxrwx  1 root root    0 Nov 11 15:54 issm1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/issm1/
  lrwxrwxrwx  1 root root    0 Nov 11 15:54 issm0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/issm0/
  drwxr-xr-x  2 root root    0 Nov 11 15:54 ./
  drwxr-xr-x 72 root root    0 Nov 11 15:54 ../
  -r--r--r--  1 root root 4096 Nov 11 15:54 abi_version
  $ cat /sys/class/infiniband_mad/abi_version 
  5

With a kernel that makes opensm fail to start:

  $ uname -v
  #1 SMP PREEMPT_DYNAMIC Debian 6.11.7-1 (2024-11-09)
  $ ls -altrF /sys/class/infiniband_mad/
  total 0
  drwxr-xr-x 73 root root    0 Nov 18 09:41 ../
  -r--r--r--  1 root root 4096 Nov 18 09:41 abi_version
  lrwxrwxrwx  1 root root    0 Nov 18 09:41 umad0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/umad0/
  lrwxrwxrwx  1 root root    0 Nov 18 09:41 umad1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/umad1/
  drwxr-xr-x  2 root root    0 Nov 18 09:43 ./
  $ cat /sys/class/infiniband_mad/abi_version
  5

As you can see, a couple of files (symlinks) are missing here...

Does this ring a bell?
Can you tell what's wrong, by just looking at this?
Or, at least, do you get some less vague idea of what's going on?

[...]
> > Before I go on and try to install the resulting Debian package, could
> > you please review the transcript of what I did (see the attached file)?
> 
> Looks good. Probably the individual answers don't matter much and the
> default should be fine. Just continue with my instructions and if the
> resulting kernels boots and behave as the respective versions packaged
> by Debian, everything is fine. Iff that fails, a more detailed review is
> needed.

Thanks for confirming, I really hope I can find a time window, where I
can bisect...


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE

[toc] | [prev] | [next] | [standalone]

#84610

[Multipart message — attachments visible in raw view] — view raw

Hello Francesco,

[for the new-comers: This is about a regression in 6.11. Details
available at https://bugs.debian.org/1086520. The TL;DR; is that on
6.10.11 opensm works as expected, while it fails to start on 6.11.7.]

On Mon, Nov 18, 2024 at 08:06:16PM +0100, Francesco Poli wrote:
> On Mon, 18 Nov 2024 09:58:03 +0100 Uwe Kleine-König wrote:
> 
> [...]
> > On Wed, Nov 13, 2024 at 11:15:03PM +0100, Francesco Poli wrote:
> > > On Mon, 11 Nov 2024 11:22:26 +0100 Uwe Kleine-König wrote:
> [...]
> > > > I guess the kernel provides a directory "/sys/class/infiniband_mad". Do
> > > > its contents look different on 6.10.x and 6.11.x?
> > > 
> > > I will look into this as soon as I can reboot the cluster head node.
> 
> I looked into this, while testing the new Debian Linux kernel that has
> just migrated to testing (which, once again, makes opensm fail to
> start, just like other 6.11.x versions).
> 
> With a working kernel:
> 
>   $ uname -v
>   #1 SMP PREEMPT_DYNAMIC Debian 6.10.11-1 (2024-09-22)
>   $ ls -altrF /sys/class/infiniband_mad/
>   total 0
>   lrwxrwxrwx  1 root root    0 Nov  4 15:58 umad0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/umad0/
>   lrwxrwxrwx  1 root root    0 Nov  4 15:58 umad1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/umad1/
>   lrwxrwxrwx  1 root root    0 Nov 11 15:54 issm1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/issm1/
>   lrwxrwxrwx  1 root root    0 Nov 11 15:54 issm0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/issm0/
>   drwxr-xr-x  2 root root    0 Nov 11 15:54 ./
>   drwxr-xr-x 72 root root    0 Nov 11 15:54 ../
>   -r--r--r--  1 root root 4096 Nov 11 15:54 abi_version
>   $ cat /sys/class/infiniband_mad/abi_version 
>   5
> 
> With a kernel that makes opensm fail to start:
> 
>   $ uname -v
>   #1 SMP PREEMPT_DYNAMIC Debian 6.11.7-1 (2024-11-09)
>   $ ls -altrF /sys/class/infiniband_mad/
>   total 0
>   drwxr-xr-x 73 root root    0 Nov 18 09:41 ../
>   -r--r--r--  1 root root 4096 Nov 18 09:41 abi_version
>   lrwxrwxrwx  1 root root    0 Nov 18 09:41 umad0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/umad0/
>   lrwxrwxrwx  1 root root    0 Nov 18 09:41 umad1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/umad1/
>   drwxr-xr-x  2 root root    0 Nov 18 09:43 ./
>   $ cat /sys/class/infiniband_mad/abi_version
>   5
> 
> As you can see, a couple of files (symlinks) are missing here...

It looks like the commit that is biting you is

https://git.kernel.org/linus/50660c5197f52b8137e223dc3ba8d43661179a1d

So if you bisect, try 50660c5197f52b8137e223dc3ba8d43661179a1d and its
parent 24943dcdc156cf294d97a36bf5c51168bf574c22 first.

I don't know about infiniband, but I'd say: Either your machine doesn't
have these issmX devices and opensm should cope with that, or these
issmX devices are available then
50660c5197f52b8137e223dc3ba8d43661179a1d is buggy.

> Does this ring a bell?

It doesn't for me, but maybe Mark Zhang or someone else among the new
recipients has an idea?

Best regards
Uwe

[toc] | [prev] | [next] | [standalone]

#84674

[Multipart message — attachments visible in raw view] — view raw

On Thu, 21 Nov 2024 11:04:13 +0100 Uwe Kleine-König wrote:

[...]
> It looks like the commit that is biting you is
> 
> https://git.kernel.org/linus/50660c5197f52b8137e223dc3ba8d43661179a1d
> 
> So if you bisect, try 50660c5197f52b8137e223dc3ba8d43661179a1d and its
> parent 24943dcdc156cf294d97a36bf5c51168bf574c22 first.

I started to bisect.

The first surprise is that 50660c5197f52b8137e223dc3ba8d43661179a1d is
good...   :-o

  $ git checkout 50660c5197f52b8137e223dc3ba8d43661179a1d
  $ make -j 12 my_defconfig bindeb-pkg

  [install and reboot with this kernel version]

  # ls /sys/class/infiniband_mad/ -altrF
  total 0
  drwxr-xr-x 70 root root    0 Nov 25 12:05 ../
  -r--r--r--  1 root root 4096 Nov 25 12:05 abi_version
  lrwxrwxrwx  1 root root    0 Nov 25 12:05 umad0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/umad0/
  lrwxrwxrwx  1 root root    0 Nov 25 12:05 umad1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/umad1/
  lrwxrwxrwx  1 root root    0 Nov 25 12:08 issm1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/issm1/
  lrwxrwxrwx  1 root root    0 Nov 25 12:08 issm0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/issm0/
  drwxr-xr-x  2 root root    0 Nov 25 12:08 ./

  [InfiniBand works]

  $ git bisect start
  $ git bisect good
  $ git checkout v6.11
  $ make -j 12 my_defconfig bindeb-pkg

  [install and reboot with this kernel version]

  # ls /sys/class/infiniband_mad/ -altrF
  total 0
  drwxr-xr-x 70 root root    0 Nov 25 12:29 ../
  -r--r--r--  1 root root 4096 Nov 25 12:29 abi_version
  lrwxrwxrwx  1 root root    0 Nov 25 12:29 umad0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/umad0/
  lrwxrwxrwx  1 root root    0 Nov 25 12:29 umad1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/umad1/
  drwxr-xr-x  2 root root    0 Nov 25 12:30 ./

  [InfiniBand fails, because OpenSM fails to start]

  $ git bisect bad
  Bisecting: 7036 revisions left to test after this (roughly 13 steps)
  [b3ce7a30847a54a7f96a35e609303d8afecd460b] Merge tag 'drm-next-2024-07-18' of https://gitlab.freedesktop.org/drm/kernel
  $ make -j 12 my_defconfig bindeb-pkg


Woooha, 13 steps are a lot...

I went on until 10 steps are left:

  [test b3ce7a30847a54a7f96a35e609303d8afecd460b]
  $ git bisect good
  Bisecting: 3385 revisions left to test after this (roughly 12 steps)
  [fbc90c042cd1dc7258ebfebe6d226017e5b5ac8c] Merge tag 'mm-stable-2024-07-21-14-50' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
  
  [test fbc90c042cd1dc7258ebfebe6d226017e5b5ac8c]
  $ git bisect bad
  Bisecting: 1763 revisions left to test after this (roughly 11 steps)
  [09ea8089abb5d851ce08a9b1a43706e42ef39db2] Merge tag 'staging-6.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging

  [test 09ea8089abb5d851ce08a9b1a43706e42ef39db2]
  $ git bisect bad
  Bisecting: 910 revisions left to test after this (roughly 10 steps)
  [4305ca0087dd99c3c3e0e2ac8a228b7e53a21c78] Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi


Since I could not afford to keep the cluster out of service any longer
(each step takes at least 20 or 25 minutes: build + install + reboot +
check InfiniBand), I decided to return the cluster to service.

I will try to continue to bisect by testing the resulting kernels on a
compute node: there's no OpenSM there and it cannot run anyway, if
there's another OpenSM on the same InfiniBand network.
However, I can check whether those issm* symlinks are created in
/sys/class/infiniband_mad/ 
I really hope that this is enough to pinpoint the first bad
commit...

Any better ideas?


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE

[toc] | [prev] | [next] | [standalone]

#84677

[Multipart message — attachments visible in raw view] — view raw

On Tue, 26 Nov 2024 09:21:37 +0800 Mark Zhang wrote:

[...]
> Yes looks like FW reports vport.num_plane > 0. What is your hw type and 
> FW version ("ethtool -i <netdev_of_the_ibdev>")? I don't think it 
> supports multiplane.

  $ /sbin/ethtool -i ibp129s0f0
  driver: mlx5_core[ib_ipoib]
  version: 6.10.11-amd64
  firmware-version: 20.40.1000 (MT_0000000224)
  expansion-rom-version: 
  bus-info: 0000:81:00.0
  supports-statistics: yes
  supports-test: yes
  supports-eeprom-access: no
  supports-register-dump: no
  supports-priv-flags: yes

Please note that I determined <netdev_of_the_ibdev> by looking at
the output of 'ibv_devices': I hope this is a correct way to answer
your question.




-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE

[toc] | [prev] | [next] | [standalone]

#84684

[Multipart message — attachments visible in raw view] — view raw

On Mon, 25 Nov 2024 21:38:37 +0200 Leon Romanovsky wrote:

> On Mon, Nov 25, 2024 at 07:54:43PM +0100, Francesco Poli wrote:
[...]
> > I will try to continue to bisect by testing the resulting kernels on a
> > compute node: there's no OpenSM there and it cannot run anyway, if
> > there's another OpenSM on the same InfiniBand network.
> > However, I can check whether those issm* symlinks are created in
> > /sys/class/infiniband_mad/ 
> > I really hope that this is enough to pinpoint the first bad
> > commit...
> 
> Yes, these symlinks should be there. Your test scenario is correct one.

OK, I have completed the bisect on a compute node without OpenSM, by
looking at the issm* symlinks, as I said.

See below.

> 
> > 
> > Any better ideas?
> 
> I think that commit: 2a5db20fa532 ("RDMA/mlx5: Add support to multi-plane device and port")
> is the one which is causing to troubles, which leads me to suspect FW.
[...]

Thanks to your guess about the possibly troublesome commit, the bisect was completed in a few steps:

  $ git checkout 2a5db20fa532
  $ make -j 12 my_defconfig bindeb-pkg
  
  [install this version on a compute node test image and reboot
  one compute node with that image: the InfiniBand network was
  working for that node, that's no surprise, since OpenSM was running
  on the head node, but no issm* symlink was created; please note
  that, surprisingly, the Ethernet network was not working, I mean
  that the Ethernet interfaces were not found by the kernel...]
  
  root@node # ls -altrF /sys/class/infiniband_mad/
  total 0
  drwxr-xr-x 60 root root    0 Nov 26 17:06 ../
  lrwxrwxrwx  1 root root    0 Nov 26 17:06 umad0 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/infiniband_mad/umad0/
  -r--r--r--  1 root root 4096 Nov 26 17:06 abi_version
  lrwxrwxrwx  1 root root    0 Nov 26 17:06 umad1 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.1/infiniband_mad/umad1/
  drwxr-xr-x  2 root root    0 Nov 26 17:08 ./
  
  $ git bisect bad
  Bisecting: 0 revisions left to test after this (roughly 0 steps)
  [65528cfb21fdb68de8ae6dccae19af180d93e143] net/mlx5: mlx5_ifc update for multi-plane support
  $ make -j 12 my_defconfig bindeb-pkg
  
  [install this version on the compute node test image and reboot
  one compute node with that image: the InfiniBand network again
  working for that node, issm* symlinks were created;
  Ethernet network again not working for that node...]
  
  root@node # ls -altrF /sys/class/infiniband_mad/
  total 0
  drwxr-xr-x 60 root root    0 Nov 26 17:31 ../
  lrwxrwxrwx  1 root root    0 Nov 26 17:31 umad0 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/infiniband_mad/umad0/
  -r--r--r--  1 root root 4096 Nov 26 17:31 abi_version
  lrwxrwxrwx  1 root root    0 Nov 26 17:31 umad1 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.1/infiniband_mad/umad1/
  lrwxrwxrwx  1 root root    0 Nov 26 17:36 issm1 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.1/infiniband_mad/issm1/
  lrwxrwxrwx  1 root root    0 Nov 26 17:36 issm0 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/infiniband_mad/issm0/
  drwxr-xr-x  2 root root    0 Nov 26 17:36 ./
  
  $ git bisect good
  2a5db20fa532198639671713c6213f96ff285b85 is the first bad commit
  commit 2a5db20fa532198639671713c6213f96ff285b85
  Author: Mark Zhang <markzhang@nvidia.com>
  Date:   Sun Jun 16 19:08:35 2024 +0300
  
      RDMA/mlx5: Add support to multi-plane device and port
  
      When multi-plane is supported, a logical port, which is aggregation of
      multiple physical plane ports, is exposed for data transmission.
      Compared with a normal mlx5 IB port, this logical port supports all
      functionalities except Subnet Management.
  
      Signed-off-by: Mark Zhang <markzhang@nvidia.com>
      Link: https://lore.kernel.org/r/7e37c06c9cb243be9ac79930cd17053903785b95.1718553901.git.leon@kernel.org
      Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
  
   drivers/infiniband/hw/mlx5/main.c               | 60 +++++++++++++++++++++----
   drivers/infiniband/hw/mlx5/mlx5_ib.h            |  2 +
   drivers/net/ethernet/mellanox/mlx5/core/vport.c |  1 +
   include/linux/mlx5/driver.h                     |  1 +
   4 files changed, 55 insertions(+), 9 deletions(-)


In other words, bingo!, your guess looks correct, the first bad commit
is the one you mentioned.


Now, I will try to upgrade the firmware of the InfiniBand NICs, as you
suggested, and check whether this solves the issue with the recent
Linux kernel versions.

Please confirm that the procedure to be followed is the one described in
<https://docs.nvidia.com/networking/display/ubuntu2204/firmware+burning>

Thanks for your time and patience, and for all the help you are kindly
providing!   :-)


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web