Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1304141 > unrolled thread

Re: System "hangs" on kernels > v4.4-rc1 - mpt2sas problem?

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: System "hangs" on kernels > v4.4-rc1 - mpt2sas problem? "Martin K. Petersen" <martin.petersen@oracle.com> - 2016-01-08 03:40 +0100
    RE: System "hangs" on kernels > v4.4-rc1 - mpt2sas problem? "Luck, Tony" <tony.luck@intel.com> - 2016-01-08 06:10 +0100
      Re: System "hangs" on kernels > v4.4-rc1 - mpt2sas problem? "Martin K. Petersen" <martin.petersen@oracle.com> - 2016-01-08 18:30 +0100
        RE: System "hangs" on kernels > v4.4-rc1 - mpt2sas problem? "Luck, Tony" <tony.luck@intel.com> - 2016-01-08 18:30 +0100
    RE: System "hangs" on kernels > v4.4-rc1 - mpt2sas problem? "Luck, Tony" <tony.luck@intel.com> - 2016-01-08 18:00 +0100
      Re: System "hangs" on kernels > v4.4-rc1 - mpt2sas problem? "Martin K. Petersen" <martin.petersen@oracle.com> - 2016-01-08 18:30 +0100
        RE: System "hangs" on kernels > v4.4-rc1 - mpt2sas problem? "Luck, Tony" <tony.luck@intel.com> - 2016-01-08 18:40 +0100
          Re: System "hangs" on kernels > v4.4-rc1 - mpt2sas problem? "Martin K. Petersen" <martin.petersen@oracle.com> - 2016-01-08 18:50 +0100
            RE: System "hangs" on kernels > v4.4-rc1 - mpt2sas problem? "Luck, Tony" <tony.luck@intel.com> - 2016-01-08 19:30 +0100
              Re: System "hangs" on kernels > v4.4-rc1 - mpt2sas problem? James Bottomley <James.Bottomley@HansenPartnership.com> - 2016-01-08 19:40 +0100
                RE: System "hangs" on kernels > v4.4-rc1 - mpt2sas problem? "Luck, Tony" <tony.luck@intel.com> - 2016-01-08 19:50 +0100
                  Re: System "hangs" on kernels > v4.4-rc1 - mpt2sas problem? James Bottomley <James.Bottomley@HansenPartnership.com> - 2016-01-08 20:00 +0100
                Re: System "hangs" on kernels > v4.4-rc1 - mpt2sas problem? "Martin K. Petersen" <martin.petersen@oracle.com> - 2016-01-08 19:50 +0100
                  RE: System "hangs" on kernels > v4.4-rc1 - mpt2sas problem? "Luck, Tony" <tony.luck@intel.com> - 2016-01-08 20:00 +0100

#1304141 — Re: System "hangs" on kernels > v4.4-rc1 - mpt2sas problem?

>>>>> "Tony" == Luck, Tony <tony.luck@intel.com> writes:

Tony,

Tony> This one has been hard for me to bisect. Partly because it just
Tony> takes so long to reboot the servers that have the problem, but
Tony> also because there may be more than one thing going on.  One thing
Tony> that seems pretty certain is that things are OK prior to:

We consolidated the mpt2sas and mpt3sas drivers in 4.4. I can see the
driver getting loaded (and failing to allocate memory for some reason)
in your first log but it's absent in the second.

Tony>     megaraid_sas: Increase timeout to 60 secs for abort frames
Tony> during shutdown

megaraid_sas is an entirely different beast from mpt[23]sas. Please let
us know which type of controller your disks are hanging off of.

-- 
Martin K. Petersen	Oracle Linux Engineering

[toc] | [next] | [standalone]

#1304179

Martin,

Thanks for looking at this.

> megaraid_sas is an entirely different beast from mpt[23]sas. Please let
> us know which type of controller your disks are hanging off of.

Can you tell from the "mpt" messages in the console log?  Here's
output of "dmesg | grep mpt" from a v4.3 boot:

[   17.021171] mpt2sas version 20.100.00.00 loaded
[   17.108039] mpt2sas0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (528247880 kB)
[   17.441605] mpt2sas0: MSI-X vectors supported: 8, no of cores: 192, max_msix_vectors: 8
[   17.443653] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 42
[   17.443654] mpt2sas0-msix1: PCI-MSI-X enabled: IRQ 43
[   17.443655] mpt2sas0-msix2: PCI-MSI-X enabled: IRQ 44
[   17.443655] mpt2sas0-msix3: PCI-MSI-X enabled: IRQ 45
[   17.443656] mpt2sas0-msix4: PCI-MSI-X enabled: IRQ 46
[   17.443656] mpt2sas0-msix5: PCI-MSI-X enabled: IRQ 47
[   17.443657] mpt2sas0-msix6: PCI-MSI-X enabled: IRQ 48
[   17.443657] mpt2sas0-msix7: PCI-MSI-X enabled: IRQ 49
[   17.443659] mpt2sas0: iomem(0x0000000093340000), mapped(0xffffc90032b80000), size(65536)
[   17.443660] mpt2sas0: ioport(0x0000000000001000), size(256)
[   17.897167] mpt2sas0: Allocated physical memory: size(6712 kB)
[   17.897168] mpt2sas0: Current Controller Queue Depth(2811), Max Controller Queue Depth(3072)
[   17.897169] mpt2sas0: Scatter Gather Elements per IO(128)
[   18.129978] mpt2sas0: LSISAS2308: FWVersion(17.00.01.00), ChipRevision(0x05), BiosVersion(07.24.01.00)
[   18.129985] mpt2sas0: Protocol=(Initiator), Capabilities=(Raid,TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Tas
k Set Full,NCQ)
[   18.131619] mpt2sas0: sending port enable !!
[   19.662657] mpt2sas0: host_add: handle(0x0001), sas_addr(0x500605b006c4ed70), phys(8)
[   25.796718] mpt2sas0: port enable: SUCCESS
[   28.565856] mptctl: Registered with Fusion MPT base driver
[   28.565857] mptctl: /dev/mptctl @ (major,minor=10,220)

If you need something more, let me know what commands to run.

-Tony

[toc] | [prev] | [next] | [standalone]

#1304745

>>>>> "Tony" == Luck, Tony <tony.luck@intel.com> writes:

Tony> If you need something more, let me know what commands to run.

Well, you definitely have a SAS2308 installed. The question is whether
that's the only controller or not. Your bisect aimed at a megaraid_sas
patch. I just want to make sure we're looking at the right set of
commits.

So if you could boot an older kernel and verify that your root fs is
indeed on a disk attached to the mpt2sas controller that would be
helpful.

-- 
Martin K. Petersen	Oracle Linux Engineering

[toc] | [prev] | [next] | [standalone]

#1304748

> Well, you definitely have a SAS2308 installed. The question is whether
> that's the only controller or not. Your bisect aimed at a megaraid_sas
> patch. I just want to make sure we're looking at the right set of
> commits.

I confused you with my e-mail ... that megaraid commit wasn't the final
output of bisect (I've not succeeded in pinning this to a specific commit
in any bisection yet). It was just one of the bad boots.

> So if you could boot an older kernel and verify that your root fs is
> indeed on a disk attached to the mpt2sas controller that would be
> helpful.

All (both) my disks are on this one and only controller. So, yes - my
root filesystem is there.

-Tony

[toc] | [prev] | [next] | [standalone]

#1304729

I also see that mpt2 and mpt3 were merged, with mpt2 option still there
as a legacy driver.  Which should I be using?   My config file has these
MPT settings:

$ grep MPT .config
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_CB710_DEBUG_ASSUMPTIONS=y
CONFIG_SCSI_MPT3SAS=m
CONFIG_SCSI_MPT2SAS_MAX_SGE=128
CONFIG_SCSI_MPT3SAS_MAX_SGE=128
CONFIG_SCSI_MPT2SAS=m

-Tony

[toc] | [prev] | [next] | [standalone]

#1304744

>>>>> "Luck," == Luck, Tony <tony.luck@intel.com> writes:

Luck,> I also see that mpt2 and mpt3 were merged, with mpt2 option still
Luck,> there as a legacy driver.  Which should I be using?

If you previously had MPT2SAS enabled in your .config then the new
kernel should automatically pull MPT3SAS in. You still need to verify in
your initrd that the mpt3sas.ko module gets included, though. How that
works depends on your distro.

-- 
Martin K. Petersen	Oracle Linux Engineering

[toc] | [prev] | [next] | [standalone]

#1304755

> If you previously had MPT2SAS enabled in your .config then the new
> kernel should automatically pull MPT3SAS in. You still need to verify in
> your initrd that the mpt3sas.ko module gets included, though. How that
> works depends on your distro.

Aha! That looks to be the problem. Looking at the initrd generated by
a simple:

 # make install

for the kernel ... I see no "mpt" drivers of any kind!.

The failure mode is very, very odd though. Shouldn't I have got a
"can't mount root" error?

/me goes to find how to tell Dracut to add the mpt3sas module to
the initrd.

Thanks

-Tony

[toc] | [prev] | [next] | [standalone]

#1304773

>>>>> "Tony" == Luck, Tony <tony.luck@intel.com> writes:

Tony> The failure mode is very, very odd though. Shouldn't I have got a
Tony> "can't mount root" error?

The init stuff can be pretty patient when it comes to waiting for the
root disk to materialize. It's conceivable that you would have gotten an
error if you had waited long enough.

Tony> /me goes to find how to tell Dracut to add the mpt3sas module to
Tony> the initrd.

Great!

-- 
Martin K. Petersen	Oracle Linux Engineering

[toc] | [prev] | [next] | [standalone]

#1304851

> Tony> /me goes to find how to tell Dracut to add the mpt3sas module to
> Tony> the initrd.
>
> Great!

/etc/dracut.conf
add_drivers+="mpt3sas"

and now my system boots.

Thanks again for the help.

-Tony

[toc] | [prev] | [next] | [standalone]

#1304878

On Fri, 2016-01-08 at 18:23 +0000, Luck, Tony wrote:
> > Tony> /me goes to find how to tell Dracut to add the mpt3sas module
> > to
> > Tony> the initrd.
> > 
> > Great!
> 
> /etc/dracut.conf
> add_drivers+="mpt3sas"
> 
> and now my system boots.
> 
> Thanks again for the help.

We're not quite out of the woods yet: dracut should have seen this
automatically because of the module alias bindings.  Is your system
somehow unusual and you don't expect the initrd automatic module
selection to work?  If not, could you help us debug why it didn't?

Thanks,

James

[toc] | [prev] | [next] | [standalone]

#1304909

> We're not quite out of the woods yet: dracut should have seen this
> automatically because of the module alias bindings.  Is your system
> somehow unusual and you don't expect the initrd automatic module
> selection to work?  If not, could you help us debug why it didn't?

I saw the same problem building a 4.4-rc* kernel on a RHEL7.2
and SLES12-SP1 system.

My build process is in both cases is:
$ tar xf linux-...
$ cd linux...
$ cp /boot/config-xyzzy .
$ yes "" | make oldconfig
$ make -j 100
$ su
# make -j 100 modules_install && make install

What sort of features of a system might be considered "unusual"?
This looks a very simple system ... just one disk controller with two
drive bays occupied.

Let me know if you'd like run any topology reporting commands.

-Tony

[toc] | [prev] | [next] | [standalone]

#1304925

On Fri, 2016-01-08 at 18:42 +0000, Luck, Tony wrote:
> > We're not quite out of the woods yet: dracut should have seen this
> > automatically because of the module alias bindings.  Is your system
> > somehow unusual and you don't expect the initrd automatic module
> > selection to work?  If not, could you help us debug why it didn't?
> 
> I saw the same problem building a 4.4-rc* kernel on a RHEL7.2
> and SLES12-SP1 system.
> 
> My build process is in both cases is:
> $ tar xf linux-...
> $ cd linux...
> $ cp /boot/config-xyzzy .
> $ yes "" | make oldconfig
> $ make -j 100
> $ su
> # make -j 100 modules_install && make install
> 
> What sort of features of a system might be considered "unusual"?
> This looks a very simple system ... just one disk controller with two
> drive bays occupied.
> 
> Let me know if you'd like run any topology reporting commands.

No, the problem will lie in the initrd building part.  I know that with
debian mkinitramfs takes the -v option which will tell you what modules
it's selecting and why, but I'm not very familiar with enterprise
distributions; chances are whatever command you use to build the initrd
does take a -v.

James

[toc] | [prev] | [next] | [standalone]

#1304921

>>>>> "James" == James Bottomley <James.Bottomley@HansenPartnership.com> writes:

James> We're not quite out of the woods yet: dracut should have seen
James> this automatically because of the module alias bindings.  Is your
James> system somehow unusual and you don't expect the initrd automatic
James> module selection to work?  If not, could you help us debug why it
James> didn't?

Depends how old dracut is. Older versions do not resolve module aliases.

-- 
Martin K. Petersen	Oracle Linux Engineering

[toc] | [prev] | [next] | [standalone]

#1304926

> Depends how old dracut is. Older versions do not resolve module aliases.

On SLES12-SP1
$ dracut --version
getopt: unrecognized option '--version'
Usage: /usr/bin/dracut [OPTION]... [<initramfs> [<kernel-version>]]

Version: 037-66.2

On RHEL 7.2
$ dracut --version
getopt: unrecognized option '--version'
Usage: /usr/bin/dracut [OPTION]... [<initramfs> [<kernel-version>]]

Version: 033-359.el7

-Tony

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web