Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #210275 > unrolled thread

New Hardware + old disks not recognized

Back to article view | Back to linux.debian.user

Contents

  New Hardware + old disks not recognized Ross Boylan <rossboylan@stanfordalumni.org> - 2019-06-23 04:10 +0200
    Re: New Hardware + old disks not recognized Pascal Hambourg <pascal@plouf.fr.eu.org> - 2019-06-23 11:10 +0200
      Re: New Hardware + old disks not recognized Ross Boylan <rossboylan@stanfordalumni.org> - 2019-06-24 01:50 +0200
        Re: New Hardware + old disks not recognized [SOLVED + lessons learned] Ross Boylan <rossboylan@stanfordalumni.org> - 2019-06-24 20:20 +0200
          Re: New Hardware + old disks not recognized [SOLVED + lessons  learned] songbird <songbird@anthive.com> - 2019-06-25 01:20 +0200
          Re: New Hardware + old disks not recognized [SOLVED + lessons learned] David Wright <deblis@lionunicorn.co.uk> - 2019-06-25 03:00 +0200
        Re: New Hardware + old disks not recognized Pascal Hambourg <pascal@plouf.fr.eu.org> - 2019-06-25 21:40 +0200
          Re: New Hardware + old disks not recognized Ross Boylan <rossboylan@stanfordalumni.org> - 2019-06-26 20:20 +0200
            Re: New Hardware + old disks not recognized Pascal Hambourg <pascal@plouf.fr.eu.org> - 2019-06-26 21:20 +0200
    Re: New Hardware + old disks not recognized songbird <songbird@anthive.com> - 2019-06-23 14:20 +0200

#210275 — New Hardware + old disks not recognized

In brief: moved all the 3.5" disks from an old system to a new one,
and now I can't boot into buster.  In the initrd environment no disks
appear in /dev; the disks are all connected through an LSI Host Bus
Adapter card (only on the new system).  I can boot into Ubuntu on the
new system, and from there can see and use all the disks.


More details:
My old system was experiencing a lot of problems and so I got a new
system.  The system came with Ubuntu 18.04 installed on an NVME SSD,
and it boots into it fine.  I took all my 3.5" disks from the old
system and put them in the new one.  From within Ubuntu all the disks
are recognized and things seem fine.

I have not been able to boot into my old system (that is, the buster
installation on the old disks) since moving the disks to the new
system.  I boot using grub and initrd after picking the disk to boot
from in the BIOS; the root file system is an encrypted volume in a an
LVM volume group.  /boot is on a separate, unencrypted partition.

My current guess is that the key problem is that none of my disks are
recognized.  Using break=mount I interrupted the initialization and
found /dev contained no entries for sd*, for nvme*, or for a disks/
directory.  All are present on Ubuntu.

My leading suspect for why the hard disks aren't recognized is that
they are all attached through
02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic
SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] (rev 02)
whereas before they were vanilla SATA (although at least one used a
SATA expansion card).  I am not using the card for RAID and was told
it didn't even support RAID.  The box says it's an LSI SAS 9201-16i
Host Bus Adapter.
The purpose of the card was just to permit connections to more drives,
but it certainly sounds as if it's a different technology (I had no
SAS and no SCSI before, though it seemed SATA was using some SCSI
drivers anyway).

AFAIK the SSD is not going through the LSI HBA.  But it's irrelevant
to booting into buster.

Note the drive with /boot is attached through the LSI as well, and the
break seems to clearly land me inside its initrd.  So the very start
of the bootstrap process can see the disk.

Ubuntu doesn't indicates it's running any proprietary drivers for
anything but video.  I do see an nvme driver loaded, as well as
scsi_transport_sas *used by mpt3sas) and some raid modules.  None of
the modules have LSI in the name (case-insensitive comparison).

The symptoms of the problem when I don't break into the init scripts
are somewhat variable, but usually I get messages
  Volume Group vgbarley not available
  Can not process volume group
repeated many times.  The system is unresponsive, but if it sits there
for a couple of minutes it says
  encrypted source ... for root does not exist
and drops into busybox.  Once in busybox, the problem noted above is
evident: no disks.

People having similar symptoms, failure to find an LVM group, on the
net seem to have solved it by ensuring that vgchange -ay gets
executed.  But, with no disks, vgchange won't have anything to work
with.

Any ideas?
Ross

[toc] | [next] | [standalone]

#210289

Le 23/06/2019 à 04:06, Ross Boylan a écrit :
> 
> My leading suspect for why the hard disks aren't recognized is that
> they are all attached through
> 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic
> SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] (rev 02)
> whereas before they were vanilla SATA (although at least one used a
> SATA expansion card).  I am not using the card for RAID and was told
> it didn't even support RAID.  The box says it's an LSI SAS 9201-16i
> Host Bus Adapter.

Possible causes of this kind of problem include :
- lack of support by the kernel (too old for the new hardware)
- missing driver in the initramfs.

Could you post the detailed identification of the adapter printed by 
"lspci -nn" ? According to the identification string the VID:DID should 
be 1000:0064 or 1000:0065. Both identifiers have been recognized by the 
mpt2sas/mpt3sas module at least since Jessie.

When at the initramfs prompt, could you check whether the mpt3sas module 
is present in
lib/modules/<version>/kernel/drivers/scsi/mpt3sas/mpt3sas.ko and loaded 
(as shown by lsmod or in /proc/modules) ? If it is missing, maybe 
"MODULES=dep" was set in /etc/initramfs-tools/initramfs.conf (including 
only modules for the current hardware at the time of initramfs build) 
instead of "MODULES=most" (including modules for most storage hardware).

[toc] | [prev] | [next] | [standalone]

#210320

I think you're right about the missing drivers, but fixing it has
proven challenging.

First, my original initrd was created with MODULES=dep.  This was
somewhat hidden by the fact that in initramfs.conf, MODULES=most. :)
Apparently it was overriden by the setting in conf.d/driver-policy,
which I have since edited.

lspci -nn confirms one of the device ids you expected.  I also
included the NVMe device too:
02:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios
Logic SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] [1000:0064] (rev
02)
71:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co
Ltd NVMe SSD Controller SM961/PM961 [144d:a804]

First try: chroot into the old system and update-initramfs.  The
result can see the disks (yay) but has no, or at least insufficient,
crypto.

Second try: rsync from the old initrd to the new one.*  Considered
using -u, but that left an empty cryptroot file in place.  Booting
using initrd post-rsync is back to the old situation with no disks
visible.  Presumably some files that should have been merged (e.g.,
modules) were overwritten by the rsync.

Third try: do lsmod in ubuntu and insert all the modules into the
initrd modules file.  Still no disks visible.  I'm not sure if I
should have been using module names as reported by lsmod or something
more like the file names, and I later noticed that the modules are
loaded in the order given.  I don't know if the order was right.

Fourth try: regenerate a "clean" initrd as in the first step and
attempt to merge in the old initrd files that are missing or
different.  For some reason this produced a lot more complaints than
the first time I did it.** That includes the modules.aliases file.
However, some of the module management files are binary (ld.so.cache).
I wasn't sure how regenerating them in a chroot would work.  Booting
with this got further: disks are recognized and I get a prompt to
decrypt the root partition.  However, my password is not accepted.  In
the past I've seen this when the right crypto modules were not loaded.

I did copy over some crypto modules and associated binaries in the
fourth try, but perhaps I need to regenerate the caches/management
files to get them to work properly.  It's a little odd they weren't
there to begin with, given the MODULES=most selection and the presence
of some crypto modules.

There are also some differences in font-related files; I'm hoping they
don't matter (e.g., cache files with different names, different uuid
in local/share/fonts/.uuid).

@songbird: maybe I should try installing buster somewhere on the SSD,
but I'm not sure that's going to get me much closer into booting my
old system.  If I ensured that it had similar drivers (i.e., setup
using LVM and encrypted root) it should be closer, but it's still not
my target system.

I wouldn't want to use a system without encryption, and at any event
my employer requires it.

Ross

* I'm using a bit of a short-hand.  I expand the initrd into a
directory, manipulate files in that directory or comparing those with
files expanded from another initrd, and then recreate the initrd file
from the directory.

** The complaints are all on point however, and the condition they
describe was true the first time too:
# update-initramfs -u -k 4.19.0-5-amd64
update-initramfs: Generating /boot/initrd.img-4.19.0-5-amd64
/usr/share/initramfs-tools/hooks/cryptroot: 64:
/usr/share/initramfs-tools/hooks/cryptroot: cannot open /proc/mounts:
No such file
cryptsetup: WARNING: Couldn't determine root device
sed: can't read /proc/cmdline: No such file or directory
/usr/share/initramfs-tools/hooks/cryptroot: 64:
/usr/share/initramfs-tools/hooks/cryptroot: cannot open /proc/mounts:
No such file
cryptsetup: WARNING: The initramfs image may not contain cryptsetup binaries
    nor crypto modules. If that's on purpose, you may want to uninstall the
    'cryptsetup-initramfs' package in order to disable the cryptsetup initramfs
    integration and avoid this warning.
W: Couldn't identify type of root file system for fsck hook

/proc/mounts isn't accessible in the chroot; even if it were, it would
not give the mounts appropriate for the system I'm trying to set up.

[toc] | [prev] | [next] | [standalone]

#210337 — Re: New Hardware + old disks not recognized [SOLVED + lessons learned]

ORIGINAL PROBLEM
Move disks into a new system (hardware) and found I couldn't boot into
the old system (OS on the disk).  The initrd couldn't even see the
drives.

CAUSE
New hardware required drivers missing from  the initrd, which had been
created with MODULES=dep.  Also, fstab referenced a drive left in the
old hardware.  My OS used crypto and LVM, although that only became
relevant once the "no drives" problem was fixed.

SOLUTION:
Boot into alternate system on the new hardware and mount the old OS.
Create a new  initrd
      1. unpack the old initrd.
      2. chroot into old OS, change to MODULES=most, and regenerate
the initrd.  This produces error messages about the result not being
right.  It's not right, but unpack it anyway.
      3. copy files present in the old and absent in the new initrd to
the new initrd.
      4. merge/inspect/evaluate file conflicts.  In particular, the
old cryptroot should replace the new, empty cryptroot.
       5.  Use depmod -b to regenerate the module dependency list
(otherwise the modules copied in step 3 will be ignored).
       6.  Adjust fstab and networking as needed for new interface
names.  These are on the real system, not the initrd.
       7.  pack up the revised initrd and boot.

5 + 6 were the steps I added since my last message.  Without 5 the
crypto modules copied in 3 were not accessible, and I couldn't unlock
the drives.
Without 6,  fstab referred to one drive that stayed in the old system;
apparently that was enough to keep systemd from booting into the
regular system (it kept dropping me in an emergency shell, albeit
after pivot to the real OS from the initrd).

I did most of my network adjustments after booting into the system I
was trying to bring up.  Most of the networking related services
needed to be restarted after fixing that up.  In retrospect there is
an

ALTERNATE SOLUTION
   1. Move drives back to old system.
   2. Ensure MODULES=most and fstab only refers to drives that will be
in the new hardware.
   3. update-initramfs.
   4.  move drives back to new system.

There are also probably ways to get a correct initrd generated from
within the chroot environment on the new hardware.  I don't understand
initramfs-tools well enough to know how to pull that off.

LESSONS
1. Don't ever get in this situation.  It's a mess.
2. If you're planning on moving to new hardware, ensure your initrd is
generated with MODULES=most BEFORE the move.
3. Just because initramfs.conf has MODULES=most doesn't mean that's
what you get.  In my case, conf.d/driver-policy overrode it.
4. Manually constructing or modifying an initrd is pretty delicate business.
5. systemd seems to react very badly to a mount failure in fstab.

THANKS
To Pascal and songbird for their help.

Ross

[toc] | [prev] | [next] | [standalone]

#210343 — Re: New Hardware + old disks not recognized [SOLVED + lessons learned]

Ross Boylan wrote:
...
> LESSONS
> 1. Don't ever get in this situation.  It's a mess.
> 2. If you're planning on moving to new hardware, ensure your initrd is
> generated with MODULES=most BEFORE the move.
> 3. Just because initramfs.conf has MODULES=most doesn't mean that's
> what you get.  In my case, conf.d/driver-policy overrode it.
> 4. Manually constructing or modifying an initrd is pretty delicate business.
> 5. systemd seems to react very badly to a mount failure in fstab.

  all good to know.  :)  i've not been in that
bad a situation before.

  that last bit (#5) has caught plenty of people 
who had old fstabs and did not change the lines 
that were optional devices to "nofail".  it did
get me too when systemd was first coming through
testing/unstable.

  i absolutely hate being dropped into a raw
console or in the emergency mode because the
fonts are usually too small for me to have much
of a chance of reading them.


> THANKS
> To Pascal and songbird for their help.

  y.w., but to me it looks like you did all the
heavy lifting here.  :)  glad you got it figured
out.


  songbird

[toc] | [prev] | [next] | [standalone]

#210345 — Re: New Hardware + old disks not recognized [SOLVED + lessons learned]

On Mon 24 Jun 2019 at 11:15:50 (-0700), Ross Boylan wrote:

> LESSONS
> 1. Don't ever get in this situation.  It's a mess.
> 2. If you're planning on moving to new hardware, ensure your initrd is
> generated with MODULES=most BEFORE the move.
> 3. Just because initramfs.conf has MODULES=most doesn't mean that's
> what you get.  In my case, conf.d/driver-policy overrode it.

AIUI initramfs.conf will always have MODULES=most unless you yourself
edit it. That's because the d-i always installs the same file
(currently Feb 18 2017 in stretch) and just adds conf.d/driver-policy
(containing MODULES=dep) if you select "targeted".

> 4. Manually constructing or modifying an initrd is pretty delicate business.
> 5. systemd seems to react very badly to a mount failure in fstab.

One strategic decision I've made for years is to always have two root
partitions on all my system disks, rather than just one. This should
make it easier to work out what's missing, by performing an
installation in the new hardware onto the old disk's spare partition.
(Obviously it can also be used for making a fresh installation of a
new Debian release without overwriting the current one.)

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#210388

Le 24/06/2019 à 01:40, Ross Boylan a écrit :
> 
> # update-initramfs -u -k 4.19.0-5-amd64
> update-initramfs: Generating /boot/initrd.img-4.19.0-5-amd64
> /usr/share/initramfs-tools/hooks/cryptroot: 64:
> /usr/share/initramfs-tools/hooks/cryptroot: cannot open /proc/mounts:
> No such file
> cryptsetup: WARNING: Couldn't determine root device
> sed: can't read /proc/cmdline: No such file or directory
> /usr/share/initramfs-tools/hooks/cryptroot: 64:
> /usr/share/initramfs-tools/hooks/cryptroot: cannot open /proc/mounts:
> No such file
> cryptsetup: WARNING: The initramfs image may not contain cryptsetup binaries
>      nor crypto modules. If that's on purpose, you may want to uninstall the
>      'cryptsetup-initramfs' package in order to disable the cryptsetup initramfs
>      integration and avoid this warning.
> W: Couldn't identify type of root file system for fsck hook
> 
> /proc/mounts isn't accessible in the chroot;

You need to mount /proc. In a chroot, it is often desirable to mount at 
least the /dev, /proc and /sys pseudo-filesystems.

> even if it were, it would
> not give the mounts appropriate for the system I'm trying to set up.

Why not ? The output of /proc/mounts is not static, it is relative to 
the process reading it.

[toc] | [prev] | [next] | [standalone]

#210420

On Tue, Jun 25, 2019 at 12:31 PM Pascal Hambourg <pascal@plouf.fr.eu.org> wrote:
>
> Le 24/06/2019 à 01:40, Ross Boylan a écrit :
> >
> > # update-initramfs -u -k 4.19.0-5-amd64
> > update-initramfs: Generating /boot/initrd.img-4.19.0-5-amd64
> > /usr/share/initramfs-tools/hooks/cryptroot: 64:
> > /usr/share/initramfs-tools/hooks/cryptroot: cannot open /proc/mounts:
> > No such file
> > cryptsetup: WARNING: Couldn't determine root device
> > sed: can't read /proc/cmdline: No such file or directory
> > /usr/share/initramfs-tools/hooks/cryptroot: 64:
> > /usr/share/initramfs-tools/hooks/cryptroot: cannot open /proc/mounts:
> > No such file
> > cryptsetup: WARNING: The initramfs image may not contain cryptsetup binaries
> >      nor crypto modules. If that's on purpose, you may want to uninstall the
> >      'cryptsetup-initramfs' package in order to disable the cryptsetup initramfs
> >      integration and avoid this warning.
> > W: Couldn't identify type of root file system for fsck hook
> >
> > /proc/mounts isn't accessible in the chroot;
>
> You need to mount /proc. In a chroot, it is often desirable to mount at
> least the /dev, /proc and /sys pseudo-filesystems.
I'm aware, and probably should have qualified my statement--proc
wasn't accessible when I ran the command.  I hadn't mounted it partly
out of laziness and mostly out of concern that stuff in the chroot
would leak into the main system, or vice-versa.   In  particular ...
>
> > even if it were, it would
> > not give the mounts appropriate for the system I'm trying to set up.
>
> Why not ? The output of /proc/mounts is not static, it is relative to
> the process reading it.

I thought /proc would give me the root of the parent system, but I
just tried it and see that's not so.  Thank you for pointing that out.
Has it always worked that way?

So do you think the chroot generated initrd would have been OK  if I'd
mounted proc?

Ross

[toc] | [prev] | [next] | [standalone]

#210424

Le 26/06/2019 à 20:15, Ross Boylan a écrit :
> 
> So do you think the chroot generated initrd would have been OK  if I'd
> mounted proc?

Yes.

[toc] | [prev] | [next] | [standalone]

#210295

Ross Boylan wrote:

> In brief: moved all the 3.5" disks from an old system to a new one,
> and now I can't boot into buster.  In the initrd environment no disks
> appear in /dev; the disks are all connected through an LSI Host Bus
> Adapter card (only on the new system).  I can boot into Ubuntu on the
> new system, and from there can see and use all the disks.
>
...
> Any ideas?

  i like to keep things simple so i don't do encryption,
etc.

  if you have space on the SDD i would create an extra
partition and install Buster to that directly and
then go from there.  likely it will also have UEFI
for which i use refind to manage the boot process
so if there isn't a boot partition for efi it may
be good to set up one for that too.


  songbird

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web