Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #233345 > unrolled thread

mdadm and whole disk array members

Back to article view | Back to linux.debian.user

Contents

  mdadm and whole disk array members Gary Dale <gary@extremeground.com> - 2021-03-22 23:30 +0100
    Re: mdadm and whole disk array members Andy Smith <andy@strugglers.net> - 2021-03-23 00:00 +0100
      Re: mdadm and whole disk array members Gary Dale <gary@extremeground.com> - 2021-03-23 00:50 +0100
        Re: mdadm and whole disk array members deloptes <deloptes@gmail.com> - 2021-03-23 13:40 +0100
          Re: mdadm and whole disk array members deloptes <deloptes@gmail.com> - 2021-03-23 13:50 +0100
            Re: mdadm and whole disk array members Reco <recoverym4n@enotuniq.net> - 2021-03-23 16:50 +0100
              Re: mdadm and whole disk array members Gary Dale <gary@extremeground.com> - 2021-03-26 02:30 +0100
                Re: mdadm and whole disk array members Felix Miata <mrmazda@earthlink.net> - 2021-03-26 05:10 +0100
                  Re: mdadm and whole disk array members Gary Dale <gary@extremeground.com> - 2021-03-26 19:10 +0100
                Re: mdadm and whole disk array members deloptes <deloptes@gmail.com> - 2021-03-26 08:00 +0100
                  Re: mdadm and whole disk array members Gary Dale <gary@extremeground.com> - 2021-03-26 19:20 +0100
            Re: mdadm and whole disk array members Gary Dale <gary@extremeground.com> - 2021-03-26 02:20 +0100
              Re: mdadm and whole disk array members Gary Dale <gary@extremeground.com> - 2021-03-26 02:30 +0100
          Re: mdadm and whole disk array members Gary Dale <gary@extremeground.com> - 2021-03-26 02:20 +0100
    Re: mdadm and whole disk array members Andy Smith <andy@strugglers.net> - 2021-03-26 17:30 +0100
      Re: mdadm and whole disk array members Gary Dale <gary@extremeground.com> - 2021-03-26 19:40 +0100
        Re: mdadm and whole disk array members Andy Smith <andy@strugglers.net> - 2021-03-26 22:00 +0100

#233345 — mdadm and whole disk array members

I've spent a few days experimenting with using whole disks in a RAID 5 
array and have come to the conclusion that it simply doesn't work well 
enough to be used.

The main problem I had was that mdadm seems to have problems assembling 
the array when it uses entire disks instead of partitions. Each time I 
restarted my computer, I would have to recreate the array. This causes 
the boot process to halt because /etc/mdadm/mdadm.conf and /etc/fstab 
both identify an array that should be started and mounted. Fortunately 
the create command was still in the bash history so I got the create 
parameters right.

However, after I added another disk to the array, that made the original 
create command obsolete. Plus the kernel assigned different drive 
letters to the drives once I plugged in a new drive, so that I couldn't 
simply add the new drive to the create command.

Fortunately I still had a decade-old script that would cycle through all 
combinations until it found one that would result in a mountable array 
(I had the script due to some problems I was having back in 2010). 
Unfortunately it didn't find any it could mount no matter what the order 
of the drives (which included one "missing").

I've found many other people complaining about similar issues when using 
whole disks to create mdadm RAID arrays. Some of these complaints go 
back many years, so this isn't new.

I suggest that, since it appears the developers can't get this work 
reliably, that the option to use the whole disk be removed and mdadm 
insist on using partitions. At the very least, mdadm --create should 
issue a warning that using a whole device instead of a partition may 
create problems.

[toc] | [next] | [standalone]

#233350

Hi Gary,

On Mon, Mar 22, 2021 at 06:20:56PM -0400, Gary Dale wrote:
> I suggest that, since it appears the developers can't get this work
> reliably, that the option to use the whole disk be removed and mdadm insist
> on using partitions. At the very least, mdadm --create should issue a
> warning that using a whole device instead of a partition may create
> problems.

I've been using whole disks in mdadm arrays for more than 15 years
across many many servers on Debian stable and have never experienced
what you describe. There must be something else at play here.

I suggest you post a detailed description of your problem to the
linux-raid mailing list and hopefully someone can help debug it.

    https://raid.wiki.kernel.org/index.php/Linux_Raid#Mailing_list

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]

#233353

On 2021-03-22 18:49, Andy Smith wrote:
> Hi Gary,
>
> On Mon, Mar 22, 2021 at 06:20:56PM -0400, Gary Dale wrote:
>> I suggest that, since it appears the developers can't get this work
>> reliably, that the option to use the whole disk be removed and mdadm insist
>> on using partitions. At the very least, mdadm --create should issue a
>> warning that using a whole device instead of a partition may create
>> problems.
> I've been using whole disks in mdadm arrays for more than 15 years
> across many many servers on Debian stable and have never experienced
> what you describe. There must be something else at play here.
>
> I suggest you post a detailed description of your problem to the
> linux-raid mailing list and hopefully someone can help debug it.
>
>      https://raid.wiki.kernel.org/index.php/Linux_Raid#Mailing_list
>
> Cheers,
> Andy
>
It's not just me but a lot of other people have been having the same 
problem. It's been reported many times as I discovered after trying to 
use whole disks. Moreover, the fixes that I'd used in the past don't 
seem to work reliably without partitions.

There doesn't seem to be a downside to using partitions considering that 
partition tables have never taken up any significant amount of space. It 
was an interesting experiment / learning experience but I've decided 
that it's not worth going further so I'm going back to using disks with 
a single partition.

[toc] | [prev] | [next] | [standalone]

#233382

Gary Dale wrote:

> It's not just me but a lot of other people have been having the same
> problem. It's been reported many times as I discovered after trying to
> use whole disks. Moreover, the fixes that I'd used in the past don't
> seem to work reliably without partitions.

A friend told me that he found out it is a problem in some BIOSes with UEFI
that can not handle a boot of md UEFI partition.
Perhaps it also depends how they handle the raid of a whole disk.
Are you trying to boot from that raid?

[toc] | [prev] | [next] | [standalone]

#233383

deloptes wrote:

> A friend told me that he found out it is a problem in some BIOSes with
> UEFI that can not handle a boot of md UEFI partition.
> Perhaps it also depends how they handle the raid of a whole disk.
> Are you trying to boot from that raid?

Forgot to ask what is in your /etc/mdadm/mdadm.conf and the IDs of the disks

IMO the problem is that if it is not a partition the mdadm can not assemble
as it is looking for a partition, but not sure how grub or whatever handle
it when you boot off the drive.

[toc] | [prev] | [next] | [standalone]

#233386

	Hi.

On Tue, Mar 23, 2021 at 01:44:23PM +0100, deloptes wrote:
> IMO the problem is that if it is not a partition the mdadm can not
> assemble as it is looking for a partition,

My mdadm.conf says:

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan,
# using wildcards if desired.
#DEVICE partitions containers


And /proc/partitions always had whole disks, their partitions, lvm
volumes and whatever else can be presented as a block device by the
kernel.
So mdadm is perfectly capable of assembling whole disk arrays, and it
does so for me for more than 10 years.

> but not sure how grub or whatever handle it when you boot off the
> drive.

GRUB2 can definitely boot from mdadm's RAID1 as it has an appropriate
module for this specific task. Installing GRUB2 on mdadm array made of
whole disks is tricky though.

UEFI itself, on the other hand - definitely can not, unless you resort
to some dirty hacks. After all, UEFI requires so-called "EFI System
Partition" aka ESP.

Reco

[toc] | [prev] | [next] | [standalone]

#233529

On 2021-03-23 11:45, Reco wrote:
> 	Hi.
>
> On Tue, Mar 23, 2021 at 01:44:23PM +0100, deloptes wrote:
>> IMO the problem is that if it is not a partition the mdadm can not
>> assemble as it is looking for a partition,
> My mdadm.conf says:
>
> # by default (built-in), scan all partitions (/proc/partitions) and all
> # containers for MD superblocks. alternatively, specify devices to scan,
> # using wildcards if desired.
> #DEVICE partitions containers
>
>
> And /proc/partitions always had whole disks, their partitions, lvm
> volumes and whatever else can be presented as a block device by the
> kernel.
> So mdadm is perfectly capable of assembling whole disk arrays, and it
> does so for me for more than 10 years.
>
>> but not sure how grub or whatever handle it when you boot off the
>> drive.
> GRUB2 can definitely boot from mdadm's RAID1 as it has an appropriate
> module for this specific task. Installing GRUB2 on mdadm array made of
> whole disks is tricky though.
>
> UEFI itself, on the other hand - definitely can not, unless you resort
> to some dirty hacks. After all, UEFI requires so-called "EFI System
> Partition" aka ESP.
>
> Reco

 From what I read in looking for solutions, the problem is common. I 
even tried one workaround of zapping any existing partition table on the 
drives. Nothing worked.

Perhaps it only works with virgin drives? Mine had been removed from 
another machine where they had been part of a different array. I zeroed 
the superblocks before creating the new array.

[toc] | [prev] | [next] | [standalone]

#233537

Gary Dale composed on 2021-03-25 21:19 (UTC-0400):

>  From what I read in looking for solutions, the problem is common. I 
> even tried one workaround of zapping any existing partition table on the 
> drives. Nothing worked.
										
"Zapped" exactly how? GPT tables are on both ends of the disks. Wiping the first
sectors won't get the job done.
-- 
Evolution as taught in public schools, like religion,
	is based on faith, not on science.

 Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!

Felix Miata  ***  http://fm.no-ip.com/

[toc] | [prev] | [next] | [standalone]

#233565

On 2021-03-26 00:04, Felix Miata wrote:
> Gary Dale composed on 2021-03-25 21:19 (UTC-0400):
>
>>   From what I read in looking for solutions, the problem is common. I
>> even tried one workaround of zapping any existing partition table on the
>> drives. Nothing worked.
> 										
> "Zapped" exactly how? GPT tables are on both ends of the disks. Wiping the first
> sectors won't get the job done.
sgdisk --zap wipes the partition tables.

[toc] | [prev] | [next] | [standalone]

#233542

Gary Dale wrote:

> Perhaps it only works with virgin drives? Mine had been removed from
> another machine where they had been part of a different array. I zeroed
> the superblocks before creating the new array.

I doubt that - IMO should be either the BIOS or the drives, or a combination
of both

[toc] | [prev] | [next] | [standalone]

#233566

On 2021-03-26 02:59, deloptes wrote:
> Gary Dale wrote:
>
>> Perhaps it only works with virgin drives? Mine had been removed from
>> another machine where they had been part of a different array. I zeroed
>> the superblocks before creating the new array.
> I doubt that - IMO should be either the BIOS or the drives, or a combination
> of both
>
It's a Gigabyte ROG STRIX B550-F board. The drives are Seagate Ironwolf 
and WD Red.

[toc] | [prev] | [next] | [standalone]

#233526

On 2021-03-23 08:44, deloptes wrote:
> deloptes wrote:
>
>> A friend told me that he found out it is a problem in some BIOSes with
>> UEFI that can not handle a boot of md UEFI partition.
>> Perhaps it also depends how they handle the raid of a whole disk.
>> Are you trying to boot from that raid?
> Forgot to ask what is in your /etc/mdadm/mdadm.conf and the IDs of the disks
>
> IMO the problem is that if it is not a partition the mdadm can not assemble
> as it is looking for a partition, but not sure how grub or whatever handle
> it when you boot off the drive.
>
>
The drives use normal /dev/sd* ids. They are not being booted from. I 
had updated /etc/mdadm/mdadm/conf with the new information for the array 
after creating it.

When I did exactly the same thing after creating a single FD00 partition 
on the drives, everything worked.

[toc] | [prev] | [next] | [standalone]

#233530

On 2021-03-25 21:14, Gary Dale wrote:
> On 2021-03-23 08:44, deloptes wrote:
>> deloptes wrote:
>>
>>> A friend told me that he found out it is a problem in some BIOSes with
>>> UEFI that can not handle a boot of md UEFI partition.
>>> Perhaps it also depends how they handle the raid of a whole disk.
>>> Are you trying to boot from that raid?
>> Forgot to ask what is in your /etc/mdadm/mdadm.conf and the IDs of 
>> the disks
>>
>> IMO the problem is that if it is not a partition the mdadm can not 
>> assemble
>> as it is looking for a partition, but not sure how grub or whatever 
>> handle
>> it when you boot off the drive.
>>
>>
> The drives use normal /dev/sd* ids. They are not being booted from. I 
> had updated /etc/mdadm/mdadm/conf with the new information for the 
> array after creating it.
>
> When I did exactly the same thing after creating a single FD00 
> partition on the drives, everything worked.
>
When I say "the same thing", I mean creating the array from the 
partitions instead of the whole drives.

[toc] | [prev] | [next] | [standalone]

#233527

On 2021-03-23 08:29, deloptes wrote:
> Gary Dale wrote:
>
>> It's not just me but a lot of other people have been having the same
>> problem. It's been reported many times as I discovered after trying to
>> use whole disks. Moreover, the fixes that I'd used in the past don't
>> seem to work reliably without partitions.
> A friend told me that he found out it is a problem in some BIOSes with UEFI
> that can not handle a boot of md UEFI partition.
> Perhaps it also depends how they handle the raid of a whole disk.
> Are you trying to boot from that raid?
>
No.

[toc] | [prev] | [next] | [standalone]

#233563

Hello,

On Mon, Mar 22, 2021 at 06:20:56PM -0400, Gary Dale wrote:
> I've found many other people complaining about similar issues when using
> whole disks to create mdadm RAID arrays. Some of these complaints go back
> many years, so this isn't new.

Any time I've seen this problem pursued (as opposed to throwing up
hands and just using partitions) it was found to be the motherboard
(well, the UEFI bit) deciding that a storage device with no GPT or
MBR needs an empty GPT adding to it. This corrupts md arrays on whole
devices.

Example:
http://forum.asrock.com/forum_posts.asp?TID=10174&title=asrock-motherboard-destroys-linux-software-raid

Is it possible that this is happening to you?

If not, once again I urge you to go on over to linux-raid list and
describe what's happening.

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]

#233568

On 2021-03-26 12:28, Andy Smith wrote:
> Hello,
>
> On Mon, Mar 22, 2021 at 06:20:56PM -0400, Gary Dale wrote:
>> I've found many other people complaining about similar issues when using
>> whole disks to create mdadm RAID arrays. Some of these complaints go back
>> many years, so this isn't new.
> Any time I've seen this problem pursued (as opposed to throwing up
> hands and just using partitions) it was found to be the motherboard
> (well, the UEFI bit) deciding that a storage device with no GPT or
> MBR needs an empty GPT adding to it. This corrupts md arrays on whole
> devices.
>
> Example:
> http://forum.asrock.com/forum_posts.asp?TID=10174&title=asrock-motherboard-destroys-linux-software-raid
>
> Is it possible that this is happening to you?
>
> If not, once again I urge you to go on over to linux-raid list and
> describe what's happening.
>
> Cheers,
> Andy
>
Actually, the link shows that they eventually did exactly what I did - 
throw up their hands and go back to using disks with a single partition.

If there are a significant number of BIOSs that overwrite mdadm raid 
information when using whole disks then that is a definition of 
unreliable. Imagine swapping out a motherboard and finding your RAID 
array has vanished.

The link to the ASRock page suggests that it is a common "feature" in 
their recent AMD motherboards, I don't know if that also appears in 
Gigabyte AMD boards. Nor can I think of a reason why it would be 
specific to AMD boards.

It doesn't appear to be related to the drives unless Seagate, WD and 
HGST all suffer from the problem.

However, I'm not convinced that the problem is the BIOS writing a 
partition table. In your link the last post talks about zapping the 
partition table to stop the behaviour. This suggests the BIOS/UEFI was 
restoring the backup partition table. However I'd already zapped the 
partition table on my disks so that wouldn't be my case.

[toc] | [prev] | [next] | [standalone]

#233579

Hello,

On Fri, Mar 26, 2021 at 02:31:21PM -0400, Gary Dale wrote:
> I'm not convinced that the problem is the BIOS writing a partition
> table. In your link the last post talks about zapping the
> partition table to stop the behaviour. This suggests the BIOS/UEFI
> was restoring the backup partition table. However I'd already
> zapped the partition table on my disks so that wouldn't be my
> case.

It should be pretty easy to check if this is what's happening,
without md being in the picture, if you have a will.

Though if you now have a working system I appreciate that you
probably don't want to tear it apart to test a theory.

I was only mentioning it as this is the only time I have ever seen
anything remotely like what you describe.

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web