Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #268749 > unrolled thread

HDD long-term data storage with ensured integrity

Back to article view | Back to linux.debian.user

Contents

  HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-06 09:52 +0200
    Re: HDD long-term data storage with ensured integrity Marc SCHAEFER <schaefer@alphanet.ch> - 2024-04-08 11:40 +0200
      Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-08 20:30 +0200
        Re: HDD long-term data storage with ensured integrity Marc SCHAEFER <schaefer@alphanet.ch> - 2024-04-08 22:10 +0200
          Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-09 00:50 +0200
          Re: HDD long-term data storage with ensured integrity Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-03 13:30 +0200
            Re: HDD long-term data storage with ensured integrity Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-05-03 14:50 +0200
            Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-05-03 23:00 +0200
              Re: HDD long-term data storage with ensured integrity Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-04 09:50 +0200
                Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-20 14:40 +0200
                  Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Franco Martelli <martellif67@gmail.com> - 2024-05-21 20:50 +0200
                    Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-22 12:10 +0200
                    Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-22 09:00 +0200
                      Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Andy Smith <andy@strugglers.net> - 2024-05-22 12:20 +0200
                        Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-22 12:30 +0200
                      Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-22 09:00 +0200
                        Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Stefan Monnier <monnier@iro.umontreal.ca> - 2024-05-22 23:10 +0200
                          Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-23 09:00 +0200
                  Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Franco Martelli <martellif67@gmail.com> - 2024-05-28 15:30 +0200
        Why LVM (was: HDD long-term data storage with ensured integrity) Stefan Monnier <monnier@iro.umontreal.ca> - 2024-04-08 23:10 +0200
          Re: Why LVM David Christensen <dpchrist@holgerdanske.com> - 2024-04-09 01:10 +0200
            Re: Why LVM Stefan Monnier <monnier@iro.umontreal.ca> - 2024-04-09 02:00 +0200
              Re: Why LVM David Christensen <dpchrist@holgerdanske.com> - 2024-04-09 12:20 +0200
    Re: HDD long-term data storage with ensured integrity piorunz <piorunz@gmx.com> - 2024-04-10 02:10 +0200
      Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-10 13:20 +0200
        Re: HDD long-term data storage with ensured integrity Curt <curty@free.fr> - 2024-04-10 17:00 +0200
        Re: HDD long-term data storage with ensured integrity Paul Leiber <paul@onlineschubla.de> - 2024-04-10 18:10 +0200
          Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-11 01:20 +0200
        Re: HDD long-term data storage with ensured integrity piorunz <piorunz@gmx.com> - 2024-04-12 17:20 +0200
          Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-13 05:10 +0200

Page 1 of 2  [1] 2  Next page →

#268749 — HDD long-term data storage with ensured integrity

On 3/31/24 02:18, DdB wrote:
 > i intend to create a huge backup server from some oldish hardware.
 > Hardware has been partly refurbished and offers 1 SSD + 8 HDD on a
 > 6core Intel with 64 GB RAM. ... the [Debian] installer ... aborts.

On 4/1/24 11:35, DdB wrote:
 > A friend of mine just let me use an external CD-Drive with the netboot
 > image. ... all is well.


Now you get to solve the same problem I have been stuck on since last 
November -- how to use those HDD's.


ZFS has been my bulk storage solution of choice for the past ~4 years, 
but the recent data corruption bugs [1, 2] have me worried.  From a 
technical perspective, it's about incorrect concurrent execution of GNU 
cp(1), Linux, and/or OpenZFS.  From a management perspective, it's about 
Capability Maturity Model (CMM) [3] and Programming Systems Product 
(PSP) [4].


The most obvious alternative to ZFS on Debian would be Btrfs.  Does 
anyone have any comments or suggestions regarding Btrfs and data 
corruption bugs, concurrency, CMM level, PSP, etc.?


Does anyone have any comments or suggestions regarding how to use 
magnetic hard disk drives, commodity x86 computers, and Debian for 
long-term data storage with ensured integrity?


David


[1] https://github.com/openzfs/zfs/issues/15526

[2] https://github.com/openzfs/zfs/issues/15933

[3] https://en.wikipedia.org/wiki/Capability_maturity_model

[4] https://en.wikipedia.org/wiki/The_Mythical_Man-Month

[toc] | [next] | [standalone]

#268777

For offline storage:

On Tue, Apr 02, 2024 at 05:53:15AM -0700, David Christensen wrote:
> Does anyone have any comments or suggestions regarding how to use magnetic
> hard disk drives, commodity x86 computers, and Debian for long-term data
> storage with ensured integrity?

I use LVM on ext4, and I add a MD5SUMS file at the root.

I then power up the drives at least once a year and check the MD5SUMS.

A simple CRC could also work, obviously.

So far, I have not detected MORE corruption with this method than the
drive ECC itself (current drives & buses are much better than they
used to be).  When I have errors detected, I replace the file with
another copy (I usually have multiple off-site copies, and sometimes
even on-site online copies, but not always).  When the errors add
up, it is time to buy another drive, usually after 5+ years or
even sometimes 10+ years.

So, just re-reading the content might be enough, once a year or so.

This is for HDD (for SDD I have no offline storage experience, it
could be shorter).

[toc] | [prev] | [next] | [standalone]

#268784

On 4/8/24 02:38, Marc SCHAEFER wrote:
> For offline storage:
> 
> On Tue, Apr 02, 2024 at 05:53:15AM -0700, David Christensen wrote:
>> Does anyone have any comments or suggestions regarding how to use magnetic
>> hard disk drives, commodity x86 computers, and Debian for long-term data
>> storage with ensured integrity?
> 
> I use LVM on ext4, and I add a MD5SUMS file at the root.
> 
> I then power up the drives at least once a year and check the MD5SUMS.
> 
> A simple CRC could also work, obviously.
> 
> So far, I have not detected MORE corruption with this method than the
> drive ECC itself (current drives & buses are much better than they
> used to be).  When I have errors detected, I replace the file with
> another copy (I usually have multiple off-site copies, and sometimes
> even on-site online copies, but not always).  When the errors add
> up, it is time to buy another drive, usually after 5+ years or
> even sometimes 10+ years.
> 
> So, just re-reading the content might be enough, once a year or so.
> 
> This is for HDD (for SDD I have no offline storage experience, it
> could be shorter).


Thank you for the reply.


So, an ext4 file system on an LVM logical volume?


Why LVM?  Are you implementing redundancy (RAID)?  Is your data larger 
than a single disk (concatenation/ JBOD)?  Something else?


David

[toc] | [prev] | [next] | [standalone]

#268785

Hello,

On Mon, Apr 08, 2024 at 11:28:04AM -0700, David Christensen wrote:
> So, an ext4 file system on an LVM logical volume?
> 
> Why LVM?  Are you implementing redundancy (RAID)?  Is your data larger than
> a single disk (concatenation/ JBOD)?  Something else?

For off-site long-term offline archiving, no, I am not using RAID.

No, it's not LVM+md, just plain LVM for flexibility.

Typically I use 16 TB hard drives, and I tend to use one LV per data
source, the LV name being the data source and the date of the copy.
Or sometimes I just copy a raw volume (ext4 or something else)
to a LV.

With smaller drives (4 TB) I tend to not use LVM, just plain ext4 on the
raw disk.

I almost never use partitionning.

However, I tend to use luks encryption (per ext4 filesystem) when the
drives are stored off-site.  So it's either LVM -> LV -> LUKS -> ext4
or raw disk -> LUKS -> ext4.

You can find some of the scripts I use to automate this off-site
long-term archiving here:

https://git.alphanet.ch/gitweb/?p=various;a=tree;f=offsite-archival/LVM-LUKS

[toc] | [prev] | [next] | [standalone]

#268788

On 4/8/24 13:04, Marc SCHAEFER wrote:
> Hello,
> 
> On Mon, Apr 08, 2024 at 11:28:04AM -0700, David Christensen wrote:
>> So, an ext4 file system on an LVM logical volume?
>>
>> Why LVM?  Are you implementing redundancy (RAID)?  Is your data larger than
>> a single disk (concatenation/ JBOD)?  Something else?
> 
> For off-site long-term offline archiving, no, I am not using RAID.
> 
> No, it's not LVM+md, just plain LVM for flexibility.
> 
> Typically I use 16 TB hard drives, and I tend to use one LV per data
> source, the LV name being the data source and the date of the copy.
> Or sometimes I just copy a raw volume (ext4 or something else)
> to a LV.
> 
> With smaller drives (4 TB) I tend to not use LVM, just plain ext4 on the
> raw disk.
> 
> I almost never use partitionning.
> 
> However, I tend to use luks encryption (per ext4 filesystem) when the
> drives are stored off-site.  So it's either LVM -> LV -> LUKS -> ext4
> or raw disk -> LUKS -> ext4.
> 
> You can find some of the scripts I use to automate this off-site
> long-term archiving here:
> 
> https://git.alphanet.ch/gitweb/?p=various;a=tree;f=offsite-archival/LVM-LUKS


Thank you for the clarification.  :-)


David

[toc] | [prev] | [next] | [standalone]

#269175

On Mon, Apr 08, 2024 at 10:04:01PM +0200, Marc SCHAEFER wrote:
> For off-site long-term offline archiving, no, I am not using RAID.

Now, as I had to think a bit about ONLINE integrity, I found this
comparison:

https://github.com/t13a/dm-integrity-benchmarks

Contenders are btrfs, zfs, and notably ext4+dm-integrity+dm-raid

I tend to have a biais favoring UNIX layered solutions against
"all-into-one" solutions, and it seems that performance-wise,
it's also quite good.

I wrote this script to convince myself of auto-correction
of the ext4+dm-integrity+dm-raid layered approach.

It gives:

[ ... ]
[  390.249699] md/raid1:mdX: read error corrected (8 sectors at 21064 on dm-11)
[  390.249701] md/raid1:mdX: redirecting sector 20488 to other mirror: dm-7
[  390.293807] md/raid1:mdX: dm-11: rescheduling sector 262168
[  390.293988] md/raid1:mdX: read error corrected (8 sectors at 262320 on dm-11)
[  390.294040] md/raid1:mdX: read error corrected (8 sectors at 262368 on dm-11)
[  390.294125] md/raid1:mdX: read error corrected (8 sectors at 262456 on dm-11)
[  390.294209] md/raid1:mdX: read error corrected (8 sectors at 262544 on dm-11)
[  390.294287] md/raid1:mdX: read error corrected (8 sectors at 262624 on dm-11)
[  390.294586] md/raid1:mdX: read error corrected (8 sectors at 263000 on dm-11)
[  390.294712] md/raid1:mdX: redirecting sector 262168 to other mirror: dm-7

pretty much convicing.

So after testing btrfs and being not convinced, after doing some test on
a production zfs -- not convinced either -- I am going to ry
ext4+dm-integrity+dm-raid. 

#! /bin/bash

set -e

function create_lo {
   local f

   f=$(losetup -f)

   losetup $f $1
   echo $f
}

# beware of the rm -r below!
tmp_dir=/tmp/$(basename $0)
mnt=/mnt

mkdir $tmp_dir

declare -a pvs
for p in pv1 pv2
do
   truncate -s 250M $tmp_dir/$p
   
   l=$(create_lo $tmp_dir/$p)
   
   pvcreate $l
   
   pvs+=($l)
done

vg=$(basename $0)-test
lv=test

vgcreate $vg ${pvs[*]}

vgdisplay $vg

lvcreate --type raid1 --raidintegrity y -m 1 -L 200M -n $lv $vg

lvdisplay $vg

# sync/integrity complete?
sleep 10
cat /proc/mdstat
echo
lvs -a -o name,copy_percent,devices $vg
echo
echo -n Type ENTER
read ignore

mkfs.ext4 -I 256 /dev/$vg/$lv
mount /dev/$vg/$lv $mnt

for f in $(seq 1 10)
do
   # ignore errors
   head -c 20M < /dev/random > $mnt/f_$f || true
done

(cd $mnt && find . -type f -print0 | xargs -0 md5sum > $tmp_dir/MD5SUMS)

# corrupting some data in one PV
count=5000
blocks=$(blockdev --getsz ${pvs[1]})
if [ $blocks -lt 32767 ]; then
   factor=1
else
   factor=$(( ($blocks - 1) / 32767))
fi

p=1
for i in $(seq 1 $count)
do
  offset=$(($RANDOM * $factor))
  echo ${pvs[$p]} $offset
  dd if=/dev/random of=${pvs[$p]} bs=$(blockdev --getpbsz ${pvs[$p]}) seek=$offset count=1
  # only doing on 1, not 0, since we have no way to avoid destroying the same sector!
  #p=$((1 - p))
done

dd if=/dev/$vg/$lv of=/dev/null bs=32M
dmesg | tail

umount $mnt

lvremove -y $vg/$lv

vgremove -y $vg

for p in ${pvs[*]}
do
   pvremove $p
   losetup -d $p
done

rm -r $tmp_dir

[toc] | [prev] | [next] | [standalone]

#269180

On 3 May 2024 13:26 +0200, from schaefer@alphanet.ch (Marc SCHAEFER):
> https://github.com/t13a/dm-integrity-benchmarks
> 
> Contenders are btrfs, zfs, and notably ext4+dm-integrity+dm-raid

ZFS' selling point is not performance, _especially_ on rotational
drives. In fact, it's fairly widely accepted that ZFS is in fact
inferior in performance compared to pretty much everything else
modern, even at the best of times; and some of its features help
mitigate its lower against-disk performance.

ZFS' value proposition lies elsewhere.

Which is fine. It's the right choice for some people; for others,
other alternatives provide better trade-offs.

-- 
Michael Kjörling                     🔗 https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

[toc] | [prev] | [next] | [standalone]

#269185

On 5/3/24 04:26, Marc SCHAEFER wrote:
> On Mon, Apr 08, 2024 at 10:04:01PM +0200, Marc SCHAEFER wrote:
>> For off-site long-term offline archiving, no, I am not using RAID.
> 
> Now, as I had to think a bit about ONLINE integrity, I found this
> comparison:
> 
> https://github.com/t13a/dm-integrity-benchmarks
> 
> Contenders are btrfs, zfs, and notably ext4+dm-integrity+dm-raid
> 
> I tend to have a biais favoring UNIX layered solutions against
> "all-into-one" solutions, and it seems that performance-wise,
> it's also quite good.
> 
> I wrote this script to convince myself of auto-correction
> of the ext4+dm-integrity+dm-raid layered approach.


Thank you for devising a benchmark and posting some data.  :-)


FreeBSD also offers a layered solution.  From the top down:

* UFS2 file system, which supports snapshots (requires partitions with 
soft updates enabled).

* gpart(8) for partitions (volumes).

* graid(8) for redundancy and self-healing.

* geli(8) providers with continuous integrity checking.


AFAICT the FreeBSD stack is mature and production quality, which I find 
very appealing.  But the feature set is not as sophisticated as ZFS, 
which leaves me wanting.  Notably, I have not found a way to replicate 
UFS snapshots directly -- the best I can dream up is synchronizing a 
snapshot to a backup UFS2 filesystem and then taking a snapshot with the 
same name.


I am coming to the conclusion that the long-term survivability of data 
requires several components -- good live file system, good backups, good 
archives, continuous internal integrity checking with self-healing, 
periodic external integrity checking (e.g. mtree(1)) with some form of 
recovery (e.g. manual), etc.. If I get the other pieces right, I could 
go with OpenZFS for the live and backup systems, and worry less about 
data corruption bugs.


David

[toc] | [prev] | [next] | [standalone]

#269188

On Fri, May 03, 2024 at 01:50:52PM -0700, David Christensen wrote:
> Thank you for devising a benchmark and posting some data.  :-)

I did not do the comparison hosted on github.  I just wrote the
script which tests the dm-integrity on dm-raid error detection
and error correction.

> FreeBSD also offers a layered solution.  From the top down:

I prefer this approach, indeed.

[toc] | [prev] | [next] | [standalone]

#269469 — Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot

Hello,

1. INITIAL SITUATION: WORKS (no dm-integrity at all)

I have a Debian bookwork uptodate system that boots correctly with
kernel 6.1.0-21-amd64.

It is setup like this:

   - /dev/nvme1n1p1 is /boot/efi

   - /dev/nvme0n1p2 and /dev/nvme1n1p2 are the two LVM physical volumes

   - a volume group, vg1 is built with those PVs

vg1 has a few LVs that have been created in RAID1 LVM mode:

lvdisplay | egrep 'Path|Mirrored'

  LV Path                /dev/vg1/root   <-- this is /
  Mirrored volumes       2
  LV Path                /dev/vg1/swap
  Mirrored volumes       2
  LV Path                /dev/vg1/scratch
  Mirrored volumes       2
  LV Path                /dev/vg1/docker
  Mirrored volumes       2

As said, this boots without any issue.

2. ADDING dm-integrity WHILE BOOTED: works!

Now, while booted, I can add dm-integrity to one of the volumes,
let's say /dev/vg1/docker (this LV has absolutely no link with the
boot process, except obviously it is listed in /etc/fstab -- it also
fails the same way if even the swap is dm-integrit enabled, or
/):

   lvconvert  --raidintegrity y --raidintegritymode bitmap vg1/docker

and wait a bit til the integrity is setup with lvs -a (100%)

Obviously, this creates and uses a few rimage/rmeta sub LVs.

Then I did this (after having boot issues):

  echo dm_integrity >> /etc/initramfs-tools/modules
  update-initramfs -u

This did not change the below issue:

3. grub BOOT FAILS IF ANY LV HAS dm-integrity, EVEN IF NOT LINKED TO /

if I reboot now, grub2 complains about rimage issues, clear the screen
and then I am at the grub2 prompt.

Booting is only possible with Debian rescue, disabling the dm-integrity
on the above volume and rebooting. Note that you still can see the
rimage/rmeta sub LVs (lvs -a), they are not deleted! (but no
dm-integrity is activated).

4. update-grub GIVES WARNINGS

Now, if I try to start update-grub while booted AND having enabled
dm-integrity on the vg1/docker volume, I get:

# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.1.0-21-amd64
Found initrd image: /boot/initrd.img-6.1.0-21-amd64
error: unknown node 'docker_rimage_0'.
[ ... many ... ]
/usr/sbin/grub-probe: error: disk `lvmid/xLE0OV-wQy7-88H9-yKCz-4DUQ-Toce-h9rQvk/FzCf1C-95eB-7B0f-DSrF-t1pg-66qp-hmP3nZ' not found.
error: unknown node 'docker_rimage_0'.
[ ... many ... ]

[ this repeats a few times ]

Found linux image: /boot/vmlinuz-6.1.0-10-amd64
Found initrd image: /boot/initrd.img-6.1.0-10-amd64
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Warning: os-prober will not be executed to detect other bootable partitions.
[ there are none ]
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
Adding boot menu entry for UEFI Firmware Settings ...
done

Any idea what could be the problem?  Any way to just make grub2 ignore
the rimage (sub)volumes at setup and boot time?  (I could live with / aka
vg1/root not using dm-integrity, as long as the data/docker/etc volumes
are integrity-protected) ?  Or how to make grub 100% compatible with a
vg1/root using dm-integrity (that would be obviously the final goal!)

Thank you for any pointers!

[toc] | [prev] | [next] | [standalone]

#269477 — Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot

On 20/05/24 at 14:35, Marc SCHAEFER wrote:
> Any idea what could be the problem?  Any way to just make grub2 ignore
> the rimage (sub)volumes at setup and boot time?  (I could live with / aka
> vg1/root not using dm-integrity, as long as the data/docker/etc volumes
> are integrity-protected) ?  Or how to make grub 100% compatible with a
> vg1/root using dm-integrity (that would be obviously the final goal!)
> 
> Thank you for any pointers!

I can only recommend you to read carefully the Wiki:

https://raid.wiki.kernel.org/index.php/Dm-integrity

HTH

kind regards
-- 
Franco Martelli

[toc] | [prev] | [next] | [standalone]

#269479 — Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot

Hello,

On Wed, May 22, 2024 at 08:57:38AM +0200, Marc SCHAEFER wrote:
> I will try this work-around and report back here.  As I said, I can
> live with /boot on RAID without dm-integrity, as long as the rest can be
> dm-integrity+raid protected.

So, enable dm-integrity on all LVs, including /, /var/lib/lxc, /scratch
and swap, now boots without any issue with grub2 as long as /boot is NOT
on the same VG where the dm-integrity over LVM RAID is enabled.

This is OK for me, I don't need /boot on dm-integrity.

update-grub gives out warning for every of the rimage subvolumes, but
can still then reboot.

I would guess the bug is thus in grub2, not yet supporting boot on a
/boot not necessarily dm-integrityfied itself, but on a VG where any
of the LV is.

Are readers seconding conclusion?  If yes, I could report a bug on grub2.

Have a nice day.

Details:
root@ds-03:~# lvs -a
  LV                       VG  Attr       LSize   Pool Origin                   Data%  Meta%  Move Log Cpy%Sync Convert
  docker                   vg1 rwi-aor--- 500.00g                                                      100.00          
  [docker_rimage_0]        vg1 gwi-aor--- 500.00g      [docker_rimage_0_iorig]                         100.00          
  [docker_rimage_0_imeta]  vg1 ewi-ao----  <4.07g                                                                      
  [docker_rimage_0_iorig]  vg1 -wi-ao---- 500.00g                                                                      
  [docker_rimage_1]        vg1 gwi-aor--- 500.00g      [docker_rimage_1_iorig]                         100.00          
  [docker_rimage_1_imeta]  vg1 ewi-ao----  <4.07g                                                                      
  [docker_rimage_1_iorig]  vg1 -wi-ao---- 500.00g                                                                      
  [docker_rmeta_0]         vg1 ewi-aor---   4.00m                                                                      
  [docker_rmeta_1]         vg1 ewi-aor---   4.00m                                                                      
  root                     vg1 rwi-aor---  10.00g                                                      100.00          
  [root_rimage_0]          vg1 gwi-aor---  10.00g      [root_rimage_0_iorig]                           100.00          
  [root_rimage_0_imeta]    vg1 ewi-ao---- 148.00m                                                                      
  [root_rimage_0_iorig]    vg1 -wi-ao----  10.00g                                                                      
  [root_rimage_1]          vg1 gwi-aor---  10.00g      [root_rimage_1_iorig]                           100.00          
  [root_rimage_1_imeta]    vg1 ewi-ao---- 148.00m                                                                      
  [root_rimage_1_iorig]    vg1 -wi-ao----  10.00g                                                                      
  [root_rmeta_0]           vg1 ewi-aor---   4.00m                                                                      
  [root_rmeta_1]           vg1 ewi-aor---   4.00m                                                                      
  scratch                  vg1 rwi-aor---  10.00g                                                      100.00          
  [scratch_rimage_0]       vg1 gwi-aor---  10.00g      [scratch_rimage_0_iorig]                        100.00          
  [scratch_rimage_0_imeta] vg1 ewi-ao---- 148.00m                                                                      
  [scratch_rimage_0_iorig] vg1 -wi-ao----  10.00g                                                                      
  [scratch_rimage_1]       vg1 gwi-aor---  10.00g      [scratch_rimage_1_iorig]                        100.00          
  [scratch_rimage_1_imeta] vg1 ewi-ao---- 148.00m                                                                      
  [scratch_rimage_1_iorig] vg1 -wi-ao----  10.00g                                                                      
  [scratch_rmeta_0]        vg1 ewi-aor---   4.00m                                                                      
  [scratch_rmeta_1]        vg1 ewi-aor---   4.00m                                                                      
  swap                     vg1 rwi-aor---   8.00g                                                      100.00          
  [swap_rimage_0]          vg1 gwi-aor---   8.00g      [swap_rimage_0_iorig]                           100.00          
  [swap_rimage_0_imeta]    vg1 ewi-ao---- 132.00m                                                                      
  [swap_rimage_0_iorig]    vg1 -wi-ao----   8.00g                                                                      
  [swap_rimage_1]          vg1 gwi-aor---   8.00g      [swap_rimage_1_iorig]                           100.00          
  [swap_rimage_1_imeta]    vg1 ewi-ao---- 132.00m                                                                      
  [swap_rimage_1_iorig]    vg1 -wi-ao----   8.00g                                                                      
  [swap_rmeta_0]           vg1 ewi-aor---   4.00m                                                                      
  [swap_rmeta_1]           vg1 ewi-aor---   4.00m                                                                      

root@ds-03:~# df # filtered
Filesystem              1K-blocks    Used Available Use% Mounted on
/dev/mapper/vg1-root     10218772 2863956   6814144  30% /
/dev/nvme0n1p1             446282  109365    308450  27% /boot
/dev/mapper/vg1-scratch  10218772      24   9678076   1% /scratch
/dev/mapper/vg1-docker  514937088  805892 487900412   1% /var/lib/docker
/dev/nvme1n1p1             486456    5972    480484   2% /boot/efi

[toc] | [prev] | [next] | [standalone]

#269480 — Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot

Hello,

On Tue, May 21, 2024 at 08:41:58PM +0200, Franco Martelli wrote:
> I can only recommend you to read carefully the Wiki:
> https://raid.wiki.kernel.org/index.php/Dm-integrity

I did, and it looks it does not seem to document anything pertaining
to my issue:

1) I don't use integritysetup (from LUKS), but LVM RAID PVs -- I don't use
   LUKS encryption anyway on that system

2) the issue is not the kernel not supporting it, because when the
   system is up, it works (I have done tests to destroy part of the
   underlying devices, they get detected and fixed correctly)

3) the issue is not with the initrd -- I added the dm-integrity module
   and rebuilt the initrd (and actually the bug happens before grub2 loads
   the kernel & init) -- or at least "not yet"!  maybe this will fail
   later :)

4) actually the issue is just grub2, be it when the system is up
   (it complains about the special subvolumes) or at boot time

Having /boot on a LVM non enabled dm-integrity logical volume does not
work either, as soon as there is ANY LVM dm-integrity enabled logical
volume anywhere (even not linked to booting), grub2 complains (at boot
time or at update-grub) about the rimage LV.

[toc] | [prev] | [next] | [standalone]

#269481 — Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot

Hello,

On Wed, May 22, 2024 at 08:57:38AM +0200, Marc SCHAEFER wrote:
> I will try this work-around and report back here.  As I said, I can
> live with /boot on RAID without dm-integrity, as long as the rest can be
> dm-integrity+raid protected.

I'm interested in how you get on.

I don't (yet) use dm-integrity, but I have seen extreme fragility in
grub with regard to LVM. For example, a colleague of mine recently
lost 5 hours of their life (and their SLA budget) when simply adding
metadata tags to some PVs prevented grub from assembling them,
resulting in a hard to debug failed boot at next boot.

Anything that involves grub having to interact with LVM just seems
really fragile.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]

#269483 — Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot

Hello,

On Wed, May 22, 2024 at 10:13:06AM +0000, Andy Smith wrote:
> metadata tags to some PVs prevented grub from assembling them,

grub is indeed very fragile if you use dm-integrity anywhere on any of
your LVs on the same VG where /boot is (or at least if in the list
of LVs, the dm-integrity protected ones come first).

I guess it's a general problem how grub2 parses LVM, yes,
as soon as their are special things going on, it somehow breaks.

However, if you don't have /boot on LVM, hand-fixing grub2 can be
trivial, e.g. here on another system with /boot/efi on 1st disk's first
partition and /boot on 2nd disk's first partition.

   linux (hd1,1)vmlinuz-5.10.0-29-amd64 root=/dev/mapper/vg1-root ro quiet
   initrd (hd1,1)initrd.img-5.10.0-29-amd64
   boot

(you even have completions in grub's interactive boot system)

and it boots.  Next step: I am going to make me a USB boot key for that
system, in case (first using a simple mount of two partitions of the
USB key on /boot, respectively /boot/efi (vfat), then update-grub,
or if it breaks, completely by hand like above -- I have been using
syslinux for the last 20 years or so for that purpose, but it gets
apparently too complicated with Secure Boot and stuff).

PS: I have from now on decided I will always use a /boot no longer
    on LVM but on a separate partition, like the /boot/efi, it
    seems, indeed, much less fragile.  Aka, back to what I
    was doing a few years ago before my confidence in grub2
    got apparently too high :)

[toc] | [prev] | [next] | [standalone]

#269482 — Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot

Additional info:

On Wed, May 22, 2024 at 08:49:56AM +0200, Marc SCHAEFER wrote:
> Having /boot on a LVM non enabled dm-integrity logical volume does not
> work either, as soon as there is ANY LVM dm-integrity enabled logical
> volume anywhere (even not linked to booting), grub2 complains (at boot
> time or at update-grub) about the rimage LV.

I found this [1], quoting: "I'd also like to share an issue I've
discovered: if /boot's partition is a LV, then there must not be a
raidintegrity LV anywhere before that LV inside the same VG. Otherwise,
update-grub will show an error (disk `lvmid/.../...' not found) and GRUB
cannot boot. So it's best if you put /boot into its own VG. (PS: Errors
like unknown node '..._rimage_0 can be ignored.)"

So, the work-around seems to be to simple have /boot not on a LVM VG where
any LV has dm-integrity enabled.

I will try this work-around and report back here.  As I said, I can
live with /boot on RAID without dm-integrity, as long as the rest can be
dm-integrity+raid protected.

[1] https://unix.stackexchange.com/questions/717763/lvm2-integrity-feature-breaks-lv-activation

[toc] | [prev] | [next] | [standalone]

#269486 — Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot

> I found this [1], quoting: "I'd also like to share an issue I've
> discovered: if /boot's partition is a LV, then there must not be a
> raidintegrity LV anywhere before that LV inside the same VG. Otherwise,
> update-grub will show an error (disk `lvmid/.../...' not found) and GRUB
> cannot boot. So it's best if you put /boot into its own VG. (PS: Errors
> like unknown node '..._rimage_0 can be ignored.)"

Hmm... I've been using a "plain old partition" for /boot (with
everything else in LVM) for "ever", originally because the boot loader
was not able to read LVM, and later out of habit.  I was thinking of
finally moving /boot into an LV to make things simpler, but I see that
it'd still be playing with fire (AFAICT booting off of LVM was still not
supported by U-Boot either last time I checked).  🙁


        Stefan

[toc] | [prev] | [next] | [standalone]

#269495 — Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot

Hello,

On Wed, May 22, 2024 at 05:03:34PM -0400, Stefan Monnier wrote:
> Hmm... I've been using a "plain old partition" for /boot (with
> everything else in LVM) for "ever", originally because the boot loader
> was not able to read LVM, and later out of habit.  I was thinking of
> finally moving /boot into an LV to make things simpler, but I see that
> it'd still be playing with fire

grub supports, for a long time:

   - / on LVM, with /boot within that filesystem
   - /boot on LVM, separately

(it also worked with LILO, because LILO would record the exact address
 where the kernel & initrd was, regardless of abstractions layers :->)

Recently, I have been playing with RAID-on-LVM (I was mostly using LVM
on md before, which worked with grub), and it works too.

Where grub fails, is if you have /boot on the same LVM volume group
where any of the LVs "before him in order" have:

   - dm-integrity
   - specific metadata

So yes, any advanced setup might break grub, and so the easiest is to
have /boot on its separate partition again for the time being.

Which makes two partitions of you also have an UEFI.

>  (AFAICT booting off of LVM was still not
> supported by U-Boot either last time I checked).  ????

No idea about that one, sorry.

[toc] | [prev] | [next] | [standalone]

#269643 — Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot

Hi Marc,

On 20/05/24 at 14:35, Marc SCHAEFER wrote:
> 3. grub BOOT FAILS IF ANY LV HAS dm-integrity, EVEN IF NOT LINKED TO /
> 
> if I reboot now, grub2 complains about rimage issues, clear the screen
> and then I am at the grub2 prompt.
> 
> Booting is only possible with Debian rescue, disabling the dm-integrity
> on the above volume and rebooting. Note that you still can see the
> rimage/rmeta sub LVs (lvs -a), they are not deleted! (but no
> dm-integrity is activated).
> 
> 4. update-grub GIVES WARNINGS
> 
> Now, if I try to start update-grub while booted AND having enabled
> dm-integrity on the vg1/docker volume, I get:
> 
> # update-grub
> Generating grub configuration file ...
> Found linux image: /boot/vmlinuz-6.1.0-21-amd64
> Found initrd image: /boot/initrd.img-6.1.0-21-amd64
> error: unknown node 'docker_rimage_0'.
> [ ... many ... ]
> /usr/sbin/grub-probe: error: disk
> `lvmid/xLE0OV-wQy7-88H9-yKCz-4DUQ-Toce-h9rQvk/FzCf1C-95eB-7B0f-DSrF-t1pg-66qp-hmP3nZ'
>   not found.
> error: unknown node 'docker_rimage_0'.
> [ ... many ... ]
> 
> [ this repeats a few times ]

Sorry for the late in the answer, but I've just noticed that the Linux 
kernel of Debian Bookworm ISO image (debian-12.0.0-amd64-DVD-1.iso) 
comes *without* "dm-integrity.ko" module, making therefore not possible 
to support volumes formatted with "--raidintegrity y" neither those 
formatted with "integritysetup" command (I think that it's a bug and it 
should be reported).

When you booted in rescue mode which ISO image have you used?

Thank for your patience, kind regards.
-- 
Franco Martelli

[toc] | [prev] | [next] | [standalone]

#268786 — Why LVM (was: HDD long-term data storage with ensured integrity)

David Christensen [2024-04-08 11:28:04] wrote:
> Why LVM?

Personally, I've been using LVM everywhere I can (i.e. everywhere
except on my OpenWRT router, tho I've also used LVM there back when my
router had an HDD.  I also use LVM on my 2GB USB rescue image).

To me the question is rather the reverse: why not?
I basically see it as a more flexible form of partitioning.

Even in the worst cases where I have a single LV volume, I appreciate
the fact that it forces me to name things, isolating me from issue
linked to predicting the name of the device and the issues that plague
UUIDs (the fact they're hard to remember, and that they're a bit too
magical/hidden for my taste, so they sometimes change when I don't want
them to and vice versa).


        Stefan

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.user

csiph-web