Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #226257 > unrolled thread

Disks renamed after update to 'testing'...?

Back to article view | Back to linux.debian.user

Contents

  Disks renamed after update to 'testing'...? "hobie of RMN" <hobie@rumormillnews.com> - 2020-08-18 01:50 +0200
    Re: Disks renamed after update to 'testing'...? David Christensen <dpchrist@holgerdanske.com> - 2020-08-18 03:30 +0200
      Re: Disks renamed after update to 'testing'...? "hobie of RMN" <hobie@rumormillnews.com> - 2020-08-18 06:10 +0200
        Re: Disks renamed after update to 'testing'...? john doe <johndoe65534@mail.com> - 2020-08-18 08:40 +0200
          Re: Disks renamed after update to 'testing'...? didier gaumet <didier.gaumet@gmail.com> - 2020-08-18 10:30 +0200
            Re: Disks renamed after update to 'testing'...? Nicolas George <george@nsup.org> - 2020-08-18 11:40 +0200
              Re: Disks renamed after update to 'testing'...? "Thomas Schmitt" <scdbackup@gmx.net> - 2020-08-18 12:20 +0200
            Re: Disks renamed after update to 'testing'...? "Thomas Schmitt" <scdbackup@gmx.net> - 2020-08-18 11:40 +0200
        Re: Disks renamed after update to 'testing'...? David Christensen <dpchrist@holgerdanske.com> - 2020-08-18 12:10 +0200
    Re: Disks renamed after update to 'testing'...? "Rick Thomas" <rick.thomas@pobox.com> - 2020-08-18 10:40 +0200
      Re: Disks renamed after update to 'testing'...? Urs Thuermann <urs@isnogud.escape.de> - 2020-08-18 20:40 +0200
        Re: Disks renamed after update to 'testing'...? David Christensen <dpchrist@holgerdanske.com> - 2020-08-18 22:00 +0200
          Re: Disks renamed after update to 'testing'...? Urs Thuermann <urs@isnogud.escape.de> - 2020-08-19 08:10 +0200
            Re: Disks renamed after update to 'testing'...? David Christensen <dpchrist@holgerdanske.com> - 2020-08-19 10:40 +0200
              Re: Disks renamed after update to 'testing'...? Urs Thuermann <urs@isnogud.escape.de> - 2020-08-19 12:10 +0200
                Re: Disks renamed after update to 'testing'...? David Christensen <dpchrist@holgerdanske.com> - 2020-08-19 23:50 +0200
                  Re: Disks renamed after update to 'testing'...? <tomas@tuxteam.de> - 2020-08-20 09:50 +0200
                    Re: Disks renamed after update to 'testing'...? Joe <joe@jretrading.com> - 2020-08-20 12:20 +0200
                    Re: Disks renamed after update to 'testing'...? rhkramer@gmail.com - 2020-08-20 17:40 +0200
                      Re: Disks renamed after update to 'testing'...? David Christensen <dpchrist@holgerdanske.com> - 2020-08-20 22:40 +0200
                        Re: Disks renamed after update to 'testing'...? Dan Ritter <dsr@randomstring.org> - 2020-08-20 23:40 +0200
                          Re: Disks renamed after update to 'testing'...? Andy Smith <andy@strugglers.net> - 2020-08-21 00:00 +0200
                            Re: Disks renamed after update to 'testing'...? David Christensen <dpchrist@holgerdanske.com> - 2020-08-21 11:50 +0200
                        Re: Disks renamed after update to 'testing'...? <tomas@tuxteam.de> - 2020-08-21 10:40 +0200
        Re: Disks renamed after update to 'testing'...? Urs Thuermann <urs@isnogud.escape.de> - 2020-08-19 09:00 +0200

Page 2 of 2 — ← Prev page 1 [2]

#226338

David Christensen wrote: 
> On 2020-08-20 08:32, rhkramer@gmail.com wrote:
> I have been pondering bit-rot mitigation on non-checksumming filesystems.
> 
> 
> Some people have mentioned md RAID.  tomas has mentioned LUKS.  I believe
> both of them add checksums to the contained contents.  So, bit-rot within a
> container should be caught by the container driver.

This is incorrect. The systems that checksum every write and
recalculate and match on every read are BTRFS and ZFS.

LVM, LUKS, and mdadm do not.

-dsr-

[toc] | [prev] | [next] | [standalone]

#226339

Hello,

On Thu, Aug 20, 2020 at 05:30:20PM -0400, Dan Ritter wrote:
> David Christensen wrote: 
> > Some people have mentioned md RAID.  tomas has mentioned LUKS.  I believe
> > both of them add checksums to the contained contents.  So, bit-rot within a
> > container should be caught by the container driver.
> 
> This is incorrect. The systems that checksum every write and
> recalculate and match on every read are BTRFS and ZFS.
> 
> LVM, LUKS, and mdadm do not.

Indeed, although dm-integrity can now be used with LUKS (with or
without encryption) to add checksums that force a read error when
they don't match. When there is redundancy (e.g. LVM or MD) a read
can then come from a good copy and the bad copy will be repaired.

Here is a practical example:

    https://gist.github.com/MawKKe/caa2bbf7edcc072129d73b61ae7815fb

I haven't yet used it in production but if anyone has I would be
really interested to see a with and without comparison of
performance.

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]

#226350

On 2020-08-20 14:58, Andy Smith wrote:

> ... dm-integrity can now be used with LUKS (with or
> without encryption) to add checksums that force a read error when
> they don't match. When there is redundancy (e.g. LVM or MD) a read
> can then come from a good copy and the bad copy will be repaired.

So, LVM and md RAID do not checksum data blocks -- they need a disk to 
report a failed read before they realize something is wrong?


> Here is a practical example:
> 
>      https://gist.github.com/MawKKe/caa2bbf7edcc072129d73b61ae7815fb


Thanks for the link.  I recall reading about dm-integrity sometime in 
the past, but my rotten old brain must have screwed it up.  At least I 
wasn't hallucinating when I guessed that LUKS had checksums.  8-O


My Debian 9 and cryptsetup are now looking even older and more inferior.


Does the Debian 10 installer make use of, or allow the use of, 
dm-integrity and/or cryptsetup with the --integrity option?  If not, 
does the necessary software exist in the d-i kernel, BusyBox, whatever, 
so I can do it by hand with the rescue console/ alternate virtual console?


David


p.s.  Similar statements can be made for OpenZFS native encryption:

https://openzfs.org/wiki/ZFS-Native_Encryption

https://openzfs.org/wiki/OpenZFS_Developer_Summit_2016#Presentations

[toc] | [prev] | [next] | [standalone]

#226346

[Multipart message — attachments visible in raw view] — view raw

On Thu, Aug 20, 2020 at 01:34:58PM -0700, David Christensen wrote:
> On 2020-08-20 08:32, rhkramer@gmail.com wrote:
> >On Thursday, August 20, 2020 03:43:55 AM tomas@tuxteam.de wrote:
> >>Contraty to the other (very valid) points, my backups are always on
> >>a LUKS drive, no partition table. Rationale is, should I lose it, the
> >>less visible information the better. Best if it looks like a broken
> >>USB stick. No partition table looks (nearly) broken :-)
> 
> I always use a partition table, to reduce the chance of confusing
> myself.  ;-)
> 
> 
> >I have two questions:
> >
> >    * I suppose that means you create the LUKS drive on, e.g., /dev/sdc rather
> >than, for example, /dev/sdc<n>?  (I suppose that should be easy to do.)

Exactly.

> >    * But, I'm wondering, how much bit rot would it take to make the entire
> >backup unusable, and what kind of precautions do you take (or could be taken)
> >to avoid that?

I have no current strategy for silent [1] bit rot. For file system
consistency, I do run from time to time an fsck after opening the
LUKS and before mounting (We are talking about roughly 60..70 GB;
were we talking about 100..1000 times as much, active bitrot
mitigation might sound more compelling).

> I have been pondering bit-rot mitigation on non-checksumming filesystems.

The big ones have that; and for really huge amounts of data (where
some corners of your data might rest unseen for years (say hundreds
of TB or so), it does make sense.

In my case, I consider the backup just as something which i expect
to "fail early" and "fail loudly". In the "normal" case it is perfectly
disposable :-)

> Some people have mentioned md RAID.  tomas has mentioned LUKS.  I
> believe both of them add checksums to the contained contents.  So,
> bit-rot within a container should be caught by the container driver.

Don't know about that, to be honest: I count on the ext4 beneath the
LUKS to catch any nasties (and to issue an early warning when the
USB stick starts degrading -- I'm still a bit queasy how cheap a
128GB USB stick can be).

> In the case of md RAID, the driver should respond by fetching the
> data from another drive and then dealing with the bad block(s); the
> application should not see any error (?).  I assume LVM RAID would
> respond like md RAID (?).

Yes. That's why I reserve RAID for "high availability" case: you
want to keep running after a failure, and your customer doesn't
notice (it would make sense to think about whether this is the
best level to introduce redundancy, but I disgress).

>   In the case of LUKS, the driver has no
> redundant data (?) and will have no choice but to report an error to
> the application (?).  I would guess LVM non-RAID would behave
> similarly (?).

Exactly. For the backup scenario, the whole backup /is/ the redundant
data. If the probability of failure of your main system in some
given time interval T is, say, 10e-7, and that of your backup's in
the same time interval is, say 10e-5 (cheaper hardware, and that),
you're looking into a catastrophe with a prob of 10e-12. If you
want to better that, use two separate backup media, then you are
into 10e-17 [2].

Cheers
[1] silent meaning some bit flips in file content, without the
   file system noticing. Which on ext4 is quite possible and
   btrfs, e.g. can (reasonably) guard against.

[2] This is, of course, "economist maths", the one which lead
   to the 2008-2009 crash: assume all those bad events are
   independent. If my house burns down, my computer is in there,
   and my only backup on a stick is in my pocket...

 - t
> 
> 
> For all three -- md, LUKS, LVM -- I don't know what happens for bit
> rot outside the container (e.g. in the container metadata).
> 
> 
> David
> 

[toc] | [prev] | [next] | [standalone]

#226288

Urs Thuermann <urs@isnogud.escape.de> writes:

> IMO the best solution is to use LVM.  I use it since 2001 on most
> drives and I don't have partitions.  And I prefer to use device names
> over using the *UUID or *LABEL prefixes.  With LVM, device names are
> predictable /dev/mapper/<vg-name>-<lv-name> with symlinks
> /dev/<vg-name>/<lv-name>.

Following up myself: The reason I prefer stable device names instead
of UUIDs or LABELs is that device names show up in some places even if
you use UUID or LABEL in /etc/fstab or in your command line:

On my laptop I have UUID in /etc/fstab but df still shows the device
name:

$ grep -w / /etc/fstab
# / was on /dev/nvme0n1p2 during installation
UUID=c73ff331-0ff5-44fb-8aef-228e64a96175 /               ext4    errors=remount-ro 0       1
$ df | grep -w /
/dev/nvme0n1p2  237470384 107725172 117659352  48% /

On my server with LVM I get:

$ grep -w / /etc/fstab
/dev/mapper/vg0-root   /               ext4    errors=remount-ro 0       1
$ df | grep -w /
/dev/mapper/vg0-root              2031440     659132   1261556  35% /


urs

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.debian.user

csiph-web