Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #269185

Re: HDD long-term data storage with ensured integrity

Show all headers | View raw

On 5/3/24 04:26, Marc SCHAEFER wrote:
> On Mon, Apr 08, 2024 at 10:04:01PM +0200, Marc SCHAEFER wrote:
>> For off-site long-term offline archiving, no, I am not using RAID.
> 
> Now, as I had to think a bit about ONLINE integrity, I found this
> comparison:
> 
> https://github.com/t13a/dm-integrity-benchmarks
> 
> Contenders are btrfs, zfs, and notably ext4+dm-integrity+dm-raid
> 
> I tend to have a biais favoring UNIX layered solutions against
> "all-into-one" solutions, and it seems that performance-wise,
> it's also quite good.
> 
> I wrote this script to convince myself of auto-correction
> of the ext4+dm-integrity+dm-raid layered approach.


Thank you for devising a benchmark and posting some data.  :-)


FreeBSD also offers a layered solution.  From the top down:

* UFS2 file system, which supports snapshots (requires partitions with 
soft updates enabled).

* gpart(8) for partitions (volumes).

* graid(8) for redundancy and self-healing.

* geli(8) providers with continuous integrity checking.


AFAICT the FreeBSD stack is mature and production quality, which I find 
very appealing.  But the feature set is not as sophisticated as ZFS, 
which leaves me wanting.  Notably, I have not found a way to replicate 
UFS snapshots directly -- the best I can dream up is synchronizing a 
snapshot to a backup UFS2 filesystem and then taking a snapshot with the 
same name.


I am coming to the conclusion that the long-term survivability of data 
requires several components -- good live file system, good backups, good 
archives, continuous internal integrity checking with self-healing, 
periodic external integrity checking (e.g. mtree(1)) with some form of 
recovery (e.g. manual), etc.. If I get the other pieces right, I could 
go with OpenZFS for the live and backup systems, and worry less about 
data corruption bugs.


David

Back to linux.debian.user | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-06 09:52 +0200
  Re: HDD long-term data storage with ensured integrity Marc SCHAEFER <schaefer@alphanet.ch> - 2024-04-08 11:40 +0200
    Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-08 20:30 +0200
      Re: HDD long-term data storage with ensured integrity Marc SCHAEFER <schaefer@alphanet.ch> - 2024-04-08 22:10 +0200
        Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-09 00:50 +0200
        Re: HDD long-term data storage with ensured integrity Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-03 13:30 +0200
          Re: HDD long-term data storage with ensured integrity Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-05-03 14:50 +0200
          Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-05-03 23:00 +0200
            Re: HDD long-term data storage with ensured integrity Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-04 09:50 +0200
              Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-20 14:40 +0200
                Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Franco Martelli <martellif67@gmail.com> - 2024-05-21 20:50 +0200
                Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-22 12:10 +0200
                Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-22 09:00 +0200
                Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Andy Smith <andy@strugglers.net> - 2024-05-22 12:20 +0200
                Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-22 12:30 +0200
                Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-22 09:00 +0200
                Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Stefan Monnier <monnier@iro.umontreal.ca> - 2024-05-22 23:10 +0200
                Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-23 09:00 +0200
                Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Franco Martelli <martellif67@gmail.com> - 2024-05-28 15:30 +0200
      Why LVM (was: HDD long-term data storage with ensured integrity) Stefan Monnier <monnier@iro.umontreal.ca> - 2024-04-08 23:10 +0200
        Re: Why LVM David Christensen <dpchrist@holgerdanske.com> - 2024-04-09 01:10 +0200
          Re: Why LVM Stefan Monnier <monnier@iro.umontreal.ca> - 2024-04-09 02:00 +0200
            Re: Why LVM David Christensen <dpchrist@holgerdanske.com> - 2024-04-09 12:20 +0200
  Re: HDD long-term data storage with ensured integrity piorunz <piorunz@gmx.com> - 2024-04-10 02:10 +0200
    Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-10 13:20 +0200
      Re: HDD long-term data storage with ensured integrity Curt <curty@free.fr> - 2024-04-10 17:00 +0200
      Re: HDD long-term data storage with ensured integrity Paul Leiber <paul@onlineschubla.de> - 2024-04-10 18:10 +0200
        Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-11 01:20 +0200
      Re: HDD long-term data storage with ensured integrity piorunz <piorunz@gmx.com> - 2024-04-12 17:20 +0200
        Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-13 05:10 +0200

csiph-web