Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #268794

Re: HDD long-term data storage with ensured integrity

From piorunz <piorunz@gmx.com>
Newsgroups linux.debian.user
Subject Re: HDD long-term data storage with ensured integrity
Date 2024-04-10 02:10 +0200
Message-ID <IrtHj-5edB-3@gated-at.bofh.it> (permalink)
References <Iq8YI-4oGU-1241@gated-at.bofh.it>
Organization linux.* mail to news gateway

Show all headers | View raw

On 02/04/2024 13:53, David Christensen wrote:
>
> Does anyone have any comments or suggestions regarding how to use
> magnetic hard disk drives, commodity x86 computers, and Debian for
> long-term data storage with ensured integrity?

I use Btrfs, on all my systems, including some servers, with soft Raid1
and Raid10 modes (because these modes are considered stable and
production ready). I decided on Btrfs not ZFS, because Btrfs allows to
migrate drives on the fly while partition is live and heavily used,
replace them with different sizes and types, mixed capacities, change
Raid levels, change amount of drives too. I could go from single drive
to Raid10 on 4 drives and back while my data is 100% available at all times.
It saved my bacon many times, including hard checksum corruption on NVMe
drive which otherwise I would never know about. Thanks to Btrfs I
located the corrupted files, fixed them, got hardware replaced under
warranty.
Also helped with corrupted RAM: Btrfs just refused to save file because
saved copy couldn't match read checksum from the source due to RAM bit
flips. Diagnosed, then replaced memory, all good.
I like a lot when one of the drives get ATA reset for whatever reason,
and all other drives continue to read and write, I can continue using
the system for hours, if I even notice. Not possible in normal
circumstances without Raid. Once the problematic drive is back, or after
reboot if it's more serious, then I do "scrub" command and everything is
resynced again. If I don't do that, then Btrfs dynamically correct
checksum errors on the fly anyway.
And list goes on - I've been using Btrfs for last 5 years, not a single
problem to date, it survived hard resets, power losses, drive failures,
countless migrations.

> [1] https://github.com/openzfs/zfs/issues/15526
>
> [2] https://github.com/openzfs/zfs/issues/15933

Problems reported here are from Linux kernel 6.5 and 6.7 on Gentoo
system. Does this even affects Debian Stable with 6.1 LTS?

--
With kindest regards, Piotr.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/
⠈⠳⣄⠀⠀⠀⠀

Back to linux.debian.user | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-06 09:52 +0200
  Re: HDD long-term data storage with ensured integrity Marc SCHAEFER <schaefer@alphanet.ch> - 2024-04-08 11:40 +0200
    Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-08 20:30 +0200
      Re: HDD long-term data storage with ensured integrity Marc SCHAEFER <schaefer@alphanet.ch> - 2024-04-08 22:10 +0200
        Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-09 00:50 +0200
        Re: HDD long-term data storage with ensured integrity Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-03 13:30 +0200
          Re: HDD long-term data storage with ensured integrity Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-05-03 14:50 +0200
          Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-05-03 23:00 +0200
            Re: HDD long-term data storage with ensured integrity Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-04 09:50 +0200
              Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-20 14:40 +0200
                Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Franco Martelli <martellif67@gmail.com> - 2024-05-21 20:50 +0200
                Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-22 12:10 +0200
                Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-22 09:00 +0200
                Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Andy Smith <andy@strugglers.net> - 2024-05-22 12:20 +0200
                Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-22 12:30 +0200
                Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-22 09:00 +0200
                Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Stefan Monnier <monnier@iro.umontreal.ca> - 2024-05-22 23:10 +0200
                Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-23 09:00 +0200
                Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Franco Martelli <martellif67@gmail.com> - 2024-05-28 15:30 +0200
      Why LVM (was: HDD long-term data storage with ensured integrity) Stefan Monnier <monnier@iro.umontreal.ca> - 2024-04-08 23:10 +0200
        Re: Why LVM David Christensen <dpchrist@holgerdanske.com> - 2024-04-09 01:10 +0200
          Re: Why LVM Stefan Monnier <monnier@iro.umontreal.ca> - 2024-04-09 02:00 +0200
            Re: Why LVM David Christensen <dpchrist@holgerdanske.com> - 2024-04-09 12:20 +0200
  Re: HDD long-term data storage with ensured integrity piorunz <piorunz@gmx.com> - 2024-04-10 02:10 +0200
    Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-10 13:20 +0200
      Re: HDD long-term data storage with ensured integrity Curt <curty@free.fr> - 2024-04-10 17:00 +0200
      Re: HDD long-term data storage with ensured integrity Paul Leiber <paul@onlineschubla.de> - 2024-04-10 18:10 +0200
        Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-11 01:20 +0200
      Re: HDD long-term data storage with ensured integrity piorunz <piorunz@gmx.com> - 2024-04-12 17:20 +0200
        Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-13 05:10 +0200

csiph-web