Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #268749 > unrolled thread

HDD long-term data storage with ensured integrity

Back to article view | Back to linux.debian.user

Contents

  HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-06 09:52 +0200
    Re: HDD long-term data storage with ensured integrity Marc SCHAEFER <schaefer@alphanet.ch> - 2024-04-08 11:40 +0200
      Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-08 20:30 +0200
        Re: HDD long-term data storage with ensured integrity Marc SCHAEFER <schaefer@alphanet.ch> - 2024-04-08 22:10 +0200
          Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-09 00:50 +0200
          Re: HDD long-term data storage with ensured integrity Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-03 13:30 +0200
            Re: HDD long-term data storage with ensured integrity Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-05-03 14:50 +0200
            Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-05-03 23:00 +0200
              Re: HDD long-term data storage with ensured integrity Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-04 09:50 +0200
                Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-20 14:40 +0200
                  Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Franco Martelli <martellif67@gmail.com> - 2024-05-21 20:50 +0200
                    Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-22 12:10 +0200
                    Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-22 09:00 +0200
                      Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Andy Smith <andy@strugglers.net> - 2024-05-22 12:20 +0200
                        Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-22 12:30 +0200
                      Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-22 09:00 +0200
                        Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Stefan Monnier <monnier@iro.umontreal.ca> - 2024-05-22 23:10 +0200
                          Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Marc SCHAEFER <schaefer@alphanet.ch> - 2024-05-23 09:00 +0200
                  Re: Debian bookwork / grub2 / LVM / RAID / dm-integrity fails to boot Franco Martelli <martellif67@gmail.com> - 2024-05-28 15:30 +0200
        Why LVM (was: HDD long-term data storage with ensured integrity) Stefan Monnier <monnier@iro.umontreal.ca> - 2024-04-08 23:10 +0200
          Re: Why LVM David Christensen <dpchrist@holgerdanske.com> - 2024-04-09 01:10 +0200
            Re: Why LVM Stefan Monnier <monnier@iro.umontreal.ca> - 2024-04-09 02:00 +0200
              Re: Why LVM David Christensen <dpchrist@holgerdanske.com> - 2024-04-09 12:20 +0200
    Re: HDD long-term data storage with ensured integrity piorunz <piorunz@gmx.com> - 2024-04-10 02:10 +0200
      Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-10 13:20 +0200
        Re: HDD long-term data storage with ensured integrity Curt <curty@free.fr> - 2024-04-10 17:00 +0200
        Re: HDD long-term data storage with ensured integrity Paul Leiber <paul@onlineschubla.de> - 2024-04-10 18:10 +0200
          Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-11 01:20 +0200
        Re: HDD long-term data storage with ensured integrity piorunz <piorunz@gmx.com> - 2024-04-12 17:20 +0200
          Re: HDD long-term data storage with ensured integrity David Christensen <dpchrist@holgerdanske.com> - 2024-04-13 05:10 +0200

Page 2 of 2 — ← Prev page 1 [2]

#268789 — Re: Why LVM

On 4/8/24 14:08, Stefan Monnier wrote:
> David Christensen [2024-04-08 11:28:04] wrote:
>> Why LVM?
> 
> Personally, I've been using LVM everywhere I can (i.e. everywhere
> except on my OpenWRT router, tho I've also used LVM there back when my
> router had an HDD.  I also use LVM on my 2GB USB rescue image).
> 
> To me the question is rather the reverse: why not?
> I basically see it as a more flexible form of partitioning.
> 
> Even in the worst cases where I have a single LV volume, I appreciate
> the fact that it forces me to name things, isolating me from issue
> linked to predicting the name of the device and the issues that plague
> UUIDs (the fact they're hard to remember, and that they're a bit too
> magical/hidden for my taste, so they sometimes change when I don't want
> them to and vice versa).
> 
> 
>          Stefan


If I have a hot-pluggable device (SD card, USB drive, hot-plug SATA/SAS 
drive and rack, etc.), can I put LVM on it such that when the device is 
connected to a Debian system with a graphical desktop (I use Xfce) an 
icon is displayed on the desktop that I can interact with to display the 
file systems in my file manager (Thunar)?


David

[toc] | [prev] | [next] | [standalone]

#268790 — Re: Why LVM

> If I have a hot-pluggable device (SD card, USB drive, hot-plug SATA/SAS
> drive and rack, etc.), can I put LVM on it such that when the device is
> connected to a Debian system with a graphical desktop (I use Xfce) an icon
> is displayed on the desktop that I can interact with to display the file
> systems in my file manager (Thunar)?

In the past: definitely not.  Currently: no idea.
I suspect not, because I think the behavior on disconnection is still
poor (you want to be extra careful to deactivate all the volumes on the
drive *before* removing it, otherwise they tend to linger "for ever").

I guess that's one area where partitions are still significantly better
than LVM.


        Stefan "who doesn't use much hot-plugging of mass storage"

[toc] | [prev] | [next] | [standalone]

#268791 — Re: Why LVM

On 4/8/24 16:54, Stefan Monnier wrote:
>> If I have a hot-pluggable device (SD card, USB drive, hot-plug SATA/SAS
>> drive and rack, etc.), can I put LVM on it such that when the device is
>> connected to a Debian system with a graphical desktop (I use Xfce) an icon
>> is displayed on the desktop that I can interact with to display the file
>> systems in my file manager (Thunar)?
> 
> In the past: definitely not.  Currently: no idea.
> I suspect not, because I think the behavior on disconnection is still
> poor (you want to be extra careful to deactivate all the volumes on the
> drive *before* removing it, otherwise they tend to linger "for ever").
> 
> I guess that's one area where partitions are still significantly better
> than LVM.
> 
> 
>          Stefan "who doesn't use much hot-plugging of mass storage"


Thank you for the clarification.  :-)


David

[toc] | [prev] | [next] | [standalone]

#268794

On 02/04/2024 13:53, David Christensen wrote:
>
> Does anyone have any comments or suggestions regarding how to use
> magnetic hard disk drives, commodity x86 computers, and Debian for
> long-term data storage with ensured integrity?

I use Btrfs, on all my systems, including some servers, with soft Raid1
and Raid10 modes (because these modes are considered stable and
production ready). I decided on Btrfs not ZFS, because Btrfs allows to
migrate drives on the fly while partition is live and heavily used,
replace them with different sizes and types, mixed capacities, change
Raid levels, change amount of drives too. I could go from single drive
to Raid10 on 4 drives and back while my data is 100% available at all times.
It saved my bacon many times, including hard checksum corruption on NVMe
drive which otherwise I would never know about. Thanks to Btrfs I
located the corrupted files, fixed them, got hardware replaced under
warranty.
Also helped with corrupted RAM: Btrfs just refused to save file because
saved copy couldn't match read checksum from the source due to RAM bit
flips. Diagnosed, then replaced memory, all good.
I like a lot when one of the drives get ATA reset for whatever reason,
and all other drives continue to read and write, I can continue using
the system for hours, if I even notice. Not possible in normal
circumstances without Raid. Once the problematic drive is back, or after
reboot if it's more serious, then I do "scrub" command and everything is
resynced again. If I don't do that, then Btrfs dynamically correct
checksum errors on the fly anyway.
And list goes on - I've been using Btrfs for last 5 years, not a single
problem to date, it survived hard resets, power losses, drive failures,
countless migrations.

> [1] https://github.com/openzfs/zfs/issues/15526
>
> [2] https://github.com/openzfs/zfs/issues/15933

Problems reported here are from Linux kernel 6.5 and 6.7 on Gentoo
system. Does this even affects Debian Stable with 6.1 LTS?

--
With kindest regards, Piotr.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/
⠈⠳⣄⠀⠀⠀⠀

[toc] | [prev] | [next] | [standalone]

#268800

On 4/9/24 17:08, piorunz wrote:
> On 02/04/2024 13:53, David Christensen wrote:
>>
>> Does anyone have any comments or suggestions regarding how to use
>> magnetic hard disk drives, commodity x86 computers, and Debian for
>> long-term data storage with ensured integrity?
> 
> I use Btrfs, on all my systems, including some servers, with soft Raid1
> and Raid10 modes (because these modes are considered stable and
> production ready). I decided on Btrfs not ZFS, because Btrfs allows to
> migrate drives on the fly while partition is live and heavily used,
> replace them with different sizes and types, mixed capacities, change
> Raid levels, change amount of drives too. I could go from single drive
> to Raid10 on 4 drives and back while my data is 100% available at all 
> times.
> It saved my bacon many times, including hard checksum corruption on NVMe
> drive which otherwise I would never know about. Thanks to Btrfs I
> located the corrupted files, fixed them, got hardware replaced under
> warranty.
> Also helped with corrupted RAM: Btrfs just refused to save file because
> saved copy couldn't match read checksum from the source due to RAM bit
> flips. Diagnosed, then replaced memory, all good.
> I like a lot when one of the drives get ATA reset for whatever reason,
> and all other drives continue to read and write, I can continue using
> the system for hours, if I even notice. Not possible in normal
> circumstances without Raid. Once the problematic drive is back, or after
> reboot if it's more serious, then I do "scrub" command and everything is
> resynced again. If I don't do that, then Btrfs dynamically correct
> checksum errors on the fly anyway.
> And list goes on - I've been using Btrfs for last 5 years, not a single
> problem to date, it survived hard resets, power losses, drive failures,
> countless migrations.


Those sound like some compelling features.


I believe the last time I tried Btrfs was Debian 9 (?).  I ran into 
problems because I did not do the required manual maintenance 
(rebalancing).  Does the Btrfs in Debian 11 or Debian 12 still require 
manual maintenance?  If so, what and how often?


>> [1] https://github.com/openzfs/zfs/issues/15526
>>
>> [2] https://github.com/openzfs/zfs/issues/15933
> 
> Problems reported here are from Linux kernel 6.5 and 6.7 on Gentoo
> system. Does this even affects Debian Stable with 6.1 LTS?


I do not know.


> -- 
> With kindest regards, Piotr.
> 
> ⢀⣴⠾⠻⢶⣦⠀
> ⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
> ⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/
> ⠈⠳⣄⠀⠀⠀⠀


David

[toc] | [prev] | [next] | [standalone]

#268803

On 2024-04-10, David Christensen <dpchrist@holgerdanske.com> wrote:
>> 
>> I use Btrfs, on all my systems, including some servers, with soft Raid1
>> and Raid10 modes (because these modes are considered stable and
>> production ready). I decided on Btrfs not ZFS, because Btrfs allows to
>> migrate drives on the fly while partition is live and heavily used,
>> replace them with different sizes and types, mixed capacities, change
>> Raid levels, change amount of drives too. I could go from single drive
>> to Raid10 on 4 drives and back while my data is 100% available at all 
>> times.
>> It saved my bacon many times, including hard checksum corruption on NVMe
>> drive which otherwise I would never know about. Thanks to Btrfs I
>> located the corrupted files, fixed them, got hardware replaced under
>> warranty.
>> Also helped with corrupted RAM: Btrfs just refused to save file because
>> saved copy couldn't match read checksum from the source due to RAM bit
>> flips. Diagnosed, then replaced memory, all good.
>> I like a lot when one of the drives get ATA reset for whatever reason,
>> and all other drives continue to read and write, I can continue using
>> the system for hours, if I even notice. Not possible in normal
>> circumstances without Raid. Once the problematic drive is back, or after
>> reboot if it's more serious, then I do "scrub" command and everything is
>> resynced again. If I don't do that, then Btrfs dynamically correct
>> checksum errors on the fly anyway.
>> And list goes on - I've been using Btrfs for last 5 years, not a single
>> problem to date, it survived hard resets, power losses, drive failures,
>> countless migrations.
>
>
> Those sound like some compelling features.

I don't believe in immortality. After many a summer dies the swan.

[toc] | [prev] | [next] | [standalone]

#268804

Am 10.04.2024 um 13:10 schrieb David Christensen:
> On 4/9/24 17:08, piorunz wrote:
>> On 02/04/2024 13:53, David Christensen wrote:
>>>
>>> Does anyone have any comments or suggestions regarding how to use
>>> magnetic hard disk drives, commodity x86 computers, and Debian for
>>> long-term data storage with ensured integrity?
>>
>> I use Btrfs, on all my systems, including some servers, with soft Raid1
>> and Raid10 modes (because these modes are considered stable and
>> production ready). I decided on Btrfs not ZFS, because Btrfs allows to
>> migrate drives on the fly while partition is live and heavily used,
>> replace them with different sizes and types, mixed capacities, change
>> Raid levels, change amount of drives too. I could go from single drive
>> to Raid10 on 4 drives and back while my data is 100% available at all 
>> times.
>> It saved my bacon many times, including hard checksum corruption on NVMe
>> drive which otherwise I would never know about. Thanks to Btrfs I
>> located the corrupted files, fixed them, got hardware replaced under
>> warranty.
>> Also helped with corrupted RAM: Btrfs just refused to save file because
>> saved copy couldn't match read checksum from the source due to RAM bit
>> flips. Diagnosed, then replaced memory, all good.
>> I like a lot when one of the drives get ATA reset for whatever reason,
>> and all other drives continue to read and write, I can continue using
>> the system for hours, if I even notice. Not possible in normal
>> circumstances without Raid. Once the problematic drive is back, or after
>> reboot if it's more serious, then I do "scrub" command and everything is
>> resynced again. If I don't do that, then Btrfs dynamically correct
>> checksum errors on the fly anyway.
>> And list goes on - I've been using Btrfs for last 5 years, not a single
>> problem to date, it survived hard resets, power losses, drive failures,
>> countless migrations.
> 
> 
> Those sound like some compelling features.
> 
> 
> I believe the last time I tried Btrfs was Debian 9 (?).  I ran into 
> problems because I did not do the required manual maintenance 
> (rebalancing).  Does the Btrfs in Debian 11 or Debian 12 still require 
> manual maintenance?  If so, what and how often?

Scrub and balance are actions which have been recommended. I am using 
btrfsmaintenance scripts [1][2] to automate this. I am doing a weekly 
balance and a monthly scrub. After some reading today, I am getting 
unsure if this is approach is correct, especially if balance is 
necessary anymore (it usually doesn't find anything to do anyway), so 
please take these periods with caution. My main message is that such 
operations can be automated using the linked scripts.

Best regards,

Paul

[1] https://packages.debian.org/bookworm/btrfsmaintenance
[2] https://github.com/kdave/btrfsmaintenance

[toc] | [prev] | [next] | [standalone]

#268806

On 4/10/24 08:49, Paul Leiber wrote:
> Am 10.04.2024 um 13:10 schrieb David Christensen:
>> Does the Btrfs in Debian 11 or Debian 12 still require 
>> manual maintenance?  If so, what and how often?
> 
> Scrub and balance are actions which have been recommended. I am using 
> btrfsmaintenance scripts [1][2] to automate this. I am doing a weekly 
> balance and a monthly scrub. After some reading today, I am getting 
> unsure if this is approach is correct, especially if balance is 
> necessary anymore (it usually doesn't find anything to do anyway), so 
> please take these periods with caution. My main message is that such 
> operations can be automated using the linked scripts.
> 
> Best regards,
> 
> Paul
> 
> [1] https://packages.debian.org/bookworm/btrfsmaintenance
> [2] https://github.com/kdave/btrfsmaintenance


Thank you.  Those scripts should be useful.


David

[toc] | [prev] | [next] | [standalone]

#268827

On 10/04/2024 12:10, David Christensen wrote:
> Those sound like some compelling features.
>
>
> I believe the last time I tried Btrfs was Debian 9 (?).  I ran into
> problems because I did not do the required manual maintenance
> (rebalancing).  Does the Btrfs in Debian 11 or Debian 12 still require
> manual maintenance?  If so, what and how often?

I don't do balance at all, it's not required.

Scrub is recommended, because it will detect any bit-rot due to hardware
errors on HDD media. It scans the entire surface of allocated sectors on
all drives. I do scrub usually monthly.

--
With kindest regards, Piotr.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/
⠈⠳⣄⠀⠀⠀⠀

[toc] | [prev] | [next] | [standalone]

#268830

On 4/12/24 08:14, piorunz wrote:
> On 10/04/2024 12:10, David Christensen wrote:
>> Those sound like some compelling features.
>>
>>
>> I believe the last time I tried Btrfs was Debian 9 (?).  I ran into
>> problems because I did not do the required manual maintenance
>> (rebalancing).  Does the Btrfs in Debian 11 or Debian 12 still require
>> manual maintenance?  If so, what and how often?
> 
> I don't do balance at all, it's not required.
> 
> Scrub is recommended, because it will detect any bit-rot due to hardware
> errors on HDD media. It scans the entire surface of allocated sectors on
> all drives. I do scrub usually monthly.


Thank you for the information.


David

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.debian.user

csiph-web