Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #267461

Re: SMART Uncorrectable_Error_Cnt rising - should I be worried?

Show all headers | View raw

On 2/15/24 07:41, The Wanderer wrote:
> On 2024-02-15 at 03:09, David Christensen wrote:
>> On 2/14/24 18:54, The Wanderer wrote:
>>> On 2024-01-09 at 14:22, The Wanderer wrote:
>>>> On 2024-01-09 at 14:01, Michael Kjörling wrote:
>>>>> On 9 Jan 2024 13:25 -0500, from The Wanderer
>>>>>> I've ordered a 22TB external drive
>>
>> Make?  Model?  How it is interfaced to your computer?
> 
> It's a WD Elements 20TB drive (I'm not sure where I got the 22 from);
> the back of the case has the part number WDBWLG0200HBK-X8 (or possibly
> -XB, the font is kind of ambiguous). The connection, per the packaging
> label, is USB-3.


Okay.


STFW it seems that drive uses CMR, which is good:

https://nascompares.com/answer/list-of-wd-cmr-and-smr-hard-drives-hdd/


> The big change of plans in the middle of my month-plus process was the
> decision to replace the entire 8-drive array with a 6-drive array, and
> the reason for that was because the 8-drive array left me with no open
> SATA ports to be able to connect spare drives in order to do drive
> replacements without needing to rebuild the whole shaboozle.


Having spare drive bays for RAID drive replacement is smart.


>> If you have a processor, memory, PCIe slot, and HBA to match those
>> SSD's, the performance of those SSD's should be very nice.
> 
> The CPU is a Ryxen 5 5600X. The RAM is G-Skill DDR4 2666MHz, in two 32GB
> DIMMs. I don't know how to assess PCIe slots and HBA, but the
> motherboard is an Asus ROG Crosshair VIII Dark Hero, which I think was
> the top-of-the-line enthusiast motherboard (with the port set my
> criteria called for) the year I built this machine.
> 
> I'm pretty sure my performance bottleneck for most things is the CPU (or
> the GPU, where that comes into play, which here it doesn't);
> storage-wise this seems so far to be at least as fast as what I had
> before, but it's hard to tell if it's faster.


It would not surprise me if the Intel D3-S4510 server drives are 
somewhat slower than the Samsung EVO 870 desktop drives.  But the Intel 
disks are designed to pull a heavy load all day for years on end.


Do you have a tool to monitor disk throughput and utilization?  I use 
Xfce panel Disk Performance Monitor applets and nmon(1) in a Terminal. 
Those plus CPU and memory monitoring tools should allow you to determine 
if your workload is CPU bound, memory bound, or I/O bound.


>> The key concept is "data lifetime". (Or alternatively, "destruction
>> policy".)
> 
> I can see that for when you have a tiered backup structure, and are
> looking at the lifetimes of each backup copy. For my live system, my
> intended data lifetime (outside of caches and data kept in /tmp) is
> basically "forever".


I try to group my data in anticipation of backup, etc., requirements. 
When I get it right, disaster preparedness and disaster recovery are easier.


>> I believe ZFS can do more hard links. (Much more?  Limited by
>> available storage space?)
> 
> I'm not sure, but I'll have to look into that, when I get to the point
> of trying to set up that tiered backup.
> ...
>>> ... without [rsnapshot hard link]
>>> deduplication there wouldn't have been enough space on the drive
>>> for more than the single copy, ...
>>
>> ZFS provides similarly useful results with built-in compression and
>> de-duplication.
> 
> I have the impression that there are risk and/or complexity aspects to
> it ...


Of course.  ZFS is sophisticated storage technology.  It looks 
deceptively simple when you are window shopping, but becomes non-trivial 
once you put real data on it, have to live with it 24x7, have to prepare 
for disasters, and have to recover from disasters.  There is a lot to 
learn and "more than enough rope to shoot yourself in the foot".


>> That sounds like an N-way merge problem ...
> 
> It does sound like that, yes. I'm already aware of jdupes, and of a few
> other tools (part of the work I already did in getting this far was
> rdfind, which is what I used to set up much of the hardlink
> deduplication that wound up biting me in the butt), but have not
> investigated LVM snapshot - and the idea of trying to script something
> like this, without an existing known-safe copy of the data to fall back
> on, leaves me *very* nervous.
> 
> Figuring out how to be prepared to roll back is the other uncertain and
> nervous-making part. In some cases it's straightforward enough, but
> doing it at the scale of the size of those copies is at best daunting.


https://html.duckduckgo.com/html?q=lvm%20snapshot%20restore


Use another computer or a VM to learn and practice LVM snapshots and 
restores, then use those skills when doing the N-way merge.


>>> out of the way, shutting down all parts of the system that might be
>>> writing to the affected filesystems, and manually copying out the
>>> final state of the *other* parts of those filesystems via rsync,
>>> bypassing rsnapshot. That was on Saturday the 10th.
>>>
>>> Then I grabbed copies of various metadata about the filesystems,
>>> the LVM, and the mdraid config; modified /etc/fstab to not mount
>>> them; deactivated the mdraid, and commented it out of
>>> /etc/mdadm/mdadm.conf; updated the initramfs; shut down; pulled all
>>> eight Samsung 870 EVO drives; installed six brand-new Intel
>>> data-center-class (or so I gather) SSDs;
>>
>> Which model?  What size?
> 
> lshw says they're INTEL SSDSCK2B03. The packaging says SSDSCK2B038T801.


Nice.


> IIRC, the product listing said they were 3.84 TB (or possibly TiB). lshw
> says 'size: 3567GiB (3840GB)'. IIRC, the tools I used to partition them
> and build the mdraid and so forth said 3.84 TB/TiB (not sure which), or
> 3840 GB/GiB (same).
> 
> For comparison, the 870 EVO drives - which were supposed to be 2TB
> apiece - were reported by some of those same tools as exactly 2000 of
> the same unit.
> 
> This does mean that I have more total space available in the new array
> than in the old one, 


8 @ 2 TB disks in RAID6 should provide 12 TB of capacity.

6 @ 3.84 TB disks in RAID6 should provide 15.36 TB of capacity.


> but I've tried to allocate only as much space as
> was in the old array, insofar as I could figure out how to do that in
> the limited environment I was working in. (The old array and/or LV setup
> had sizes listed along the lines of '<10TiB', but my best attempt at
> replicating it gave something which reports sizes along the lines of
> '10TiB', so I suspect that my current setup is actually slightly too
> large to fit on the old disks.)


LVM should give you the ability to resize logical volumes as required 
going forward.


>> Data integrity validation is tough without a mechanism.  Adding an
>> rsnapshot(1) postexec MD5SUMS, etc., file into the root of each
>> backup tree could solve this need, but could waste a lot of time and
>> energy checksumming files that have not changed.
> 
> AFAIK, all such things require you to be starting from a point with a
> known-good copy of the data, which is a luxury I don't currently have
> (as far as validating my current data goes). It's something to keep in
> mind when planning a more proper backup system, however.
> 
>> One of the reasons I switched to ZFS was because ZFS has built-in
>> data and metadata integrity checking (and repair; depending upon
>> redundancy).
> 
> I'm not sure I understand how this would be useful in the case I have at
> hand; that probably means that I'm not understanding the picture properly.


Bit rot is the enemy of forever:

https://html.duckduckgo.com/html?q=bit%20rot


The sooner you have MD5SUMS, etc., the sooner you  can start monitoring 
for file damage by any means.  The rsnapshot(1) community may already 
have a working solution.


>>> (This does leave me without having restored the read-only backup
>>> data from the old system state. I care less about that; I'll want
>>> it eventually, but it isn't important enough to warrant postponing
>>> getting the system back in working order.)
>>>
>>>
>>> I do still want/need to figure out what to do about an *actual*
>>> backup system, to external storage, since the rsnapshot thing
>>> apparently isn't going to be viable for my circumstance and use
>>> case. There is, however, now *time* to work on doing that, without
>>> living under the shadow of a known immediate/imminent data-loss
>>> hardware failure.
>>
>> rsync(1) should be able to copy backups onto an external HDD.
> 
> Yeah, but that only provides one tier of backup; 


It appears I misunderstood.


So, live data on 6 @ Intel SSD's and rsnapshot(1) backups on the 20 GB 
WD Elements USB HDD?


> the advantage of
> rsnapshot (or similar) is the multiple deduplicated tiers, which gives
> you options if it turns out the latest backup already included the
> damage you're trying to recover from.


If rsnapshot(1) is your chosen backup tool, you will want to learn 
everything you can about it.  Beyond RTFM rsnapshot(1), STFW I see:

https://rsnapshot.org/rsnapshot/docs/docbook/rest.html

http://www2.rsnapshot.org/

https://sourceforge.net/p/rsnapshot/mailman/rsnapshot-discuss/


> (USB-3 will almost certainly not be a viable option for an automatic
> scheduled backup of the sort rsnapshot's documentation suggests, because
> the *fastest* backup cycle I saw from my working with the data I had was
> over three hours, and the initial pass to copy the data out to the drive
> in the first place took nearly *20* hours. A cron job to run even an
> incremental backup even once a day, much less the several times a day
> suggested for the deeper rsnapshot tiers, would not be *remotely*
> workable in that sort of environment. Though on the flip side, that's
> not just a USB-3 bottleneck, but also the bottleneck of the spinning
> mechanical hard drive inside the external case...)


I think the Raspberry Pi, etc., users on this list live with USB storage 
and have found it to be reliable enough for personal and SOHO network use.


I have used the rsync(1) command for many years, both interactively and 
via scripts, but RTFM indicates rsync(1) can be run as a server.  I 
wonder if that would help performance, as a service can cache things 
that a command must find on every run (?).


David

Back to linux.debian.user | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-01-09 14:20 +0100
  Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Dan Ritter <dsr@randomstring.org> - 2024-01-09 16:00 +0100
    Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-01-09 16:30 +0100
      Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-01-09 19:50 +0100
  Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Curt <curty@free.fr> - 2024-01-09 17:20 +0100
    Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-01-09 19:20 +0100
  Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-01-09 17:30 +0100
    Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-01-09 19:30 +0100
      Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-01-09 20:10 +0100
        Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-01-09 20:30 +0100
          Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-02-15 04:00 +0100
            Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? songbird <songbird@anthive.com> - 2024-02-15 07:20 +0100
              Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-02-15 16:50 +0100
            Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? David Christensen <dpchrist@holgerdanske.com> - 2024-02-15 09:20 +0100
              Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-02-15 16:50 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-02-15 18:40 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? David Christensen <dpchrist@holgerdanske.com> - 2024-02-16 11:00 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? "Roy J. Tellason, Sr." <roy@rtellason.com> - 2024-02-16 20:00 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? David Christensen <dpchrist@holgerdanske.com> - 2024-02-16 21:00 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Gremlin <scott-andrews@columbus.rr.com> - 2024-02-16 22:50 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? "Roy J. Tellason, Sr." <roy@rtellason.com> - 2024-02-17 19:50 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? gene heskett <gheskett@shentel.net> - 2024-02-18 02:50 +0100
            Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? debian-user@howorth.org.uk - 2024-02-15 13:20 +0100
              Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-02-15 16:10 +0100
  Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? David Christensen <dpchrist@holgerdanske.com> - 2024-01-09 23:40 +0100
    Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? David Christensen <dpchrist@holgerdanske.com> - 2024-01-09 23:50 +0100
    Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-01-10 10:40 +0100
      Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? David Christensen <dpchrist@holgerdanske.com> - 2024-01-10 16:40 +0100
        Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Curt <curty@free.fr> - 2024-01-10 18:10 +0100
          Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-01-10 18:40 +0100
            Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? David Christensen <dpchrist@holgerdanske.com> - 2024-01-11 03:30 +0100
          Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Dan Ritter <dsr@randomstring.org> - 2024-01-10 18:40 +0100
          Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? David Christensen <dpchrist@holgerdanske.com> - 2024-01-11 03:20 +0100
            Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Dan Ritter <dsr@randomstring.org> - 2024-01-11 15:10 +0100
              Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? David Christensen <dpchrist@holgerdanske.com> - 2024-01-11 16:10 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Dan Ritter <dsr@randomstring.org> - 2024-01-11 19:30 +0100
              Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Stefan Monnier <monnier@iro.umontreal.ca> - 2024-01-11 21:30 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Michael Stone <mstone@debian.org> - 2024-01-12 00:00 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Dan Ritter <dsr@randomstring.org> - 2024-01-12 12:20 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-02-15 16:50 +0100

csiph-web