SMART Uncorrectable_Error_Cnt rising - should I be worried?

From	The Wanderer <wanderer@fastmail.fm>
Newsgroups	linux.debian.user
Subject	SMART Uncorrectable_Error_Cnt rising - should I be worried?
Date	2024-01-09 14:20 +0100
Message-ID	<HUkbn-1U36-5@gated-at.bofh.it> (permalink)
Organization	This space intentionally left blank.

Show all headers | View raw

[Multipart message — attachments visible in raw view] - view raw

This is not directly Debian-related, except insofar as the system
involved is running Debian, but we've already had a somewhat similar
thread recently and this forum is as likely as any I'm aware of to have
people who might have the experience to address the question(s). I would
be open to recommendations for alternate / better forums for this
inquiry, if people have such.


For background: I have an eight-drive RAID-6 array of 2TB SSDs, built
back in early-to-mid 2021.

Until recently, as far as I'm aware there have not been any problems
related to it.


Within the past few weeks, I got root-mail notifications from smartd
that the ATA error count on two of the drives had increased - one from 0
to a fairly low value (I think between 10 and 20), the other from 0 to
1. I figured this was nothing to worry about - because of the relatively
low values, because the other drives had not shown any such thing, and
because of the expected stability and lifetime of good-quality SSDs.


On Sunday (two days ago), I got root-mail notifications from smartd
about *all* of the drives in the array. This time, the total error
counts had gone up to values in the multiple hundreds per drive. Since
then (yesterday), I've also gotten further notification mails about at
least one of the drives increasing further. So far today I have not
gotten any such notifications.

One thing I don't know, which may or may not be important, is whether
these alert mails are being triggered when the error-count increase
happens, or when a scheduled check of some type is run. If it's the
latter, then it might be that there's a monthly check and that's the
reason why all eight drives got mails sent at once, but if it's the
former, then the so-close-in-time alerts from all eight drives would
seem more likely to reflect a real problem.


I've looked at the SMART attributes for the drives, and am having a hard
time determining whether or not there's anything worth being actually
concerned about here. Some of the information I'm seeing seems to
suggest yes, but other information seems to suggest no.

Relevant-seeming excerpts from the output of 'smartctl -a' on one of the
drives is attached (rather than inline, to avoid line-wrapping). I can
provide full output of that command for that drive, or even for all of
the drives, if desired.


Things that seem to suggest that there may be reason to be concerned
include, but may not be limited to:

The Uncorrectable_Error_Cnt, which is the value referenced by the alert
mails, has risen well above its apparent previous value of 0, and signs
are that it may be going to keep rising.

The Runtime_Bad_Block count is nonzero.

The ECC_Error_Rate is nonzero (and, at least in the case of this
specific drive, also equal to the Uncorrectable_Error_Cnt).

Most of the attributes are listed as of type "Old_age". That strikes me
as unexpected; two and a half years of mostly-read-based operation does
not seem like enough to qualify a SSD as "old", although my expectations
here may well be off. (I would be inclined to expect five-to-ten years
of operation out of a non-defective drive, assuming reasonable physical
treatment otherwise, if not considerably more.)

As mentioned above, the increase in Uncorrectable_Error_Cnt has happened
at nearly the same time (relative to drive installation date) for all
the drives, and for some of the drives it seems to be continuing to
increase.


I don't know how to interpret the "Pre-fail" notation for the other
attributes. That terminology could be intended to mean "This drive has
entered the final stage before failure, and its failure is expected to
be imminent" - or it could equally well be the status that the
attributes *start* in, with the intended meaning "This drive has not yet
reached a stage where there is any reason to think it might fail".


Things that seem to suggest that there may *not* be a reason to be
concerned include, but may not be limited to:

The "VALUE" column for each of the attributes remains high; most are in
the range from 098 to 100, and excluding the Airflow_Temperature_Cel
figure, the lowest is 095, for Power_On_Hours. From what I've managed to
find in reading online, this column is typically a percentage value,
with lower percentages indicating that the drive is closer to failure.

The Total_LBAs_Written value, when combined with the Sector Size,
results (if my math is correct) in a total-data-written figure of
between 3TB and 4TB. That should be *well* under the advertised write
endurance of this drive, given that the drive is 2TB and (both IIRC and
from what I've found in reading up on such things again after these
errors started to occur) those advertised values for similar-capacity
drives seem to start in the hundreds of TB and go up.


So... as the Subject asks, should I be worried? How do I interpret these
results, and at what point do they start to reflect something to take
action over? If there is not reason to be worried, what *do* these
alerts indicate, and at what point *should* I start to be worried about
them?

I already *am* worried, to the point of having heartburn and difficulty
sleeping over the possibility of data loss (there's enough on here that
external backup would be somewhat difficult to arrange), but I'm not
sure whether or not that is warranted.

My default plan is to identify an appropriate model and buy a pair of
replacement drives, but not install them yet; buy another two drives
every six months, until I have a full replacement set; and start failing
drives out of the RAID array and installing replacements as soon as one
either fails, or looks like it's imminently about to fail. But if the
mass notification mails indicate that all eight are nearing failure,
that might not be enough - and if they don't indicate any likelihood of
failure this year, then buying replacement drives yet might be
premature.

What drives I choose to buy as replacement would also be influenced by
how likely it is that this indicates impending failure. If it doesn't,
then drives similar to what I already have would probably still be
appropriate; if it does, then I'm going to want to go up-market and buy
long-endurance drives intended for high uptime - i.e., data-center
storage drives, which are likely to be more expensive.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw

Back to linux.debian.user | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-01-09 14:20 +0100
  Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Dan Ritter <dsr@randomstring.org> - 2024-01-09 16:00 +0100
    Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-01-09 16:30 +0100
      Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-01-09 19:50 +0100
  Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Curt <curty@free.fr> - 2024-01-09 17:20 +0100
    Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-01-09 19:20 +0100
  Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-01-09 17:30 +0100
    Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-01-09 19:30 +0100
      Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-01-09 20:10 +0100
        Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-01-09 20:30 +0100
          Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-02-15 04:00 +0100
            Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? songbird <songbird@anthive.com> - 2024-02-15 07:20 +0100
              Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-02-15 16:50 +0100
            Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? David Christensen <dpchrist@holgerdanske.com> - 2024-02-15 09:20 +0100
              Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-02-15 16:50 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-02-15 18:40 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? David Christensen <dpchrist@holgerdanske.com> - 2024-02-16 11:00 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? "Roy J. Tellason, Sr." <roy@rtellason.com> - 2024-02-16 20:00 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? David Christensen <dpchrist@holgerdanske.com> - 2024-02-16 21:00 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Gremlin <scott-andrews@columbus.rr.com> - 2024-02-16 22:50 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? "Roy J. Tellason, Sr." <roy@rtellason.com> - 2024-02-17 19:50 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? gene heskett <gheskett@shentel.net> - 2024-02-18 02:50 +0100
            Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? debian-user@howorth.org.uk - 2024-02-15 13:20 +0100
              Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-02-15 16:10 +0100
  Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? David Christensen <dpchrist@holgerdanske.com> - 2024-01-09 23:40 +0100
    Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? David Christensen <dpchrist@holgerdanske.com> - 2024-01-09 23:50 +0100
    Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-01-10 10:40 +0100
      Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? David Christensen <dpchrist@holgerdanske.com> - 2024-01-10 16:40 +0100
        Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Curt <curty@free.fr> - 2024-01-10 18:10 +0100
          Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-01-10 18:40 +0100
            Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? David Christensen <dpchrist@holgerdanske.com> - 2024-01-11 03:30 +0100
          Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Dan Ritter <dsr@randomstring.org> - 2024-01-10 18:40 +0100
          Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? David Christensen <dpchrist@holgerdanske.com> - 2024-01-11 03:20 +0100
            Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Dan Ritter <dsr@randomstring.org> - 2024-01-11 15:10 +0100
              Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? David Christensen <dpchrist@holgerdanske.com> - 2024-01-11 16:10 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Dan Ritter <dsr@randomstring.org> - 2024-01-11 19:30 +0100
              Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Stefan Monnier <monnier@iro.umontreal.ca> - 2024-01-11 21:30 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Michael Stone <mstone@debian.org> - 2024-01-12 00:00 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? Dan Ritter <dsr@randomstring.org> - 2024-01-12 12:20 +0100
                Re: SMART Uncorrectable_Error_Cnt rising - should I be worried? The Wanderer <wanderer@fastmail.fm> - 2024-02-15 16:50 +0100

csiph-web