Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1605183 > unrolled thread

Re: [PATCH 3/4] RAS: Add a Corrected Errors Collector

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 3/4] RAS: Add a Corrected Errors Collector "Luck, Tony" <tony.luck@intel.com> - 2017-03-20 23:50 +0100
    Re: [PATCH 3/4] RAS: Add a Corrected Errors Collector Borislav Petkov <bp@alien8.de> - 2017-03-22 19:10 +0100
      Re: [PATCH 3/4] RAS: Add a Corrected Errors Collector Borislav Petkov <bp@alien8.de> - 2017-03-23 16:30 +0100
        Re: [PATCH 3/4] RAS: Add a Corrected Errors Collector "Luck, Tony" <tony.luck@intel.com> - 2017-03-23 18:30 +0100
          Re: [PATCH 3/4] RAS: Add a Corrected Errors Collector Borislav Petkov <bp@alien8.de> - 2017-03-23 18:30 +0100
            Re: [PATCH 3/4] RAS: Add a Corrected Errors Collector "Luck, Tony" <tony.luck@intel.com> - 2017-03-23 19:30 +0100
              Re: [PATCH 3/4] RAS: Add a Corrected Errors Collector Borislav Petkov <bp@alien8.de> - 2017-03-24 12:10 +0100

#1605183 — Re: [PATCH 3/4] RAS: Add a Corrected Errors Collector

On Thu, Mar 09, 2017 at 11:08:17AM +0100, Borislav Petkov wrote:
> +config RAS_CEC
> +	bool "Correctable Errors Collector"
> +	depends on X86_MCE && MEMORY_FAILURE && DEBUG_FS
> +	---help---
> +	  This is a small cache which collects correctable memory errors per 4K
> +	  page PFN and counts their repeated occurrence. Once the counter for a
> +	  PFN overflows, we try to soft-offline that page as we take it to mean
> +	  that it has reached a relatively high error count and would probably
> +	  be best if we don't use it anymore.

You added "count_threshold" for me ... so the condition isn't quite "overflows"
like it was in the early versions.

We may need to give some thought on what to do if the attempt to offline
the page fails (e.g. because the page belongs to the kernel). Right now
you delete it from the list, but we will see more errors as the page is
still in use. Eventually the counter will hit count_threshold and we will
try to offline again. Rinse, repeat.

Someone also recently sent me a log from a machine with corrected errors
in over 9000 unique addresses. Need a parameter to allocate more than one
page for the collector, or a way to grow the space.

-Tony

[toc] | [next] | [standalone]

#1606868

On Mon, Mar 20, 2017 at 03:48:24PM -0700, Luck, Tony wrote:
> You added "count_threshold" for me ... so the condition isn't quite "overflows"
> like it was in the early versions.

It is a max count which, when reached, causes the soft offline attempt.
What did you mean with "overflows" exactly then?

> We may need to give some thought on what to do if the attempt to offline
> the page fails (e.g. because the page belongs to the kernel). Right now
> you delete it from the list, but we will see more errors as the page is
> still in use. Eventually the counter will hit count_threshold and we will
> try to offline again. Rinse, repeat.

Well, what *is* there we can do? If the offlining code can't offline
it, there's not a whole lot we *can* do. The error would keep repeating
as a corrected error, rinse, repeat and we will keep trying to offline
containing page.

That is, until it degrades to an uncorrectable error and then we're
dead.

Either way, the collector can't really do anything about it. This would
be beyond its functionality anyway.

IMO.

> Someone also recently sent me a log from a machine with corrected errors
> in over 9000 unique addresses. Need a parameter to allocate more than one
> page for the collector, or a way to grow the space.

Well, so even with the amount of unique addresses higher than the CEC
slots, we should be able to deal with them ok: the moment we enter more
than CLEAN_ELEMS pfns, we will trigger a spring cleaning which will
degrade the already logged errors. Once the array is filled up, we will
replace the LRU pfn with the new one.

And so on.

And this way it would fulfill its purpose of *not* generating error
records into the decoding chain after it. If one of those 9000 errors
overflows, we will try to offline the page.

Either way we work as advertized.

Lemme try to write a small script exercising exactly that scenario to
see whether I'm actually not talking crap here :-)

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

[toc] | [prev] | [next] | [standalone]

#1607620

On Wed, Mar 22, 2017 at 07:03:39PM +0100, Borislav Petkov wrote:
> Lemme try to write a small script exercising exactly that scenario to
> see whether I'm actually not talking crap here :-)

Ok, here's a snapshot from the CEC after letting it run for a couple of
hours in a guest with a script running twice in parallel and injecting
random PFNs. We have 0 offlined pages because a PFN number doesn't
repeat frequently enough to cause an overflow.

When I force the occurrence of a single PFN for 1023 and more times and
do that more than once, this happens:

[ 6629.091239] RAS: Soft-offlining pfn: 0x7fff
[ 6629.093036] __get_any_page: 0x7fff free buddy page
[ 6653.259476] RAS: Soft-offlining pfn: 0x7fff
[ 6653.260100] soft offline: 0x7fff page already poisoned

...

Stats:
CEs: 32614
offlined pages: 2
^^^^^^^^^^^^^^^^^

Flags: 0x0
Timer interval: 86400 seconds
Decays: 254
Action threshold: 1023

The "already poisoned" thing shouldn't happen in real life because once
the page frame is poisoned, it shouldn't generate MCEs.




Every 2.0s: head -n 40 array; tail -n 40 array                                                   Thu Mar 23 17:15:15 2017

{ n: 512
 000: [0000000000000056|c01]
 001: [000000000000011f|801]
 002: [0000000000000171|401]
 003: [00000000000001ce|401]
 004: [000000000000024a|401]
 005: [000000000000026e|401]
 006: [000000000000034d|c01]
 007: [0000000000000395|c01]
 008: [00000000000003b9|801]
 009: [0000000000000458|003]
 010: [000000000000045c|401]
 011: [00000000000004f9|401]
 012: [00000000000005d1|c01]
 013: [0000000000000677|801]
 014: [000000000000069d|401]
 015: [00000000000006b3|401]
 016: [00000000000006f5|c01]
 017: [00000000000006fc|401]
 018: [000000000000074d|401]
 019: [0000000000000764|c01]
 020: [00000000000008a8|801]
 021: [0000000000000951|401]
 022: [0000000000000994|401]
 023: [0000000000000aa8|401]
 024: [0000000000000ac7|801]
 025: [0000000000000af2|801]
 026: [0000000000000bb5|801]
 027: [0000000000000bd5|401]
 028: [0000000000000be0|c01]
 029: [0000000000000c30|c01]
 030: [0000000000000c61|801]
 031: [0000000000000c8a|401]
 032: [0000000000000d0d|801]
 033: [0000000000000d2a|003]
 034: [0000000000000d4d|401]
 035: [0000000000000d87|c01]
 036: [0000000000000da4|c01]
 037: [0000000000000e06|401]
 038: [0000000000000e23|c01]

 ...

 480: [0000000000007d22|005]
 481: [0000000000007d5f|002]
 482: [0000000000007d9f|004]
 483: [0000000000007db1|c01]
 484: [0000000000007dbf|002]
 485: [0000000000007dcf|002]
 486: [0000000000007dd8|401]
 487: [0000000000007df0|001]
 488: [0000000000007df4|002]
 489: [0000000000007e1f|003]
 490: [0000000000007e35|801]
 491: [0000000000007e73|003]
 492: [0000000000007e77|401]
 493: [0000000000007e80|002]
 494: [0000000000007e9c|002]
 495: [0000000000007eac|002]
 496: [0000000000007ecb|002]
 497: [0000000000007ed8|801]
 498: [0000000000007edc|003]
 499: [0000000000007ee3|801]
 500: [0000000000007f05|004]
 501: [0000000000007f15|002]
 502: [0000000000007f51|004]
 503: [0000000000007f5e|003]
 504: [0000000000007f80|801]
 505: [0000000000007f92|003]
 506: [0000000000007fb2|002]
 507: [0000000000007fd9|002]
 508: [0000000000007fdf|002]
 509: [0000000000007fe5|004]
 510: [0000000000007ff4|801]
 511: [0000000000007ffa|001]
}
Stats:
CEs: 30074
offlined pages: 0
Flags: 0x0
Timer interval: 86400 seconds
Decays: 234
Action threshold: 1023

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

[toc] | [prev] | [next] | [standalone]

#1607732

On Thu, Mar 23, 2017 at 04:22:28PM +0100, Borislav Petkov wrote:
> On Wed, Mar 22, 2017 at 07:03:39PM +0100, Borislav Petkov wrote:
> > Lemme try to write a small script exercising exactly that scenario to
> > see whether I'm actually not talking crap here :-)
> 
> Ok, here's a snapshot from the CEC after letting it run for a couple of
> hours in a guest with a script running twice in parallel and injecting
> random PFNs. We have 0 offlined pages because a PFN number doesn't
> repeat frequently enough to cause an overflow.
> 
> When I force the occurrence of a single PFN for 1023 and more times and
> do that more than once, this happens:
> 
> [ 6629.091239] RAS: Soft-offlining pfn: 0x7fff
> [ 6629.093036] __get_any_page: 0x7fff free buddy page
> [ 6653.259476] RAS: Soft-offlining pfn: 0x7fff
> [ 6653.260100] soft offline: 0x7fff page already poisoned
> 
> ...
> 
> Stats:
> CEs: 32614
> offlined pages: 2
> ^^^^^^^^^^^^^^^^^
> 
> Flags: 0x0
> Timer interval: 86400 seconds
> Decays: 254
> Action threshold: 1023
> 
> The "already poisoned" thing shouldn't happen in real life because once
> the page frame is poisoned, it shouldn't generate MCEs.

It can happen if Linux didn't actually take the page offline
(because it was a kernel page). The CEC code only knows that
it queued this page to be taken offline ... and has no way
to know if that succeeded or not.

Some people have grumbled about mcelog(8) doing the same thing.

So is it worth keeping track of the page numbers that we
tried to offline?  If they show up again we shouldn't add
them back into the array.

-Tony

[toc] | [prev] | [next] | [standalone]

#1607737

On Thu, Mar 23, 2017 at 10:20:31AM -0700, Luck, Tony wrote:
> It can happen if Linux didn't actually take the page offline
> (because it was a kernel page). The CEC code only knows that
> it queued this page to be taken offline ... and has no way
> to know if that succeeded or not.

Right, that's the case when the offlining fails for whatever reason.

> So is it worth keeping track of the page numbers that we
> tried to offline?  If they show up again we shouldn't add
> them back into the array.

Meh, I don't like the idea of keeping an evergrowing list of PFNs we
can't do anything about anyway.

And actually, you want the kernel to keep complaining about not being
able to offline those because then admins should consider speeding up
the arrival of the maintenance window - the kernel memory itself is
going sick so that not even RAS actions help here.

I'm wondering if we should make the offlining code dump a more
comprehensible message with hints what to do...

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

[toc] | [prev] | [next] | [standalone]

#1607773

On Thu, Mar 23, 2017 at 06:28:39PM +0100, Borislav Petkov wrote:
> Meh, I don't like the idea of keeping an evergrowing list of PFNs we
> can't do anything about anyway.

Keeping every PFN would be overkill (most of them should be taken
offline with no issues).  A fixed array of a few of them with timestamps
to drop the oldest would likely be a good enough(TM) solution.

> And actually, you want the kernel to keep complaining about not being
> able to offline those because then admins should consider speeding up
> the arrival of the maintenance window - the kernel memory itself is
> going sick so that not even RAS actions help here.

Worst case is pretty ugly. A frequently used kernel page with a stuck
bit could be added to the CEC array, overflow, and generate a message
at a pretty high rate.

> I'm wondering if we should make the offlining code dump a more
> comprehensible message with hints what to do...

Maybe ... but it gets into opinion rather than science. Some folks
think that very low numbers of corrected errors warrant DIMM replacement.
Others think that you can keep running almost forever with a several
stuck bits per DIMM.

Some of the best decisions would be made by correlating error logs
from multiple reboots ... which the kernel can't do.

-Tony

[toc] | [prev] | [next] | [standalone]

#1608324

On Thu, Mar 23, 2017 at 11:20:44AM -0700, Luck, Tony wrote:
> Keeping every PFN would be overkill (most of them should be taken
> offline with no issues).  A fixed array of a few of them with timestamps
> to drop the oldest would likely be a good enough(TM) solution.

The reason being? Prevent the CEC from adding it and trying to
unsuccessfully offline it again?

If so, that means, we will query that list on every element insertion so
it needs to be something we can search pretty quickly.

> Worst case is pretty ugly. A frequently used kernel page with a stuck
> bit could be added to the CEC array, overflow, and generate a message
> at a pretty high rate.

Oh sure, but it would still be lower rate than generating a message for
*each* correctable error. And I really think that these messages should
*not* be supressed as they're important. The CEC kinda ratelimits them a
bit though...

> Maybe ... but it gets into opinion rather than science. Some folks
> think that very low numbers of corrected errors warrant DIMM replacement.
> Others think that you can keep running almost forever with a several
> stuck bits per DIMM.
> 
> Some of the best decisions would be made by correlating error logs
> from multiple reboots ... which the kernel can't do.

... and maybe even then it doesn't fit everybody's use case.

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web