Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1690604 > unrolled thread

Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac "Kani, Toshimitsu" <toshi.kani@hpe.com> - 2017-07-18 22:00 +0200
    Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-07-18 23:20 +0200
      Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Borislav Petkov <bp@alien8.de> - 2017-07-19 08:00 +0200
        RE: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac "Luck, Tony" <tony.luck@intel.com> - 2017-07-19 17:20 +0200
          Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Borislav Petkov <bp@alien8.de> - 2017-07-19 18:00 +0200
            RE: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac "Luck, Tony" <tony.luck@intel.com> - 2017-07-19 20:10 +0200
      Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac "Kani, Toshimitsu" <toshi.kani@hpe.com> - 2017-07-19 18:50 +0200
        Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Borislav Petkov <bp@alien8.de> - 2017-07-20 06:40 +0200
          Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac "Kani, Toshimitsu" <toshi.kani@hpe.com> - 2017-07-20 22:00 +0200
            Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-07-20 22:20 +0200
              Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac "Kani, Toshimitsu" <toshi.kani@hpe.com> - 2017-07-20 23:10 +0200
            Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Borislav Petkov <bp@alien8.de> - 2017-07-21 15:40 +0200
              Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Borislav Petkov <bp@alien8.de> - 2017-07-21 15:50 +0200
                Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Borislav Petkov <bp@alien8.de> - 2017-07-21 17:20 +0200
                  Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac "Kani, Toshimitsu" <toshi.kani@hpe.com> - 2017-07-21 17:40 +0200
                    Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-07-21 17:50 +0200
                      Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac "Kani, Toshimitsu" <toshi.kani@hpe.com> - 2017-07-21 18:50 +0200
                        Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-07-21 19:10 +0200
                          Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Borislav Petkov <bp@alien8.de> - 2017-07-21 19:30 +0200
                            Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac "Kani, Toshimitsu" <toshi.kani@hpe.com> - 2017-07-21 20:50 +0200
                              Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Borislav Petkov <bp@alien8.de> - 2017-07-22 08:30 +0200
                                Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac "Kani, Toshimitsu" <toshi.kani@hpe.com> - 2017-07-24 17:00 +0200
                                  Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Borislav Petkov <bp@alien8.de> - 2017-07-24 17:10 +0200
                                    Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac "Kani, Toshimitsu" <toshi.kani@hpe.com> - 2017-07-24 17:30 +0200
                                      Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Borislav Petkov <bp@alien8.de> - 2017-07-24 17:40 +0200
                                        Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac "Kani, Toshimitsu" <toshi.kani@hpe.com> - 2017-07-24 18:00 +0200
                                          Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Borislav Petkov <bp@alien8.de> - 2017-07-24 18:40 +0200
                                            Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac "Kani, Toshimitsu" <toshi.kani@hpe.com> - 2017-07-24 19:50 +0200
                                              Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Boris Petkov <bp@alien8.de> - 2017-07-24 20:00 +0200
                                                Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac "Kani, Toshimitsu" <toshi.kani@hpe.com> - 2017-07-24 20:00 +0200
                                                  Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Borislav Petkov <bp@alien8.de> - 2017-07-24 20:20 +0200
                                          Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-07-24 20:00 +0200
                                            Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac "Kani, Toshimitsu" <toshi.kani@hpe.com> - 2017-07-24 20:20 +0200
                                        Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-07-24 18:10 +0200
                                          Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Borislav Petkov <bp@alien8.de> - 2017-07-24 18:50 +0200
                                            Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-07-24 20:20 +0200
                                              Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Borislav Petkov <bp@alien8.de> - 2017-07-24 20:40 +0200
                                                Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac "Kani, Toshimitsu" <toshi.kani@hpe.com> - 2017-07-26 01:10 +0200
                    Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Borislav Petkov <bp@alien8.de> - 2017-07-21 18:00 +0200
                      Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac "Kani, Toshimitsu" <toshi.kani@hpe.com> - 2017-07-21 18:40 +0200
                Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac "Kani, Toshimitsu" <toshi.kani@hpe.com> - 2017-07-21 17:20 +0200
              Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-07-21 15:50 +0200
    Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac Borislav Petkov <bp@alien8.de> - 2017-07-19 08:00 +0200

Page 1 of 3  [1] 2 3  Next page →

#1690604 — Re: [PATCH 3/3] ghes_edac: add platform check to enable ghes_edac

On Tue, 2017-07-18 at 08:00 +0200, Borislav Petkov wrote:
> On Mon, Jul 17, 2017 at 03:59:12PM -0600, Toshi Kani wrote:
> > The ghes_edac driver was introduced in 2013 [1], but it has not
> > been enabled by any distro yet.  This driver obtains error info
> > from firmware interfaces, which are not properly implemented on
> > many platforms, as the driver always emits the messages below:
> > 
> >  This EDAC driver relies on BIOS to enumerate memory and get error
> > reports.  Unfortunately, not all BIOSes reflect the memory layout
> > correctly  So, the end result of using this driver varies from
> > vendor to vendor  If you find incorrect reports, please contact
> > your hardware vendor  to correct its BIOS.
> > 
> > To get out from this situation, add a platform type check to
> > selectively enable the driver on the platforms that are known to
> > have proper firmware implementation.  Platform vendors can add
> > their platforms to the list when they support ghes_edac.
> 
> So maintaining whitelists for things has always been a PITA and we
> should try to avoid it, if possible. (We can always do it if nothing
> saner comes along.)

Agreed.

> Now, below is a dirty patch converting ghes_edac to a normal module.
> On systems where we have GHES, the firmware generally disables the
> detection of the presence of ECC hardware, thus preventing the
> platform EDAC driver from loading.

I have HPE Haswell and Skylake test systems with GHES, but they do not
hide IMCs from the OS.  So, the sb_edac and skx_edac drivers get
attached on these systems when ghes_edac is disabled.

> Let me clarify: I have an AMD HP box which, when GHES is enabled in
> the BIOS, says that ECC is disabled in the memory controller and the
> amd64_edac driver doesn't load for that memory controller.

Hmm... what's the platform name of this box?  I can look into this case
if you need.

> And I think we should try this first: have the firmware disable
> detection methods so that the platform drivers don't load.

I do not think we can rely on this method.

> Then, ghes_edac can be a simple module and no other driver would
> attempt loading.

I like the use of notifier chain, which is much cleaner.

> The question is: does the platform do this disabling now?

Unfortunately, that is not the case today.  The IMCs cannot be hidden
with the Device Hide registers for Skylake at least.

Thanks,
-Toshi

[toc] | [next] | [standalone]

#1690662

Em Tue, 18 Jul 2017 19:58:54 +0000
"Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu:

> On Tue, 2017-07-18 at 08:00 +0200, Borislav Petkov wrote:
> > On Mon, Jul 17, 2017 at 03:59:12PM -0600, Toshi Kani wrote:  
> > > The ghes_edac driver was introduced in 2013 [1], but it has not
> > > been enabled by any distro yet.  This driver obtains error info
> > > from firmware interfaces, which are not properly implemented on
> > > many platforms, as the driver always emits the messages below:
> > > 
> > >  This EDAC driver relies on BIOS to enumerate memory and get error
> > > reports.  Unfortunately, not all BIOSes reflect the memory layout
> > > correctly  So, the end result of using this driver varies from
> > > vendor to vendor  If you find incorrect reports, please contact
> > > your hardware vendor  to correct its BIOS.
> > > 
> > > To get out from this situation, add a platform type check to
> > > selectively enable the driver on the platforms that are known to
> > > have proper firmware implementation.  Platform vendors can add
> > > their platforms to the list when they support ghes_edac.  
> > 
> > So maintaining whitelists for things has always been a PITA and we
> > should try to avoid it, if possible. (We can always do it if nothing
> > saner comes along.)  
> 
> Agreed.
> 
> > Now, below is a dirty patch converting ghes_edac to a normal module.
> > On systems where we have GHES, the firmware generally disables the
> > detection of the presence of ECC hardware, thus preventing the
> > platform EDAC driver from loading.  
> 
> I have HPE Haswell and Skylake test systems with GHES, but they do not
> hide IMCs from the OS.  So, the sb_edac and skx_edac drivers get
> attached on these systems when ghes_edac is disabled.
> 
> > Let me clarify: I have an AMD HP box which, when GHES is enabled in
> > the BIOS, says that ECC is disabled in the memory controller and the
> > amd64_edac driver doesn't load for that memory controller.  
> 
> Hmm... what's the platform name of this box?  I can look into this case
> if you need.
> 
> > And I think we should try this first: have the firmware disable
> > detection methods so that the platform drivers don't load.  
> 
> I do not think we can rely on this method.
> 
> > Then, ghes_edac can be a simple module and no other driver would
> > attempt loading.  
> 
> I like the use of notifier chain, which is much cleaner.
> 
> > The question is: does the platform do this disabling now?  
> 
> Unfortunately, that is not the case today.  The IMCs cannot be hidden
> with the Device Hide registers for Skylake at least.

We had a similar discussion several years ago when I wrote this driver.
On that time, I talked with Red Hat, HP, Dell, Intel people and with
some customers with large clusters.

The way it is, ghes_edac is a poor man's driver. What it hopefully
provide is a detection that an error happened, without really telling
the user what component should be replaced.

Ok, on machines with their own error reporting mechanism (like
HP servers), a sys admin can look on some proprietary software
(or bios), in order to identify what happened.

Yet, BIOS doesn't provide any glue about what's the memory architecture,
as it maps memory as if it was a single DIMM memory:

(from ghes_edac_register)

	layers[0].type = EDAC_MC_LAYER_ALL_MEM;
	layers[0].size = num_dimm;
	layers[0].is_virt_csrow = true;

So, even on systems where the BIOS actually knows how the memory
cards are wired, it will mask the memory controller data.

Now, the EDAC driver can also be used to identify what
channels are used. That helps the sys admin to know if the
memories are connected in a way that it will be using multiple
channels, or not, helping to setup the machine to obtain
the maximum possible performance.

So, for example, on my Intel-based HP server, I can check
such info with:

$ ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: HP model ProLiant ML350 Gen9
$ ras-mc-ctl --layout
       +-----------------------------------------------------------------------+
       |                mc0                |                mc1                |
       | channel0  | channel1  | channel2  | channel0  | channel1  | channel2  |
-------+-----------------------------------------------------------------------+
slot2: |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |
slot1: |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |
slot0: |  16384 MB  |     0 MB  |  16384 MB  |  16384 MB  |     0 MB  |  16384 MB  |
-------+---------------------------------------------------------------------------+

So, I know that both CPUs will be connected to my memories, and,
on both, it is using 2 channels.

If I was using the ghes driver, that information would be hidden.

So, due to all problems with ghes, it is enabled only if there are no
better solution, e. g. on systems where there's no way to talk directly
to the hardware (like on E7 Xeon machines, where the memory controller
is actually on a separate chip that are controlled only by the BIOS).

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1690990

On Tue, Jul 18, 2017 at 06:15:45PM -0300, Mauro Carvalho Chehab wrote:
> The way it is, ghes_edac is a poor man's driver. What it hopefully
> provide is a detection that an error happened, without really telling
> the user what component should be replaced.

I beg to differ. From the UEFI spec:

"The module number of the memory error location. (NODE, CARD, and MODULE
should provide the information necessary to identify the failing FRU)."

So this tuple is sufficient to pinpoint the DIMM, IIUC.

Which means, ghes_edac can have a single layer of DIMMs without channels.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

[toc] | [prev] | [next] | [standalone]

#1691835

> "The module number of the memory error location. (NODE, CARD, and MODULE
> should provide the information necessary to identify the failing FRU)."
>
> So this tuple is sufficient to pinpoint the DIMM, IIUC.
>
> Which means, ghes_edac can have a single layer of DIMMs without channels.

The tricky part is that you have to rely on SMBIOS/DMI to know what DIMMs are
on the system when the driver initializes so you can populate /sys/.*/edac

Later when GHES gives you a NODE/CARD/MODULE) in an error record.  You need
to match these up. But SMBIOS only gave you two strings "Locator" and "Bank
Locator" which have no defined syntax. You are at the mercy of the BIOS writer
to put in something parseable. Some writers used zero based counts, others are
Fortran fans and use one-based. Still other use letters.  About the one guarantee
is that they will make almost no effort to match the silkscreen labels on the motherboard
itself.

E.g. my Broadwell-EX has things like:

        Locator: CHANNEL D DIMM 1
        Bank Locator: Memriser8

Channel is A,B,C,D. DIMM is 0, 1, 2. Memriser is {1..8} so this manages to use all
three counting options!

-Tony

[toc] | [prev] | [next] | [standalone]

#1691959

On Wed, Jul 19, 2017 at 03:14:32PM +0000, Luck, Tony wrote:
> Later when GHES gives you a NODE/CARD/MODULE) in an error record.  You need
> to match these up. But SMBIOS only gave you two strings "Locator" and "Bank
> Locator" which have no defined syntax. You are at the mercy of the BIOS writer
> to put in something parseable.

Well, at some point it is only so much we can do, right?

I mean, if FW says it wants to do firmware-first and we go and adhere
to that, it should be expected that said FW vendor marks the silkscreen
labels and DMI data accordingly.

I mean, it is time for FW to put its money where its mouth is, no?

How else would you do this?

Firmware First but the kernel does the figuring out which DIMMs are
where. So FW can't have the cake and eat it too.

:-)

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

[toc] | [prev] | [next] | [standalone]

#1692078

>> Later when GHES gives you a NODE/CARD/MODULE) in an error record.  You need
>> to match these up. But SMBIOS only gave you two strings "Locator" and "Bank
>> Locator" which have no defined syntax. You are at the mercy of the BIOS writer
>> to put in something parseable.
>
> Well, at some point it is only so much we can do, right?
>
> I mean, if FW says it wants to do firmware-first and we go and adhere
> to that, it should be expected that said FW vendor marks the silkscreen
> labels and DMI data accordingly.
>
> I mean, it is time for FW to put its money where its mouth is, no?
>
> How else would you do this?

By thinking a bit more and realizing that what I wrote up above misses that
at byte offset 78 in the UEFI memory error section there is "Module Handle"
which tells you which SMBIOS entry to use.

So this should work just fine (as long as BIOS fills out all these fields ...
there's a "Validation Bits" mask at the start of the error structure that says
which fields have been populated).

-Tony

[toc] | [prev] | [next] | [standalone]

#1692022

On Tue, 2017-07-18 at 18:15 -0300, Mauro Carvalho Chehab wrote:
> Em Tue, 18 Jul 2017 19:58:54 +0000
 :
> We had a similar discussion several years ago when I wrote this
> driver. On that time, I talked with Red Hat, HP, Dell, Intel people
> and with some customers with large clusters.
> 
> The way it is, ghes_edac is a poor man's driver. What it hopefully
> provide is a detection that an error happened, without really telling
> the user what component should be replaced.

"poor man's driver" is a bit misleading, but yes, firmware-first
platforms have RAS features built-into the platforms, and they do not
need intelligence in EDAC drivers, which may conflict with the
platform's RAS features.  I cannot speak for other vendors, but HPE
platforms log errors and provide FRU info.  ghes_edac allows to report
errors to OS management tools like rasdaemon in addition to platform-
specific managements.

> Ok, on machines with their own error reporting mechanism (like
> HP servers), a sys admin can look on some proprietary software
> (or bios), in order to identify what happened.
> 
> Yet, BIOS doesn't provide any glue about what's the memory
> architecture, as it maps memory as if it was a single DIMM memory:
> 
> (from ghes_edac_register)
> 
> 	layers[0].type = EDAC_MC_LAYER_ALL_MEM;
> 	layers[0].size = num_dimm;
> 	layers[0].is_virt_csrow = true;
> 
> So, even on systems where the BIOS actually knows how the memory
> cards are wired, it will mask the memory controller data.
> 
> Now, the EDAC driver can also be used to identify what
> channels are used. That helps the sys admin to know if the
> memories are connected in a way that it will be using multiple
> channels, or not, helping to setup the machine to obtain
> the maximum possible performance.
> 
> So, for example, on my Intel-based HP server, I can check
> such info with:
> 
> $ ras-mc-ctl --mainboard
> ras-mc-ctl: mainboard: HP model ProLiant ML350 Gen9
> $ ras-mc-ctl --layout
>        +-------------------------------------------------------------
> ----------+
>        |                mc0                |                mc1      
>           |
>        | channel0  | channel1  | channel2  | channel0  | channel1  |
> channel2  |
> -------+-------------------------------------------------------------
> ----------+
> slot2: |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0
> MB  |     0 MB  |
> slot1: |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0
> MB  |     0 MB  |
> slot0: |  16384 MB  |     0 MB  |  16384 MB  |  16384 MB  |     0
> MB  |  16384 MB  |
> -------+-------------------------------------------------------------
> --------------+
> 
> So, I know that both CPUs will be connected to my memories, and,
> on both, it is using 2 channels.
> 
> If I was using the ghes driver, that information would be hidden.
> 
> So, due to all problems with ghes, it is enabled only if there are no
> better solution, e. g. on systems where there's no way to talk
> directly to the hardware (like on E7 Xeon machines, where the memory
> controller is actually on a separate chip that are controlled only by
> the BIOS).

Thanks for the info!  That's very helpful.  I will check to see if
ghes_edac provides enough info that we need.
-Toshi

[toc] | [prev] | [next] | [standalone]

#1692405

On Wed, Jul 19, 2017 at 04:40:25PM +0000, Kani, Toshimitsu wrote:
>  ghes_edac allows to report errors to OS management tools like
> rasdaemon in addition to platform- specific managements.

So ghes_edac *is* a poor man's driver in the sense that it doesn't do
anything fancy but repeat like a parrot data it has gotten from the
firmware and shoving it into the EDAC counters. At least that's the
intention. Nothing more. All the action stuff like error detection and
recovery should be done by the firmware.

But considering how SNAFU'd firmware is, I wouldn't expect any great RAS
functionality there. Of course, I'd be delighted to be proven wrong.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

[toc] | [prev] | [next] | [standalone]

#1693197

On Thu, 2017-07-20 at 06:33 +0200, Borislav Petkov wrote:
> On Wed, Jul 19, 2017 at 04:40:25PM +0000, Kani, Toshimitsu wrote:
> >  ghes_edac allows to report errors to OS management tools like
> > rasdaemon in addition to platform- specific managements.
> 
> So ghes_edac *is* a poor man's driver in the sense that it doesn't do
> anything fancy but repeat like a parrot data it has gotten from the
> firmware and shoving it into the EDAC counters. At least that's the
> intention. Nothing more. 

Right for ghes_edac.

> All the action stuff like error detection and recovery should be done
> by the firmware.

GHES / firmware-first still requires OS recovery actions when an error
cannot be corrected by the platform.  They are handled by ghes_proc(),
and ghes_edac remains its error-reporting wrapper.

> But considering how SNAFU'd firmware is, I wouldn't expect any great
> RAS functionality there. Of course, I'd be delighted to be proven
> wrong.

Firmware has better knowledge about the platform and can provide better
RAS when implemented properly.  I agree that user experiences may vary
on platforms.

Thanks,
-Toshi

[toc] | [prev] | [next] | [standalone]

#1693207

Em Thu, 20 Jul 2017 19:50:03 +0000
"Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu:

> On Thu, 2017-07-20 at 06:33 +0200, Borislav Petkov wrote:
> > On Wed, Jul 19, 2017 at 04:40:25PM +0000, Kani, Toshimitsu wrote:  
> > >  ghes_edac allows to report errors to OS management tools like
> > > rasdaemon in addition to platform- specific managements.  
> > 
> > So ghes_edac *is* a poor man's driver in the sense that it doesn't do
> > anything fancy but repeat like a parrot data it has gotten from the
> > firmware and shoving it into the EDAC counters. At least that's the
> > intention. Nothing more.   
> 
> Right for ghes_edac.
> 
> > All the action stuff like error detection and recovery should be done
> > by the firmware.  
> 
> GHES / firmware-first still requires OS recovery actions when an error
> cannot be corrected by the platform.  They are handled by ghes_proc(),
> and ghes_edac remains its error-reporting wrapper.
> 
> > But considering how SNAFU'd firmware is, I wouldn't expect any great
> > RAS functionality there. Of course, I'd be delighted to be proven
> > wrong.  
> 
> Firmware has better knowledge about the platform and can provide better
> RAS when implemented properly.  I agree that user experiences may vary
> on platforms.

It may have a better knowledge, when the vendor ships different BIOS
for platforms with different motherboard silkscreens, but a lot of
vendors just use the same BIOS on different models, with the same
information at "Locator" and "Bank Locator" data at DMI tables,
that don't match what's printed at the board's silkscreen.

So, GHES ends by exposing wrong data. Also, such BIOS fail
to properly expose such knowledge to drivers/userspace.

On the discussions I had with HP, back in 2012, the idea was to try 
to have some sort of way for the GHES driver to query the BIOS
on a reliable way, in order to get its layout, in a way
that tools like ras-mc-ctl would properly report the memory
configuration (with --layout) and the motherboard silkscreen
labels (with --print-labels). Unfortunately, at least on that
time, the discussions with HP didn't proceed.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1693232

On Thu, 2017-07-20 at 17:15 -0300, Mauro Carvalho Chehab wrote:
> Em Thu, 20 Jul 2017 19:50:03 +0000
> "Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu:
 :
> > Firmware has better knowledge about the platform and can provide
> > better RAS when implemented properly.  I agree that user
> > experiences may vary on platforms.
> 
> It may have a better knowledge, when the vendor ships different BIOS
> for platforms with different motherboard silkscreens, but a lot of
> vendors just use the same BIOS on different models, with the same
> information at "Locator" and "Bank Locator" data at DMI tables,
> that don't match what's printed at the board's silkscreen.
> 
> So, GHES ends by exposing wrong data. Also, such BIOS fail
> to properly expose such knowledge to drivers/userspace.

I see.  Yeah, I can see such problems could be overlooked since normal
tests run just fine even if there is a mismatch in such info...

> On the discussions I had with HP, back in 2012, the idea was to try 
> to have some sort of way for the GHES driver to query the BIOS
> on a reliable way, in order to get its layout, in a way
> that tools like ras-mc-ctl would properly report the memory
> configuration (with --layout) and the motherboard silkscreen
> labels (with --print-labels). Unfortunately, at least on that
> time, the discussions with HP didn't proceed.

Thanks for the info.  I hope we can enable it this time around.
-Toshi

[toc] | [prev] | [next] | [standalone]

#1693706

On Thu, Jul 20, 2017 at 07:50:03PM +0000, Kani, Toshimitsu wrote:
> GHES / firmware-first still requires OS recovery actions when an error
> cannot be corrected by the platform.  They are handled by ghes_proc(),
> and ghes_edac remains its error-reporting wrapper.

I mean all the recovery actions the firmware does because it gets to see
the error first. Otherwise, Firmware First is the the dumbest repeater
layer in the history of layers.

> Firmware has better knowledge about the platform and can provide better
> RAS when implemented properly.

s/when/if/

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

[toc] | [prev] | [next] | [standalone]

#1693715

On Fri, Jul 21, 2017 at 10:40:01AM -0300, Mauro Carvalho Chehab wrote:
> What happens when the error can be corrected? Does it still report it to
> userspace, or just silently hide the error?
> 
> If I remember well about a past discussion with some vendor, I was told
> that the firmware can hide some errors from being reported. Is it
> still the case?

I've heard the same thing but I have no idea what they're actually
doing. But it would make sense because the intention is not to worry
users unnecessarily if it can hide the error and if there are no adverse
consequences from it.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

[toc] | [prev] | [next] | [standalone]

#1693783

On Fri, Jul 21, 2017 at 03:08:41PM +0000, Kani, Toshimitsu wrote:
> Yes, that is correct.  Corrected errors are reported to the OS when
> they exceeded the platform's threshold.

Are those thresholds user-configurable?

If not, what are you telling users who want to see *every* corrected
error for measuring DIMM wear and so on...?

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

[toc] | [prev] | [next] | [standalone]

#1693801

On Fri, 2017-07-21 at 17:13 +0200, Borislav Petkov wrote:
> On Fri, Jul 21, 2017 at 03:08:41PM +0000, Kani, Toshimitsu wrote:
> > Yes, that is correct.  Corrected errors are reported to the OS when
> > they exceeded the platform's threshold.
> 
> Are those thresholds user-configurable?

I suppose it'd depend on vendors, but I do not think users can do it
properly unless they have depth knowledge about the hardware.

> If not, what are you telling users who want to see *every* corrected
> error for measuring DIMM wear and so on...?

Corrected errors are normal and expected to occur on healthy hardware. 
They do not need user's attention until they repeatedly occurred at a
same place.

Thanks,
-Toshi

[toc] | [prev] | [next] | [standalone]

#1693809

Em Fri, 21 Jul 2017 15:34:50 +0000
"Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu:

> On Fri, 2017-07-21 at 17:13 +0200, Borislav Petkov wrote:
> > On Fri, Jul 21, 2017 at 03:08:41PM +0000, Kani, Toshimitsu wrote:  
> > > Yes, that is correct.  Corrected errors are reported to the OS when
> > > they exceeded the platform's threshold.  
> > 
> > Are those thresholds user-configurable?  
> 
> I suppose it'd depend on vendors, but I do not think users can do it
> properly unless they have depth knowledge about the hardware.
> 
> > If not, what are you telling users who want to see *every* corrected
> > error for measuring DIMM wear and so on...?  
> 
> Corrected errors are normal and expected to occur on healthy hardware. 
> They do not need user's attention until they repeatedly occurred at a
> same place.

Yes, they're expected to happen. Still, some sys admins have their own
measurements about what's "normal" for their scenario, and want
to monitor every single corrected error, running their own
algorithm to warn if the number of corrected errors is above their
"normal" rate.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1693846

On Fri, 2017-07-21 at 12:44 -0300, Mauro Carvalho Chehab wrote:
> Em Fri, 21 Jul 2017 15:34:50 +0000
> "Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu:
> 
> > On Fri, 2017-07-21 at 17:13 +0200, Borislav Petkov wrote:
> > > On Fri, Jul 21, 2017 at 03:08:41PM +0000, Kani, Toshimitsu
> > > wrote:  
> > > > Yes, that is correct.  Corrected errors are reported to the OS
> > > > when they exceeded the platform's threshold.  
> > > 
> > > Are those thresholds user-configurable?  
> > 
> > I suppose it'd depend on vendors, but I do not think users can do
> > it properly unless they have depth knowledge about the hardware.
> > 
> > > If not, what are you telling users who want to see *every*
> > > corrected error for measuring DIMM wear and so on...?  
> > 
> > Corrected errors are normal and expected to occur on healthy
> > hardware.  They do not need user's attention until they repeatedly
> > occurred at a same place.
> 
> Yes, they're expected to happen. Still, some sys admins have their
> own measurements about what's "normal" for their scenario, and want
> to monitor every single corrected error, running their own
> algorithm to warn if the number of corrected errors is above their
> "normal" rate.

I suppose these admins had to do it because their platforms reported
all corrected errors.  It addresses such administrators' burden.

Thanks,
-Toshi

[toc] | [prev] | [next] | [standalone]

#1693868

Em Fri, 21 Jul 2017 16:40:20 +0000
"Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu:

> On Fri, 2017-07-21 at 12:44 -0300, Mauro Carvalho Chehab wrote:
> > Em Fri, 21 Jul 2017 15:34:50 +0000
> > "Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu:
> >   
> > > On Fri, 2017-07-21 at 17:13 +0200, Borislav Petkov wrote:  
> > > > On Fri, Jul 21, 2017 at 03:08:41PM +0000, Kani, Toshimitsu
> > > > wrote:    
> > > > > Yes, that is correct.  Corrected errors are reported to the OS
> > > > > when they exceeded the platform's threshold.    
> > > > 
> > > > Are those thresholds user-configurable?    
> > > 
> > > I suppose it'd depend on vendors, but I do not think users can do
> > > it properly unless they have depth knowledge about the hardware.
> > >   
> > > > If not, what are you telling users who want to see *every*
> > > > corrected error for measuring DIMM wear and so on...?    
> > > 
> > > Corrected errors are normal and expected to occur on healthy
> > > hardware.  They do not need user's attention until they repeatedly
> > > occurred at a same place.  
> > 
> > Yes, they're expected to happen. Still, some sys admins have their
> > own measurements about what's "normal" for their scenario, and want
> > to monitor every single corrected error, running their own
> > algorithm to warn if the number of corrected errors is above their
> > "normal" rate.  
> 
> I suppose these admins had to do it because their platforms reported
> all corrected errors.  It addresses such administrators' burden.

I see the value of having a threshold in BIOS, provided that it is
well documented, and whose value can be adjusted, if needed.

One of the things I wanted to implement in ras-daemon were an
algorithm that would be doing such threshold in software.
The problem is that it would require field experience. So,
I talked with a few vendors, to see if they could help doing
it, but, on that time, none rised their hands :-)

The thing with a BIOS threshold is that the user has no way to
audit the algorithm. So, when BIOS start reporting such errors,
it may be already too late: the systems may be in the verge of 
losing data (or some data was already lost).

That's critical on cluster systems with thousands of machines:
while the impact of disabling a cluster node to do some maintainance
is marginal, the impact of an uncorrected error on a single
machine may compromise weeks of expensive processing.

That's why some users prefer to monitor every single corrected
error, and compare with the probability distribution they
know that the risk of uncorrected errors is acceptable.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1693899

On Fri, Jul 21, 2017 at 02:01:31PM -0300, Mauro Carvalho Chehab wrote:
> I see the value of having a threshold in BIOS, provided that it is
> well documented, and whose value can be adjusted, if needed.
> 
> One of the things I wanted to implement in ras-daemon were an
> algorithm that would be doing such threshold in software.

We have that now in the kernel: drivers/ras/cec.c

We did it exactly for that purpose - not upsetting users unnecessarily.

> The thing with a BIOS threshold is that the user has no way to
> audit the algorithm. So, when BIOS start reporting such errors,
> it may be already too late: the systems may be in the verge of 
> losing data (or some data was already lost).

Not only that: thresholds depend on the DIMM types which means, BIOS
must know what DIMM types are in there which I doubt. So exposing that
to configuration instead of "deciding" for people would be better.

> That's critical on cluster systems with thousands of machines:
> while the impact of disabling a cluster node to do some maintainance
> is marginal, the impact of an uncorrected error on a single
> machine may compromise weeks of expensive processing.
> 
> That's why some users prefer to monitor every single corrected
> error, and compare with the probability distribution they
> know that the risk of uncorrected errors is acceptable.

Yap, you need to have stuff like that configurable - BIOS can't predict
all possible use cases.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

[toc] | [prev] | [next] | [standalone]

#1693923

On Fri, 2017-07-21 at 19:23 +0200, Borislav Petkov wrote:
 :
> Not only that: thresholds depend on the DIMM types which means,
BIOS
> must know what DIMM types are in there which I doubt. 

BIOS knows DIMM model from the SPD data.

> So exposing that to configuration instead of "deciding" for people
> would be better.

Enterprise platforms have very different model (I do not say it's
better for everyone from the cost perspective).  Typically, such
platform vendors work with DIMM vendors directly to come with their
supported DIMMs with own part numbers, which are certified for the
platforms with extensive validation testings.

Thanks,
-Toshi

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | linux.kernel

csiph-web