Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1320934 > unrolled thread

[PATCH 0/2] Fix BTT data corruptions after crash

Back to article view | Back to linux.kernel

Contents

  [PATCH 0/2] Fix BTT data corruptions after crash Toshi Kani <toshi.kani@hpe.com> - 2016-01-28 19:20 +0100
    Re: [PATCH 0/2] Fix BTT data corruptions after crash Dan Williams <dan.j.williams@intel.com> - 2016-01-28 21:20 +0100
      Re: [PATCH 0/2] Fix BTT data corruptions after crash Toshi Kani <toshi.kani@hpe.com> - 2016-01-28 21:50 +0100
        Re: [PATCH 0/2] Fix BTT data corruptions after crash Dan Williams <dan.j.williams@intel.com> - 2016-01-30 18:50 +0100
          Re: [PATCH 0/2] Fix BTT data corruptions after crash Toshi Kani <toshi.kani@hpe.com> - 2016-02-01 21:00 +0100

#1320934 — [PATCH 0/2] Fix BTT data corruptions after crash

Data corruption issues were observed in tests which initiated a system
crash/reset while accessing BTT devices.  This problem is reproducible.

The BTT driver calls pmem_rw_bytes() to update data in pmem devices.
This interface calls __copy_user_nocache(), which uses non-temporal
stores so that the stores to pmem are persistent.

__copy_user_nocache() uses non-temporal stores when a request size is
8 bytes or larger (and is aligned by 8 bytes).  The BTT driver updates
the BTT map table, which entry size is 4 bytes.  Therefore, updates to
the map table entries remain cached, and are not written to pmem after
a crash.  Since the BTT driver makes previous blocks free and uses them
for subsequent writes, the map table ends up pointing to blocks allocated
for other LBAs after a crash.

Patch 1 extends __copy_user_nocache() to use non-temporal store for
4 byte copy.  This patch fixes the BTT data corruption issue.

Patch 2 changes arch_memcpy_to_pmem() to flush processor caches when
a request is not naturally aligned or is less than 4 bytes.  This is
defensive change.

---
Toshi Kani (2):
 1/2 x86/lib/copy_user_64.S: Handle 4-byte uncached copy
 2/2 pmem: Flush cache on unaligned request

---
 arch/x86/include/asm/pmem.h | 11 +++++++++++
 arch/x86/lib/copy_user_64.S | 44 +++++++++++++++++++++++++++++++++-----------
 2 files changed, 44 insertions(+), 11 deletions(-)

[toc] | [next] | [standalone]

#1321025

On Thu, Jan 28, 2016 at 10:12 AM, Toshi Kani <toshi.kani@hpe.com> wrote:
> Data corruption issues were observed in tests which initiated a system
> crash/reset while accessing BTT devices.  This problem is reproducible.
>
> The BTT driver calls pmem_rw_bytes() to update data in pmem devices.
> This interface calls __copy_user_nocache(), which uses non-temporal
> stores so that the stores to pmem are persistent.
>
> __copy_user_nocache() uses non-temporal stores when a request size is
> 8 bytes or larger (and is aligned by 8 bytes).  The BTT driver updates
> the BTT map table, which entry size is 4 bytes.  Therefore, updates to
> the map table entries remain cached, and are not written to pmem after
> a crash.  Since the BTT driver makes previous blocks free and uses them
> for subsequent writes, the map table ends up pointing to blocks allocated
> for other LBAs after a crash.
>
> Patch 1 extends __copy_user_nocache() to use non-temporal store for
> 4 byte copy.  This patch fixes the BTT data corruption issue.
>

Nice find!

> Patch 2 changes arch_memcpy_to_pmem() to flush processor caches when
> a request is not naturally aligned or is less than 4 bytes.  This is
> defensive change.

I'm wondering if we should just document that this routine does not
support unaligned transfers?  Maybe backed by a debug mode that does
the alignment check.

[toc] | [prev] | [next] | [standalone]

#1321043

On Thu, 2016-01-28 at 12:12 -0800, Dan Williams wrote:
> On Thu, Jan 28, 2016 at 10:12 AM, Toshi Kani <toshi.kani@hpe.com> wrote:
> > Data corruption issues were observed in tests which initiated a system
> > crash/reset while accessing BTT devices.  This problem is reproducible.
> > 
> > The BTT driver calls pmem_rw_bytes() to update data in pmem devices.
> > This interface calls __copy_user_nocache(), which uses non-temporal
> > stores so that the stores to pmem are persistent.
> > 
> > __copy_user_nocache() uses non-temporal stores when a request size is
> > 8 bytes or larger (and is aligned by 8 bytes).  The BTT driver updates
> > the BTT map table, which entry size is 4 bytes.  Therefore, updates to
> > the map table entries remain cached, and are not written to pmem after
> > a crash.  Since the BTT driver makes previous blocks free and uses them
> > for subsequent writes, the map table ends up pointing to blocks 
> > allocated for other LBAs after a crash.
> > 
> > Patch 1 extends __copy_user_nocache() to use non-temporal store for
> > 4 byte copy.  This patch fixes the BTT data corruption issue.
> > 
> 
> Nice find!

:-)

> > Patch 2 changes arch_memcpy_to_pmem() to flush processor caches when
> > a request is not naturally aligned or is less than 4 bytes.  This is
> > defensive change.
> 
> I'm wondering if we should just document that this routine does not
> support unaligned transfers?  Maybe backed by a debug mode that does
> the alignment check.

Yes, I agree.  For this debug mode, do you have something in mind?  Or
should we add a new CONFIG option like CONFIG_PMEM_DEBUG?

Thanks,
-Toshi

[toc] | [prev] | [next] | [standalone]

#1322456

On Thu, Jan 28, 2016 at 12:43 PM, Toshi Kani <toshi.kani@hpe.com> wrote:
> On Thu, 2016-01-28 at 12:12 -0800, Dan Williams wrote:
[..]
>> I'm wondering if we should just document that this routine does not
>> support unaligned transfers?  Maybe backed by a debug mode that does
>> the alignment check.
>
> Yes, I agree.  For this debug mode, do you have something in mind?  Or
> should we add a new CONFIG option like CONFIG_PMEM_DEBUG?
>

I hesitated to say yes to this since some simple alignment checks
seems like a thin reason to add a new Kconfig symbol.  However, one
way we can test that memcpy_to_pmem() properly bypasses the cache is
to invalidate the cache contents that it touches.  This would have
caught this bug without needing to do a power cycle test.  In
otherwords in debug mode run an 'invd' loop after the copy.

[toc] | [prev] | [next] | [standalone]

#1323443

On Sat, 2016-01-30 at 09:44 -0800, Dan Williams wrote:
> On Thu, Jan 28, 2016 at 12:43 PM, Toshi Kani <toshi.kani@hpe.com> wrote:
> > On Thu, 2016-01-28 at 12:12 -0800, Dan Williams wrote:
> [..]
> > > I'm wondering if we should just document that this routine does not
> > > support unaligned transfers?  Maybe backed by a debug mode that does
> > > the alignment check.
> > 
> > Yes, I agree.  For this debug mode, do you have something in mind?  Or
> > should we add a new CONFIG option like CONFIG_PMEM_DEBUG?
> > 
> 
> I hesitated to say yes to this since some simple alignment checks
> seems like a thin reason to add a new Kconfig symbol.  However, one
> way we can test that memcpy_to_pmem() properly bypasses the cache is
> to invalidate the cache contents that it touches.  This would have
> caught this bug without needing to do a power cycle test.  In
> otherwords in debug mode run an 'invd' loop after the copy.

That sounds like a good idea to try out.  Since this approach no longer
depends on patch 1/2, I will separate this change from patch 1/2.

Thanks,
-Toshi  

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web