Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1392284 > unrolled thread

Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io Jeff Moyer <jmoyer@redhat.com> - 2016-05-02 17:20 +0200
    Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io Dan Williams <dan.j.williams@intel.com> - 2016-05-02 20:00 +0200
      Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io Dave Chinner <david@fromorbit.com> - 2016-05-03 02:50 +0200
        Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io "Rudoff, Andy" <andy.rudoff@intel.com> - 2016-05-03 03:30 +0200
          Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io Dave Chinner <david@fromorbit.com> - 2016-05-03 05:00 +0200
            Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io "Rudoff, Andy" <andy.rudoff@intel.com> - 2016-05-03 20:40 +0200
              Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io Dave Chinner <david@fromorbit.com> - 2016-05-04 03:40 +0200
    Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io Dave Chinner <david@fromorbit.com> - 2016-05-03 01:10 +0200
      Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io "Verma, Vishal L" <vishal.l.verma@intel.com> - 2016-05-03 01:20 +0200
      Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io Dan Williams <dan.j.williams@intel.com> - 2016-05-03 01:30 +0200
        Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io Dave Chinner <david@fromorbit.com> - 2016-05-03 04:00 +0200
          Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io Dan Williams <dan.j.williams@intel.com> - 2016-05-03 19:30 +0200
            Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io Dave Chinner <david@fromorbit.com> - 2016-05-04 05:20 +0200
              Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io Dan Williams <dan.j.williams@intel.com> - 2016-05-04 07:10 +0200

#1392284 — Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io

Dave Chinner <david@fromorbit.com> writes:

> On Mon, Apr 25, 2016 at 11:53:13PM +0000, Verma, Vishal L wrote:
>> On Tue, 2016-04-26 at 09:25 +1000, Dave Chinner wrote:
> You're assuming that only the DAX aware application accesses it's
> files.  users, backup programs, data replicators, fileystem
> re-organisers (e.g.  defragmenters) etc all may access the files and
> they may throw errors. What then?

I'm not sure how this is any different from regular storage.  If an
application gets EIO, it's up to the app to decide what to do with that.

>> > Where does the application find the data that was lost to be able to
>> > rewrite it?
>> 
>> The data that was lost is gone -- this assumes the application has some
>> ability to recover using a journal/log or other redundancy - yes, at the
>> application layer. If it doesn't have this sort of capability, the only
>> option is to restore files from a backup/mirror.
>
> So the architecture has a built in assumption that only userspace
> can handle data loss?

Remember that the proposed programming model completely bypasses the
kernel, so yes, it is expected that user-space will have to deal with
the problem.

> What about filesytsems like NOVA, that use log structured design to
> provide DAX w/ update atomicity and can potentially also provide
> redundancy/repair through the same mechanisms? Won't pmem native
> filesystems with built in data protection features like this remove
> the need for adding all this to userspace applications?

I don't think we'll /only/ support NOVA for pmem.  So we'll have to deal
with this for existing file systems, right?

> If so, shouldn't that be the focus of development rahter than
> placing the burden on userspace apps to handle storage repair
> situations?

It really depends on the programming model.  In the model Vishal is
talking about, either the applications themselves or the libraries they
link to are expected to implement the redundancies where necessary.

>> > There's an implicit assumption that applications will keep redundant
>> > copies of their data at the /application layer/ and be able to
>> > automatically repair it?

That's one way to do things.  It really depends on the application what
it will do for recovery.

>> > And then there's the implicit assumption that it will unlink and
>> > free the entire file before writing a new copy

I think Vishal was referring to restoring from backup.  cp itself will
truncate the file before overwriting, iirc.

>> To summarize, the two cases we want to handle are:
>> 1. Application has inbuilt recovery:
>>   - hits badblock
>>   - figures out it is able to recover the data
>>   - handles SIGBUS or EIO
>>   - does a (sector aligned) write() to restore the data
>
> The "figures out" step here is where >95% of the work we'd have to
> do is. And that's in filesystem and block layer code, not
> userspace, and userspace can't do that work in a signal handler.
> And it  can still fall down to the second case when the application
> doesn't have another copy of the data somewhere.

I read that "figures out" step as the application determining whether or
not it had a redundant copy.

> FWIW, we don't have a DAX enabled filesystem that can do
> reverse block mapping, so we're a year or two away from this being a
> workable production solution from the filesystem perspective. And
> AFAICT, it's not even on the roadmap for dm/md layers.

Do we even need that?  What if we added an FIEMAP flag for determining
bad blocks.  The file system could simply walk the list of extents for
the file and check the corresponding disk blocks.  No reverse mapping
required.  Also note that DM/MD don't support direct_access(), either,
so I don't think they're relevant for this discussion.

>> 2. Application doesn't have any inbuilt recovery mechanism
>>   - hits badblock
>>   - gets SIGBUS (or EIO) and crashes
>>   - Sysadmin restores file from backup
>
> Which is no different to an existing non-DAX application getting an
> EIO/sigbus from current storage technologies.
>
> Except: in the existing storage stack, redundancy and correction has
> already had to have failed for the application to see such an error.
> Hence this is normally considered a DR case as there's had to be
> cascading failures (e.g.  multiple disk failures in a RAID) to get
> to this stage, not a single error in a single sector in
> non-redundant storage.
>
> We need some form of redundancy and correction in the PMEM stack to
> prevent single sector errors from taking down services until an
> administrator can correct the problem. I'm trying to understand
> where this is supposed to fit into the picture - at this point I
> really don't think userspace applications are going to be able to do
> this reliably....

Not all storage is configured into a RAID volume, and in some instances,
the application is better positioned to recover the data (gluster/ceph,
for example).  It really comes down to whether applications or libraries
will want to implement redundancy themselves in order to get a bump in
performance by not going through the kernel.  And I think I know what
your opinion is on that front.  :-)

Speaking of which, did you see the numbers Dan shared at LSF on how much
overhead there is in calling into the kernel for syncing?  Dan, can/did
you publish that spreadsheet somewhere?

Cheers,
Jeff

[toc] | [next] | [standalone]

#1392417

On Mon, May 2, 2016 at 8:18 AM, Jeff Moyer <jmoyer@redhat.com> wrote:
> Dave Chinner <david@fromorbit.com> writes:
[..]
>> We need some form of redundancy and correction in the PMEM stack to
>> prevent single sector errors from taking down services until an
>> administrator can correct the problem. I'm trying to understand
>> where this is supposed to fit into the picture - at this point I
>> really don't think userspace applications are going to be able to do
>> this reliably....
>
> Not all storage is configured into a RAID volume, and in some instances,
> the application is better positioned to recover the data (gluster/ceph,
> for example).  It really comes down to whether applications or libraries
> will want to implement redundancy themselves in order to get a bump in
> performance by not going through the kernel.  And I think I know what
> your opinion is on that front.  :-)
>
> Speaking of which, did you see the numbers Dan shared at LSF on how much
> overhead there is in calling into the kernel for syncing?  Dan, can/did
> you publish that spreadsheet somewhere?

Here it is:

https://docs.google.com/spreadsheets/d/1pwr9psy6vtB9DOsc2bUdXevJRz5Guf6laZ4DaZlkhoo/edit?usp=sharing

On the "Filtered" tab I have some of the comparisons where:

noop => don't call msync and don't flush caches in userspace

persist => cache flushing only in userspace and only on individual cache lines

persist_4k => cache flushing only in userspace, but flushing is
performed in 4K aligned units

msync => same granularity flushing as the 'persist' case, but the
kernel internally promotes this to a 4K sized / aligned flush

msync_0 => synthetic case where msync() returns immediately and does
no other work

The takeaway is that msync() is 9-10x slower than userspace cache management.

Let me know if there are any questions and I can add an NVML developer
to this thread...

[toc] | [prev] | [next] | [standalone]

#1392694

On Mon, May 02, 2016 at 10:53:25AM -0700, Dan Williams wrote:
> On Mon, May 2, 2016 at 8:18 AM, Jeff Moyer <jmoyer@redhat.com> wrote:
> > Dave Chinner <david@fromorbit.com> writes:
> [..]
> >> We need some form of redundancy and correction in the PMEM stack to
> >> prevent single sector errors from taking down services until an
> >> administrator can correct the problem. I'm trying to understand
> >> where this is supposed to fit into the picture - at this point I
> >> really don't think userspace applications are going to be able to do
> >> this reliably....
> >
> > Not all storage is configured into a RAID volume, and in some instances,
> > the application is better positioned to recover the data (gluster/ceph,
> > for example).  It really comes down to whether applications or libraries
> > will want to implement redundancy themselves in order to get a bump in
> > performance by not going through the kernel.  And I think I know what
> > your opinion is on that front.  :-)
> >
> > Speaking of which, did you see the numbers Dan shared at LSF on how much
> > overhead there is in calling into the kernel for syncing?  Dan, can/did
> > you publish that spreadsheet somewhere?
> 
> Here it is:
> 
> https://docs.google.com/spreadsheets/d/1pwr9psy6vtB9DOsc2bUdXevJRz5Guf6laZ4DaZlkhoo/edit?usp=sharing
> 
> On the "Filtered" tab I have some of the comparisons where:

Those numbers are really wacky - the inconsistent decimal place
representation makes it really, really hard to read the differences
in orders of magnitude, too. Let's take the first numbers - noop, 64
byte ops are:

threads		ops/s
1		 90M
2		310M
4		 65M
8		175M
16		426M

Why aren't these linear? And if the test is not running in an
environment where these are controlled and linear, how valid are the
rest of the tests and hence the comparison.

> noop => don't call msync and don't flush caches in userspace
> 
> persist => cache flushing only in userspace and only on individual cache lines

So these look a lot more linear than the no-op behaviour, so I'll
just ignore the no-op results for now.

> persist_4k => cache flushing only in userspace, but flushing is
> performed in 4K aligned units

Urg, your "vs persist" percentages are all wrong. You can't have a
"-1000%" difference, you have "persist 4k" running at 10% of the
speed of "persist".

So, with that in mind, the "persist_4k" speed is:

		 ops/s		single thread
Size		vs "persist"	4k flush rate
  64		 10%		 834k
 128		 13%		 849k
 256		 15%		 410k(one off variation?)
 512		 20%		 860k
1024		 25%		 850k
2048		 50%		 840k
4096		none		 836k
8192		none		 410k

What we see here is that the CPU(s) can flush the 4k pages at a rate
of roughly 850,000 flushes/s, whilst the 64 byte flush rate is
around 8.8M flushes/s.  This is clearly demonstrated in the numbers
- as the dirty object size approaches the cache flush granularity,
the speed approaches single cacheline flush granularity speed.

Comparing 4k vs 64b flushes, we have 63 clean cache line flushes
taking roughly the same time as 9 dirty cache line flushes. Nice
numbers - that means a clean cache line flush has ~14% of the
overhead of dirty cache line flush. Seems rather high - it's tens of
CPU cycles to determine that the flush is a no-op for that
cacheline.

Fixing this seems like a hardware optimisation issue to me, but I
still have to question how many applications are going to have such
fine-grained random synchronous memory writes that this actually
matters in practice? If we are doing such small writes across
multiple different 4k pages, then TLB overhead for all the page
faults is going to be as much of an issue as 4k cache flushes...

> msync => same granularity flushing as the 'persist' case, but the
> kernel internally promotes this to a 4K sized / aligned flush

So you're calling msync for every modification that is made? What
application needs to do that? Anyway, page flush rates paint an
interesting picture:

	single thread		 versus
Size	4k flush rate		persist_4k
  64	 655k			 78%
 128	 655k			 81%
 256	 670k			163%  (* persist 4k number low) 
 512	 681k			 79%
1024	 666k			 78%
2048	 650k			 77%
4096	 652k			 78%
8192	 390k			 95%

msync adds relatively little overhead (~20% extra overhead) compared
to the performance loss from the 4k flush granularity change. And
given this appears to be a worst case test scenario (and I'm sure
msync could be improved), I don't think this demonstrates a problem
with using msync.

IMO, these numbers don't support the argument that the *msync
model* for data integrity for DAX is flawed, unworkable, or too
slow. What I see is a performance problem resulting from the
overhead of flushing clean cachelines.  i.e. there's data here that
supports the argument for reducing the overhead of flushing clean
cachelines in the hardware and/or better tracking of dirty
cachelines within the kernel, but not data that says the msync()
based data integrity model is the source of the problem.

i.e. separate the programming model from the performance issue, and
we can see that the performance problem is not caused by the
programming model - it's caused by the kernel implementation of the
model.

> The takeaway is that msync() is 9-10x slower than userspace cache management.

An alternative viewpoint: that flushing clean cachelines is
extremely expensive on Intel CPUs. ;)

i.e. Same numbers, different analysis from a different PoV, and
that gives a *completely different conclusion*.

Think about it for the moment. The hardware inefficiency being
demonstrated could be fixed/optimised in the next hardware product
cycle(s) and so will eventually go away. OTOH, we'll be stuck with
whatever programming model we come up with for the next 30-40 years,
and we'll never be able to fix flaws in it because applications will
be depending on them. Do we really want to be stuck with a pmem
model that is designed around the flaws and deficiencies of ~1st
generation hardware?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

[toc] | [prev] | [next] | [standalone]

#1392844

>> The takeaway is that msync() is 9-10x slower than userspace cache management.
>
>An alternative viewpoint: that flushing clean cachelines is
>extremely expensive on Intel CPUs. ;)
>
>i.e. Same numbers, different analysis from a different PoV, and
>that gives a *completely different conclusion*.
>
>Think about it for the moment. The hardware inefficiency being
>demonstrated could be fixed/optimised in the next hardware product
>cycle(s) and so will eventually go away. OTOH, we'll be stuck with
>whatever programming model we come up with for the next 30-40 years,
>and we'll never be able to fix flaws in it because applications will
>be depending on them. Do we really want to be stuck with a pmem
>model that is designed around the flaws and deficiencies of ~1st
>generation hardware?

Hi Dave,

Not sure I agree with your completely different conclusion.  (Not sure
I completely disagree either, but please let me raise some practical
points.)

First of all, let's say you're completely right and flushing clean
cache lines is extremely expensive.  So your solution is to wait for
the chip to be fixed?  Remember the model we're putting forward (which
we're working on documenting, because I fully agree with the lack of
documentation point you keep raising) requires the application to ASK
for the file system's permission before assuming flushing from user space
to persistence is allowed.  So that doesn't stick us with 30-40 years of
a flawed model.  I don't think the model is wrong, having spent lots of
research time on it, but if I'm full of crap, all we have to do is stop
telling the app that flushing from user space is allowed and it must go
back to using msync().  This is my understanding of what Dan suggested
at LSF and this is what I'm currently writing up.  By the way, the NVM
Libraries already contain the logic to ask if flushing from user space
is allowed, falling back to msync() if not.  Currently those libraries
check for DAX mappings.  But the points you raised about metadata changes
happening during page faults made us realize we have to ask the file
system to opt-in to allowing user space flushing, so that's what we're
changing the library to do.  See, we are listening :-)

Anyway, I doubt that flushing a clean cache line is extremely expensive.
Remember the code is building transactions to maintain a consistent
in-memory data structure in the face of sudden failure like powerloss.
So it is using the flushes to create store barriers, but not the block-
based store barriers we're used to in the storage world, but cache-line-
sized store barriers (usually multiples of cache lines, but most commonly
smaller than 4k of them).  So I think when you turn a cache line flush
into an msync(), you're seeing some dirty stuff get flushed before it
is time to flush it.  I'm not sure though, but certainly we could spend
more time testing & measuring.

More importantly, I think it is interesting to decide what we want the
pmem programming model to be long-term.  I think we want applications to
just map pmem, do normal stores to it, and assume they are persistent.
This is quite different from the 30-year-old POSIX Model where msync()
is required.  But I think it is cleaner, easier to understand, and less
error-prone.  So why doesn't it work that way right now?  Because we're
finding it impractical.  Using write-through caching for pmem simply
doesn't perform well, and depending on the platform to flush the CPU
caches on shutdown/powerfail is not practical yet.  But I think the day
will come when it is practical.

So given that long-term target, the idea is for an application to ask if
the msync() calls are required, or if just flushing the CPU caches is
sufficient for persistence.  Then, we're also adding an ACPI property
that allows SW to discover if the caches are flushed automatically
on shutdown/powerloss.  Initially that will only be true for custom
platforms, but hopefully it can be available more broadly in the future.
The result will be that the programming model gets simpler as more and
more hardware requires less explicit flushing.

Now I'll go back to writing up the big picture for this programming
model so I can ask you for comments on that as well...


-andy

[toc] | [prev] | [next] | [standalone]

#1393001

On Tue, May 03, 2016 at 01:26:46AM +0000, Rudoff, Andy wrote:
> 
> >> The takeaway is that msync() is 9-10x slower than userspace cache management.
> >
> >An alternative viewpoint: that flushing clean cachelines is
> >extremely expensive on Intel CPUs. ;)
> >
> >i.e. Same numbers, different analysis from a different PoV, and
> >that gives a *completely different conclusion*.
> >
> >Think about it for the moment. The hardware inefficiency being
> >demonstrated could be fixed/optimised in the next hardware product
> >cycle(s) and so will eventually go away. OTOH, we'll be stuck with
> >whatever programming model we come up with for the next 30-40 years,
> >and we'll never be able to fix flaws in it because applications will
> >be depending on them. Do we really want to be stuck with a pmem
> >model that is designed around the flaws and deficiencies of ~1st
> >generation hardware?
> 
> Hi Dave,
> 
> Not sure I agree with your completely different conclusion.  (Not sure
> I completely disagree either, but please let me raise some practical
> points.)
> 
> First of all, let's say you're completely right and flushing clean
> cache lines is extremely expensive.  So your solution is to wait for
> the chip to be fixed? 

No, I'm not saying that's the solution - I'm pointing out that if
clean cache line flushing overhead is less of a problem in future,
the optimisations made now will not be necessary. However, we'll be
still stuck with the API/model that has encoded those optimisations
as a necessary thing for applications to know about and do the
correct thing with. I.e. we end up with a library of applications
that are optimised for a problem that no longer exists...

> Remember the model we're putting forward (which
> we're working on documenting, because I fully agree with the lack of
> documentation point you keep raising) requires the application to ASK
> for the file system's permission before assuming flushing from user space
> to persistence is allowed.

And when the filesystem says no because the fs devs don't want to
have to deal with broken apps because app devs learn that "this is a
go fast knob" and data integrity be damned? It's "fsync is slow so I
won't use it" all over again...

> Anyway, I doubt that flushing a clean cache line is extremely expensive.
> Remember the code is building transactions to maintain a consistent
> in-memory data structure in the face of sudden failure like powerloss.
> So it is using the flushes to create store barriers, but not the block-
> based store barriers we're used to in the storage world, but cache-line-
> sized store barriers (usually multiples of cache lines, but most commonly
> smaller than 4k of them).  So I think when you turn a cache line flush
> into an msync(), you're seeing some dirty stuff get flushed before it
> is time to flush it.  I'm not sure though, but certainly we could spend
> more time testing & measuring.

Sure, but is that what Dan was testing? I don't know - he just
presented a bunch of numbers without a description of the workload,
posting the benchmark code, etc. hence I can only *make assumptions*
about what the numbers mean.

I'm somewhat tired of having to make assumptions because nobody is
describing what they are doing sufficiently and then getting called
out for it, or having to ask lots of questions because other people
have made assumptions about how they think something is going to
work without explaining how the dots connect together. It's a waste
of everyone's time to be playing this ass-u-me game...

The fact that nobody has been able to explain the how the overall
model is supposed to work from physical error all the way out to
userspace makes me think that this is all being made up on the spot.
There are big pieces of the picture missing, and nobody seems to be
able to communicate a clear vision of the architecture we are
supposed to be discussing, let alone implementing...

> More importantly, I think it is interesting to decide what we want the
> pmem programming model to be long-term.  I think we want applications to
> just map pmem, do normal stores to it, and assume they are persistent.
> This is quite different from the 30-year-old POSIX Model where msync()
> is required.

Yes, it's different, but we still have to co-ordinate multiple
layers of persistence (i.e. metadata that references the data).

> But I think it is cleaner, easier to understand, and less
> error-prone.  So why doesn't it work that way right now?  Because we're
> finding it impractical.  Using write-through caching for pmem simply
> doesn't perform well, and depending on the platform to flush the CPU
> caches on shutdown/powerfail is not practical yet.  But I think the day
> will come when it is practical.

Right - it's also simply not practical to intercept every userspace
store to ensure the referencing metadata is also persistent, so we
still need synchronisation mechanisms to ensure that such state is
acheived.  Either that, or the entire dynamic filesystem state needs
to be stored in write-through persistent memory as well. We're a
long, long way from that.

And, please keep in mind: many application developers will not
design for pmem because they also have to support traditional
storage backed by page cache. If they use msync(), the app will work
on any storage stack, but just be much, much faster on pmem+DAX. So,
really, we have to make the msync()-only model work efficiently, so
we may as well design for that in the first place....

> So given that long-term target, the idea is for an application to ask if
> the msync() calls are required, or if just flushing the CPU caches is
> sufficient for persistence.  Then, we're also adding an ACPI property
> that allows SW to discover if the caches are flushed automatically
> on shutdown/powerloss.  Initially that will only be true for custom
> platforms, but hopefully it can be available more broadly in the future.
> The result will be that the programming model gets simpler as more and
> more hardware requires less explicit flushing.

That's a different problem, and one that requires a filesystem to
also store all it's dynamic information in pmem. i.e. there's not
point flushing pmem caches if the powerloss loses dirty metadata
that is held in system RAM. We really need completely new
pmem-native filesystems to make this work - it's a completely
separate problem to whether msync() should be the API that provided
fundamental data integrity guarantees or not.

Which brings up another point: advanced new functionality
is going to require native pmem filesystems. These are unlikely to
be block device based, and instead will directly interface with the
low level CPU and pmem APIs. I don't expect these to use the DAX
infrastructure, either, because that assumes block device based
operations. The will, however, still have to have POSIX compatible
behaviour, and so we go full circle in expecting that an app
written for mmap+DAX on an existing block based filesystem will work
identically on funky new byte-addressable native pmem filesytems.

Encoding cache flushing for data integrity into the userspace
applications assumes that such future pmem-based storage will have
identical persistence requirements to the existing hardware. This,
to me, seems very unlikely to be the case (especially when
considering different platforms (e.g. power, ARM)) and so, again,
application developers are likely to have to fall back to using a
kernel provided data integrity primitive they know they can rely on
(i.e. msync()).....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

[toc] | [prev] | [next] | [standalone]

#1393742

>
>And when the filesystem says no because the fs devs don't want to
>have to deal with broken apps because app devs learn that "this is a
>go fast knob" and data integrity be damned? It's "fsync is slow so I
>won't use it" all over again...
...
>
>And, please keep in mind: many application developers will not
>design for pmem because they also have to support traditional
>storage backed by page cache. If they use msync(), the app will work
>on any storage stack, but just be much, much faster on pmem+DAX. So,
>really, we have to make the msync()-only model work efficiently, so
>we may as well design for that in the first place....

Both of these snippets seem to be arguing that we should make msync/fsync
more efficient.  But I don't think anyone is arguing the opposite.  Is
someone saying we shouldn't make the msync()-only model work efficiently?

Said another way: the common case for DAX will be applications simply
following the POSIX model.  open, mmap, msync...  That will work fine
and of course we should optimize that path as much as possible.  Less
common are latency-sensitive applications built to leverage to byte-
addressable nature of pmem.  File systems supporting this model will
indicate it using a new ioctl that says doing CPU cache flushes is
sufficient to flush stores to persistence.  But I don't see how that
direction is getting turned into an argument against msync() efficiency.

>Which brings up another point: advanced new functionality
>is going to require native pmem filesystems.

I agree there's opportunity for new filesystems (and old) to leverage
what pmem provides.  But the word "require" implies that's the only
way to go and we know that's not the case.  Using ext4+dax to map
pmem into an application allows that application to use the pmem
directly and a good number of software projects are doing exactly that.

-andy

[toc] | [prev] | [next] | [standalone]

#1393891

On Tue, May 03, 2016 at 06:30:04PM +0000, Rudoff, Andy wrote:
> >
> >And when the filesystem says no because the fs devs don't want to
> >have to deal with broken apps because app devs learn that "this is a
> >go fast knob" and data integrity be damned? It's "fsync is slow so I
> >won't use it" all over again...
> ...
> >
> >And, please keep in mind: many application developers will not
> >design for pmem because they also have to support traditional
> >storage backed by page cache. If they use msync(), the app will work
> >on any storage stack, but just be much, much faster on pmem+DAX. So,
> >really, we have to make the msync()-only model work efficiently, so
> >we may as well design for that in the first place....
> 
> Both of these snippets seem to be arguing that we should make msync/fsync
> more efficient.  But I don't think anyone is arguing the opposite.  Is
> someone saying we shouldn't make the msync()-only model work efficiently?

Not directly. The argument presented is "we need a flag to avoid
msync, because msync is inefficient", which is followed by "look,
here's numbers that show msync() being slow, so just give us the
flag already". Experience tells me that the moment a workaround is
in place, nobody will go back and try to fix the problem that the
workaround is mitigating.

Now we know that it's the page granularity cache flushing overhead
that causes the performance differential rather than it being caused
by using msync(), we should be looking at ways to reduce the cache
flushing overhead, not completely bypassing it.

> Said another way: the common case for DAX will be applications simply
> following the POSIX model.  open, mmap, msync...  That will work fine
> and of course we should optimize that path as much as possible.  Less
> common are latency-sensitive applications built to leverage to byte-
> addressable nature of pmem.  File systems supporting this model will
> indicate it using a new ioctl that says doing CPU cache flushes is
> sufficient to flush stores to persistence.

You keep saying this whilst ignoring the repeated comments about how
this can not be guaranteed by all filesystems, and hence apps will
not be able to depend on having such behaviour present. The only
guarantee for persistence that an app will be able to rely on is
msync().

> But I don't see how that
> direction is getting turned into an argument against msync() efficiency.

Promoting a model that works around inefficiency rather than solving
it is no different to saying you don't care about fixing the
inefficiency....

I've said my piece, I'm not going to waste any more time going
around this circle again.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

[toc] | [prev] | [next] | [standalone]

#1392561

On Mon, May 02, 2016 at 11:18:36AM -0400, Jeff Moyer wrote:
> Dave Chinner <david@fromorbit.com> writes:
> 
> > On Mon, Apr 25, 2016 at 11:53:13PM +0000, Verma, Vishal L wrote:
> >> On Tue, 2016-04-26 at 09:25 +1000, Dave Chinner wrote:
> > You're assuming that only the DAX aware application accesses it's
> > files.  users, backup programs, data replicators, fileystem
> > re-organisers (e.g.  defragmenters) etc all may access the files and
> > they may throw errors. What then?
> 
> I'm not sure how this is any different from regular storage.  If an
> application gets EIO, it's up to the app to decide what to do with that.

Sure - they'll fail. But the question I'm asking is that if the
application that owns the data is supposed to do error recovery,
what happens when a 3rd party application hits an error? If that
consumes the error, the the app that owns the data won't ever get a
chance to correct the error.

This is a minefield - a 3rd party app that swallows and clears DAX
based IO errors is a data corruption vector. can yo imagine if
*grep* did this? The model that is being promoted here effectively
allows this sort of behaviour - I don't really think we
should be architecting an error recovery strategy that has the
capability to go this wrong....

> >> > Where does the application find the data that was lost to be able to
> >> > rewrite it?
> >> 
> >> The data that was lost is gone -- this assumes the application has some
> >> ability to recover using a journal/log or other redundancy - yes, at the
> >> application layer. If it doesn't have this sort of capability, the only
> >> option is to restore files from a backup/mirror.
> >
> > So the architecture has a built in assumption that only userspace
> > can handle data loss?
> 
> Remember that the proposed programming model completely bypasses the
> kernel, so yes, it is expected that user-space will have to deal with
> the problem.

No, it doesn't completely bypass the kernel - the kernel is the
infrastructure that catches the errors in the first place, and it
owns and controls all the metadata that corresponds to the physical
location of that error. The only thing the kernel doesn't own is the
*contents* of that location.

> > What about filesytsems like NOVA, that use log structured design to
> > provide DAX w/ update atomicity and can potentially also provide
> > redundancy/repair through the same mechanisms? Won't pmem native
> > filesystems with built in data protection features like this remove
> > the need for adding all this to userspace applications?
> 
> I don't think we'll /only/ support NOVA for pmem.  So we'll have to deal
> with this for existing file systems, right?

Yes, but that misses my point that it seems that the design is only
focussed on userspace and existing filesystems and there is no
consideration of kernel side functionality that could do transparent
recovery....

> > If so, shouldn't that be the focus of development rahter than
> > placing the burden on userspace apps to handle storage repair
> > situations?
> 
> It really depends on the programming model.  In the model Vishal is
> talking about, either the applications themselves or the libraries they
> link to are expected to implement the redundancies where necessary.

IOWs, filesystems no longer have any control over data integrity.
Yet it's the filesystem developers who will still be responsible for
data integrity and when the filesystem has a data coruption event
we'll get blamed and the filesystem gets a bad name, even though
it's entirely the applications fault. We've seen this time and time
again - application developers cannot be trusted to guarantee data
integrity. yes, some apps will be fine, but do you really expect
application devs that refuse to use fsync because it's too slow are
going to have a different approach to integrity when it comes to
DAX?

> >> > There's an implicit assumption that applications will keep redundant
> >> > copies of their data at the /application layer/ and be able to
> >> > automatically repair it?
> 
> That's one way to do things.  It really depends on the application what
> it will do for recovery.
> 
> >> > And then there's the implicit assumption that it will unlink and
> >> > free the entire file before writing a new copy
> 
> I think Vishal was referring to restoring from backup.  cp itself will
> truncate the file before overwriting, iirc.

Which version of cp? what happens if they use --sparse and the error
is in a zeroed region? There's so many assumptions about undefined userspace
environment, application and user behaviour being made here, and
it's all being handwaved away.

I'm asking for this to be defined, demonstrated and documented as a
working model that cannot be abused and doesn't have holes the size
of trucks in it, not handwaving...

> >> To summarize, the two cases we want to handle are:
> >> 1. Application has inbuilt recovery:
> >>   - hits badblock
> >>   - figures out it is able to recover the data
> >>   - handles SIGBUS or EIO
> >>   - does a (sector aligned) write() to restore the data
> >
> > The "figures out" step here is where >95% of the work we'd have to
> > do is. And that's in filesystem and block layer code, not
> > userspace, and userspace can't do that work in a signal handler.
> > And it  can still fall down to the second case when the application
> > doesn't have another copy of the data somewhere.
> 
> I read that "figures out" step as the application determining whether or
> not it had a redundant copy.

Another undocumented assumption, that doesn't simplify what needs to
be done. Indeed, userspace can't do that until it is in SIGBUS
context, which tends to imply applications need to do a major amount
of work from within the signal handler....

> > FWIW, we don't have a DAX enabled filesystem that can do
> > reverse block mapping, so we're a year or two away from this being a
> > workable production solution from the filesystem perspective. And
> > AFAICT, it's not even on the roadmap for dm/md layers.
> 
> Do we even need that?  What if we added an FIEMAP flag for determining
> bad blocks.

So you're assuming that the filesystem has been informed of the bad
blocks and has already marked the bad regions of the file in it's
extent list?

How does that happen? What mechanism is used for the underlying
block device to inform the filesytem that theirs a bad LBA, and how
does the filesytem the map that to a path/file/offset with reverse
mapping? Or is there some other magic that hasn't been explained
happening here?

> The file system could simply walk the list of extents for
> the file and check the corresponding disk blocks.  No reverse mapping
> required.

You're expecting the filesystem to poll the block device to find bad
sectors? Ignoring the fact this is the sort of brute force scan we
need reverse mapping to avoid, how does the filesystem know what
file/extent list it should be searching when the block device
informs it there is a bad sector somewhere? i.e. what information
does the MCE convey to the block device, and what does the block
device pass to the filesytem so the filesystem can do one of these
scans? If the block device is only passing LBAs or a generic "new
bad block has been found" message, the filesystem still has to do an
full scan of it's metadata to find the owner of the LBA(s) that have
gone bad....

Nobody is explaining these important little details - there seems to
be an assumption that everyone "knows" how this is all going to work
and that we have infrastructure that can make it work.

Just because we might be able to present bad block information to
userspace via FIEMAP doesn't mean that it's trivial to implement.
THE FIEMAP flag is trivial - connecting the dots is the hard part
and nobody is explaining to me how that is all supposed to be done.

> Also note that DM/MD don't support direct_access(), either,
> so I don't think they're relevant for this discussion.

But they could for linear concatenation, which would be extremely
useful. e.g. stitching per-node non-linear pmem into a single linear
LBA range....


> >> 2. Application doesn't have any inbuilt recovery mechanism
> >>   - hits badblock
> >>   - gets SIGBUS (or EIO) and crashes
> >>   - Sysadmin restores file from backup
> >
> > Which is no different to an existing non-DAX application getting an
> > EIO/sigbus from current storage technologies.
> >
> > Except: in the existing storage stack, redundancy and correction has
> > already had to have failed for the application to see such an error.
> > Hence this is normally considered a DR case as there's had to be
> > cascading failures (e.g.  multiple disk failures in a RAID) to get
> > to this stage, not a single error in a single sector in
> > non-redundant storage.
> >
> > We need some form of redundancy and correction in the PMEM stack to
> > prevent single sector errors from taking down services until an
> > administrator can correct the problem. I'm trying to understand
> > where this is supposed to fit into the picture - at this point I
> > really don't think userspace applications are going to be able to do
> > this reliably....
> 
> Not all storage is configured into a RAID volume, and in some instances,
> the application is better positioned to recover the data (gluster/ceph,
> for example).

Right, but they still rely on the filesystem to provide data
integrity guarantees to work correctly. While they have "node level"
redundancy, operations within the node still need to work correctly
and so they'd still need all the kernel/fs side functionality to
provide them with error information (like fiemap bad blocks) on top
of all the new error detectiona nd correction code they'd need to
support this...

FWIW, the whole point of DAX on existing filesystems was to not need
major changes to existing filesystems to support fast pmem
operations.  i.e. to get something working quickly while pmem native
filesytems are developed to support pmem and all it's quirks in a
clean and efficient manner.

Instead, what I'm seeing now is a trend towards forcing existing
filesystems to support the requirements and quirks of DAX and pmem,
without any focus on pmem native solutions. i.e. I'm hearing "we
need major surgery to existing filesystems and block devices to make
DAX work" rather than "how do we make this efficient for a pmem
native solution rather than being bound to block device semantics"?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

[toc] | [prev] | [next] | [standalone]

#1392564

On Tue, 2016-05-03 at 09:04 +1000, Dave Chinner wrote:
> On Mon, May 02, 2016 at 11:18:36AM -0400, Jeff Moyer wrote:
> > 
> > Dave Chinner <david@fromorbit.com> writes:
> > 
> > > 
> > > On Mon, Apr 25, 2016 at 11:53:13PM +0000, Verma, Vishal L wrote:
> > > > 
> > > > On Tue, 2016-04-26 at 09:25 +1000, Dave Chinner wrote:
> > > You're assuming that only the DAX aware application accesses it's
> > > files.  users, backup programs, data replicators, fileystem
> > > re-organisers (e.g.  defragmenters) etc all may access the files
> > > and
> > > they may throw errors. What then?
> > I'm not sure how this is any different from regular storage.  If an
> > application gets EIO, it's up to the app to decide what to do with
> > that.
> Sure - they'll fail. But the question I'm asking is that if the
> application that owns the data is supposed to do error recovery,
> what happens when a 3rd party application hits an error? If that
> consumes the error, the the app that owns the data won't ever get a
> chance to correct the error.
> 
> This is a minefield - a 3rd party app that swallows and clears DAX
> based IO errors is a data corruption vector. can yo imagine if
> *grep* did this? The model that is being promoted here effectively
> allows this sort of behaviour - I don't really think we
> should be architecting an error recovery strategy that has the
> capability to go this wrong....
> 

Just to address this bit - No. Any number of backup/3rd party
application can hit the error and _fail_ but surely they won't try to
_write_ the bad location? Only a write to the bad sector will clear it
in this model - and until such time, all reads will just keep erroring
out. This works for DAX/mmap based reads/writes too - mmap-stores
won't/can't clear errors - you have to go through the block path, and in
the altest version of my patch set, that has to be explicitly through
O_DIRECT.

[toc] | [prev] | [next] | [standalone]

#1392566

On Mon, May 2, 2016 at 4:04 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Mon, May 02, 2016 at 11:18:36AM -0400, Jeff Moyer wrote:
>> Dave Chinner <david@fromorbit.com> writes:
>>
>> > On Mon, Apr 25, 2016 at 11:53:13PM +0000, Verma, Vishal L wrote:
>> >> On Tue, 2016-04-26 at 09:25 +1000, Dave Chinner wrote:
>> > You're assuming that only the DAX aware application accesses it's
>> > files.  users, backup programs, data replicators, fileystem
>> > re-organisers (e.g.  defragmenters) etc all may access the files and
>> > they may throw errors. What then?
>>
>> I'm not sure how this is any different from regular storage.  If an
>> application gets EIO, it's up to the app to decide what to do with that.
>
> Sure - they'll fail. But the question I'm asking is that if the
> application that owns the data is supposed to do error recovery,
> what happens when a 3rd party application hits an error? If that
> consumes the error, the the app that owns the data won't ever get a
> chance to correct the error.
>
> This is a minefield - a 3rd party app that swallows and clears DAX
> based IO errors is a data corruption vector. can yo imagine if
> *grep* did this? The model that is being promoted here effectively
> allows this sort of behaviour - I don't really think we
> should be architecting an error recovery strategy that has the
> capability to go this wrong....

Since when does grep write to a file on error?

>
>> >> > Where does the application find the data that was lost to be able to
>> >> > rewrite it?
>> >>
>> >> The data that was lost is gone -- this assumes the application has some
>> >> ability to recover using a journal/log or other redundancy - yes, at the
>> >> application layer. If it doesn't have this sort of capability, the only
>> >> option is to restore files from a backup/mirror.
>> >
>> > So the architecture has a built in assumption that only userspace
>> > can handle data loss?
>>
>> Remember that the proposed programming model completely bypasses the
>> kernel, so yes, it is expected that user-space will have to deal with
>> the problem.
>
> No, it doesn't completely bypass the kernel - the kernel is the
> infrastructure that catches the errors in the first place, and it
> owns and controls all the metadata that corresponds to the physical
> location of that error. The only thing the kernel doesn't own is the
> *contents* of that location.
>
>> > What about filesytsems like NOVA, that use log structured design to
>> > provide DAX w/ update atomicity and can potentially also provide
>> > redundancy/repair through the same mechanisms? Won't pmem native
>> > filesystems with built in data protection features like this remove
>> > the need for adding all this to userspace applications?
>>
>> I don't think we'll /only/ support NOVA for pmem.  So we'll have to deal
>> with this for existing file systems, right?
>
> Yes, but that misses my point that it seems that the design is only
> focussed on userspace and existing filesystems and there is no
> consideration of kernel side functionality that could do transparent
> recovery....
>
>> > If so, shouldn't that be the focus of development rahter than
>> > placing the burden on userspace apps to handle storage repair
>> > situations?
>>
>> It really depends on the programming model.  In the model Vishal is
>> talking about, either the applications themselves or the libraries they
>> link to are expected to implement the redundancies where necessary.
>
> IOWs, filesystems no longer have any control over data integrity.
> Yet it's the filesystem developers who will still be responsible for
> data integrity and when the filesystem has a data coruption event
> we'll get blamed and the filesystem gets a bad name, even though
> it's entirely the applications fault. We've seen this time and time
> again - application developers cannot be trusted to guarantee data
> integrity. yes, some apps will be fine, but do you really expect
> application devs that refuse to use fsync because it's too slow are
> going to have a different approach to integrity when it comes to
> DAX?

Yes, completely agree.  The applications that will implement competent
error recovery with these mechanisms will be vanishingly small, and
there is definite room for a kernel data-redundancy solution that
builds on these patches.

>
>> >> > There's an implicit assumption that applications will keep redundant
>> >> > copies of their data at the /application layer/ and be able to
>> >> > automatically repair it?
>>
>> That's one way to do things.  It really depends on the application what
>> it will do for recovery.
>>
>> >> > And then there's the implicit assumption that it will unlink and
>> >> > free the entire file before writing a new copy
>>
>> I think Vishal was referring to restoring from backup.  cp itself will
>> truncate the file before overwriting, iirc.
>
> Which version of cp? what happens if they use --sparse and the error
> is in a zeroed region? There's so many assumptions about undefined userspace
> environment, application and user behaviour being made here, and
> it's all being handwaved away.
>
> I'm asking for this to be defined, demonstrated and documented as a
> working model that cannot be abused and doesn't have holes the size
> of trucks in it, not handwaving...

You lost me...  how are these patches abusing the existing semantics
of -EIO and write to clear?

>> >> To summarize, the two cases we want to handle are:
>> >> 1. Application has inbuilt recovery:
>> >>   - hits badblock
>> >>   - figures out it is able to recover the data
>> >>   - handles SIGBUS or EIO
>> >>   - does a (sector aligned) write() to restore the data
>> >
>> > The "figures out" step here is where >95% of the work we'd have to
>> > do is. And that's in filesystem and block layer code, not
>> > userspace, and userspace can't do that work in a signal handler.
>> > And it  can still fall down to the second case when the application
>> > doesn't have another copy of the data somewhere.
>>
>> I read that "figures out" step as the application determining whether or
>> not it had a redundant copy.
>
> Another undocumented assumption, that doesn't simplify what needs to
> be done. Indeed, userspace can't do that until it is in SIGBUS
> context, which tends to imply applications need to do a major amount
> of work from within the signal handler....
>
>> > FWIW, we don't have a DAX enabled filesystem that can do
>> > reverse block mapping, so we're a year or two away from this being a
>> > workable production solution from the filesystem perspective. And
>> > AFAICT, it's not even on the roadmap for dm/md layers.
>>
>> Do we even need that?  What if we added an FIEMAP flag for determining
>> bad blocks.
>
> So you're assuming that the filesystem has been informed of the bad
> blocks and has already marked the bad regions of the file in it's
> extent list?
>
> How does that happen? What mechanism is used for the underlying
> block device to inform the filesytem that theirs a bad LBA, and how
> does the filesytem the map that to a path/file/offset with reverse
> mapping? Or is there some other magic that hasn't been explained
> happening here?

In 4.5 we added this:

commit 99e6608c9e7414ae4f2168df8bf8fae3eb49e41f
Author: Vishal Verma <vishal.l.verma@intel.com>
Date:   Sat Jan 9 08:36:51 2016 -0800

    block: Add badblock management for gendisks

    NVDIMM devices, which can behave more like DRAM rather than block
    devices, may develop bad cache lines, or 'poison'. A block device
    exposed by the pmem driver can then consume poison via a read (or
    write), and cause a machine check. On platforms without machine
    check recovery features, this would mean a crash.

    The block device maintaining a runtime list of all known sectors that
    have poison can directly avoid this, and also provide a path forward
    to enable proper handling/recovery for DAX faults on such a device.

    Use the new badblock management interfaces to add a badblocks list to
    gendisks.

    Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
    Signed-off-by: Dan Williams <dan.j.williams@intel.com>

[toc] | [prev] | [next] | [standalone]

#1392963

On Mon, May 02, 2016 at 04:25:51PM -0700, Dan Williams wrote:
> On Mon, May 2, 2016 at 4:04 PM, Dave Chinner <david@fromorbit.com> wrote:
> > On Mon, May 02, 2016 at 11:18:36AM -0400, Jeff Moyer wrote:
> >> Dave Chinner <david@fromorbit.com> writes:
> >>
> >> > On Mon, Apr 25, 2016 at 11:53:13PM +0000, Verma, Vishal L wrote:
> >> >> On Tue, 2016-04-26 at 09:25 +1000, Dave Chinner wrote:
> >> > You're assuming that only the DAX aware application accesses it's
> >> > files.  users, backup programs, data replicators, fileystem
> >> > re-organisers (e.g.  defragmenters) etc all may access the files and
> >> > they may throw errors. What then?
> >>
> >> I'm not sure how this is any different from regular storage.  If an
> >> application gets EIO, it's up to the app to decide what to do with that.
> >
> > Sure - they'll fail. But the question I'm asking is that if the
> > application that owns the data is supposed to do error recovery,
> > what happens when a 3rd party application hits an error? If that
> > consumes the error, the the app that owns the data won't ever get a
> > chance to correct the error.
> >
> > This is a minefield - a 3rd party app that swallows and clears DAX
> > based IO errors is a data corruption vector. can yo imagine if
> > *grep* did this? The model that is being promoted here effectively
> > allows this sort of behaviour - I don't really think we
> > should be architecting an error recovery strategy that has the
> > capability to go this wrong....
> 
> Since when does grep write to a file on error?

That's precisely my point - it doesn't right now because there is no
onus on userspace applications to correct data errors when they are
found.

However, if the accepted model becomes "userspace needs to try to correct
errors in data automatically", the above scenario is a distinct
possiblity. I'm not saying grep will do this - I'm taking
the logical argument being presented to the extreme - but I'm sure
that there will be developers that have enough knowledge to know
they are supposed to do something with errors on pmem devices, but
not have enough knowledge to know the correct things to do.

And then the app mishandles a EINVAL error (or something like that)
and so we end up with buggy userspace apps trying to correct
errors in good data and causing data loss that way.

Do we really want to introduce a data integrity and error recovery
model where this sort of "bug" is a distinct possibly?

> >> >> > There's an implicit assumption that applications will keep redundant
> >> >> > copies of their data at the /application layer/ and be able to
> >> >> > automatically repair it?
> >>
> >> That's one way to do things.  It really depends on the application what
> >> it will do for recovery.
> >>
> >> >> > And then there's the implicit assumption that it will unlink and
> >> >> > free the entire file before writing a new copy
> >>
> >> I think Vishal was referring to restoring from backup.  cp itself will
> >> truncate the file before overwriting, iirc.
> >
> > Which version of cp? what happens if they use --sparse and the error
> > is in a zeroed region? There's so many assumptions about undefined userspace
> > environment, application and user behaviour being made here, and
> > it's all being handwaved away.
> >
> > I'm asking for this to be defined, demonstrated and documented as a
> > working model that cannot be abused and doesn't have holes the size
> > of trucks in it, not handwaving...
> 
> You lost me...  how are these patches abusing the existing semantics
> of -EIO and write to clear?

I haven't said that. I said there are assumptions about how
userspace will handle the error, but they aren't documented
anywhere. "copy a file using cp" is not a robust recovery solution -
it provides no guarantees about how the bad file and regions will be
recycled and the errors cleared. This effectively of puts it all on
the filesystems to deal with, even though you're trying to design an
error handling model that bypasses the filesystems and goes straight
to userspace.

If I can't understand how this is all supposed to work
because none of it is documented, then we have no chance that the
average admin is going to be able to understand it.

> 
> >> >> To summarize, the two cases we want to handle are:
> >> >> 1. Application has inbuilt recovery:
> >> >>   - hits badblock
> >> >>   - figures out it is able to recover the data
> >> >>   - handles SIGBUS or EIO
> >> >>   - does a (sector aligned) write() to restore the data
> >> >
> >> > The "figures out" step here is where >95% of the work we'd have to
> >> > do is. And that's in filesystem and block layer code, not
> >> > userspace, and userspace can't do that work in a signal handler.
> >> > And it  can still fall down to the second case when the application
> >> > doesn't have another copy of the data somewhere.
> >>
> >> I read that "figures out" step as the application determining whether or
> >> not it had a redundant copy.
> >
> > Another undocumented assumption, that doesn't simplify what needs to
> > be done. Indeed, userspace can't do that until it is in SIGBUS
> > context, which tends to imply applications need to do a major amount
> > of work from within the signal handler....
> >
> >> > FWIW, we don't have a DAX enabled filesystem that can do
> >> > reverse block mapping, so we're a year or two away from this being a
> >> > workable production solution from the filesystem perspective. And
> >> > AFAICT, it's not even on the roadmap for dm/md layers.
> >>
> >> Do we even need that?  What if we added an FIEMAP flag for determining
> >> bad blocks.
> >
> > So you're assuming that the filesystem has been informed of the bad
> > blocks and has already marked the bad regions of the file in it's
> > extent list?
> >
> > How does that happen? What mechanism is used for the underlying
> > block device to inform the filesytem that theirs a bad LBA, and how
> > does the filesytem the map that to a path/file/offset with reverse
> > mapping? Or is there some other magic that hasn't been explained
> > happening here?
> 
> In 4.5 we added this:
> 
> commit 99e6608c9e7414ae4f2168df8bf8fae3eb49e41f
> Author: Vishal Verma <vishal.l.verma@intel.com>
> Date:   Sat Jan 9 08:36:51 2016 -0800
> 
>     block: Add badblock management for gendisks

Yes, I know, and it doesn't answer any of the questions I just
asked. What you just told me is that there is something that is kept
three levels of abstraction away from a filesystem. So:

	- What mechanism is to be used for the underlying block
	  device to inform the filesytem that a new bad block was
	  added to this list? What context comes along with that
	  notification?
	- how does the filesystem query the bad block list without
	  adding layering violations?
	- when does the filesystem need to query the bad block list?
	- how will the bad block list propagate through DM/MD
	  layers?
	- how does the filesytem the map the bad block to a
	  path/file/offset without reverse mapping - does this error
	  handling interface really imply the filesystem needs to
	  implement brute force scans at notification time?
	- Is the filesystem expectd to find the active application or
	  address_space access that triggered the bad block
	  notification to handle them correctly? (e.g. prevent a
	  page fault from failing because we can recover from the
	  error immediately)
	- what exactly is the filesystem supposed to do with the bad
	  block? e.g:
		- is the block persistently bad until the filesystem
		  rewrites it? Over power cycles? Will we get
		  multiple notifications (e.g. once per boot)?
		- Is the filesystem supposed to intercept
		  reads/writes to bad blocks once it knows about
		  them?
		- how is the filesystem supposed to communicate that
		  there is a bad block in a file back to userspace?
		  Or is userspace supposed to infer that there's a
		  bad block from EIO and so has to run FIEMAP to
		  determine if the error really was due to a bad
		  block?
		- what happens if there is no running application
		  that we can report the error to or will handle the
		  error (e.g. found error by a media scrub or during
		  boot)?
	- if the bad block is in filesystem free space, what should
	  the filesystem do with it?

What I'm failing to communicate is that having and maintaining
things like bad block lists in a block device is the easy part of
the problem.

Similarly reporting a bad block flag in FIEMAP is only a few
lines of code to implement, but that assumes the filesystem has
already propagated the bad block information into it's internal
extents lists.

That's the hard part of all this: connecting the two pieces together
in a sane, reliable, consistent and useful manner. This will form
the user API, so we need to sort it out before applications start to
use it. However, if I'm struggling to understand how I'm supposed to
connecct up the parts inside a filesytem, then expecting application
developers to be able to connect the dots in a sane manner is
bordering on fantasy....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

[toc] | [prev] | [next] | [standalone]

#1393639

On Mon, May 2, 2016 at 6:51 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Mon, May 02, 2016 at 04:25:51PM -0700, Dan Williams wrote:
[..]
> Yes, I know, and it doesn't answer any of the questions I just
> asked. What you just told me is that there is something that is kept
> three levels of abstraction away from a filesystem. So:

Ok, let's answer them.

A lot of your questions seem to assume the filesystem has a leading
role to play with error recovery, that isn't the case with traditional
disk errors and we're not looking to change that situation.  The
filesystem can help with forensics after an error escapes the kernel
and is communicated to userspace, but the ability to reverse map a
sector to a file is just a convenience to identify potential data
loss.

For redundancy in the DAX case I can envision DAX-aware RAID that
makes the potential exposure to bad blocks smaller, but it will always
be the case that the array can be out-of-sync / degraded and has no
choice but to communicate the error to userspace.  So, the answers
below address what we do when we are in that state, and include some
thoughts about follow-on enabling we can do at the DM/MD layer.

>         - What mechanism is to be used for the underlying block
>           device to inform the filesytem that a new bad block was
>           added to this list?

The filesystem doesn't need this notification and doesn't get it today
from RAID.  It's handy for the bad block list to be available to
fs/dax.c and the block layer, but I don't see ext4/xfs having a role
to play with the list and certainly not care about "new error detected
events".  For a DM/MD driver it also does not need to know about new
errors because it will follow the traditional disk model where errors
are handled on access, or discovered and scrubbed during a periodic
array scan.

That said, new errors may get added to the list by having the pmem
driver trigger a rescan of the device whenever a latent error is
discovered (i.e. memcpy_from_pmem() returns -EIO).  The update of the
bad block list is asynchronous.  We also have a task on the todo list
to allow the pmem rescan action to be triggered via sysfs.

>           What context comes along with that
>           notification?

The only notification the file system gets is -EIO on access.
However, assuming we had a DAX-aware RAID driver what infrastructure
would we need to prevent SIGBUS from reaching the application if we
happened to have a redundant copy of the data?

One feature we've talked about for years at LSF/MM but never made any
progress on is a way for a file system to discover and query if the
storage layer can reconstruct data from redundant information.
Assuming we had such an interface there's still the matter of plumbing
a machine check fault through a physical-address-to-sector conversion
and request the block device driver to attempt to provide a redundant
copy.

The in-kernel recovery path, assuming RAID is present, needs more
thought especially considering the limited NMI context of a machine
check notification and the need to trap back into driver code.  I see
the code in fs/dax.c getting involved to translate a
process-physical-address back to a sector, but otherwise the rest of
the filesystem need not be involved.

>         - how does the filesystem query the bad block list without
>           adding layering violations?

Why does the file system need to read the list?

Apologies for answering this question with a question, but these
patches don't assume the filesystem will do anything with a bad block
list.

>         - when does the filesystem need to query the bad block list?
>         - how will the bad block list propagate through DM/MD
>           layers?
>         - how does the filesytem the map the bad block to a
>           path/file/offset without reverse mapping - does this error
>           handling interface really imply the filesystem needs to
>           implement brute force scans at notification time?

No, there is no implication that reverse mapping is a requirement.

>         - Is the filesystem expectd to find the active application or
>           address_space access that triggered the bad block
>           notification to handle them correctly? (e.g. prevent a
>           page fault from failing because we can recover from the
>           error immediately)

With these patches no, but it would be nice to incrementally add that
ability.  I.e. trap machine check faults on non-anonymous memory and
send a request down the stack to recover the sector if the storage
layer has a redundant copy.  Again, fs/dax.c would need extensions to
do this coordination, but I don't foresee the filesystem getting
involved beyond that point.

>         - what exactly is the filesystem supposed to do with the bad
>           block? e.g:
>                 - is the block persistently bad until the filesystem
>                   rewrites it? Over power cycles? Will we get
>                   multiple notifications (e.g. once per boot)?

Bad blocks on persistent memory media remain bad after a reboot.  Per
the ACPI spec the DIMM device tracks the errors and reports them in
response to an "address range scrub" command.  Each boot the libnvdimm
sub-system kicks off a scrub and populates the bad block list per pmem
namespace.  As mentioned above, we want to add the ability to
re-trigger this scrub on-demand, in response to a memcpy_from_pmem()
discovering an error, or after a SIGBUS is communicated to userspace.

>                 - Is the filesystem supposed to intercept
>                   reads/writes to bad blocks once it knows about
>                   them?

No, the driver handles that.

>                 - how is the filesystem supposed to communicate that
>                   there is a bad block in a file back to userspace?

-EIO on access.

>                   Or is userspace supposed to infer that there's a
>                   bad block from EIO and so has to run FIEMAP to
>                   determine if the error really was due to a bad
>                   block?

The information is there to potentially do forensics on why an I/O
encountered an error, but there is no expectation that userspace
follow up on each -EIO with a FIEMAP.

>                 - what happens if there is no running application
>                   that we can report the error to or will handle the
>                   error (e.g. found error by a media scrub or during
>                   boot)?

Same as RAID today, if the array is in sync the bad block will get
re-written during the scrub hopefully in advance of when an
application might discover it.  If no RAID is present then the only
notification is an on-access error.

>         - if the bad block is in filesystem free space, what should
>           the filesystem do with it?

Nothing.  When the free space becomes allocated we rely on the fact
that the filesystem will first zero the blocks.  That zeroing process
will clear the media error.

Now, in rare cases clearing the error might itself fail, but in that
case we just degenerate to the latent error discovery case.

> What I'm failing to communicate is that having and maintaining
> things like bad block lists in a block device is the easy part of
> the problem.
>
> Similarly reporting a bad block flag in FIEMAP is only a few
> lines of code to implement, but that assumes the filesystem has
> already propagated the bad block information into it's internal
> extents lists.
>
> That's the hard part of all this: connecting the two pieces together
> in a sane, reliable, consistent and useful manner. This will form
> the user API, so we need to sort it out before applications start to
> use it.

Applications are already using most of this model today.  The new
enabling we should consider is a way to take advantage of redundancy
at the storage driver layer to prevent errors from being reported to
userspace when redundant data is available.  Preventing
machine-check-SIGBUS signals from reaching applications is a new
general purpose error handling mechanism that might also be useful for
DRAM errors outside of pmem+DAX.

> However, if I'm struggling to understand how I'm supposed to
> connecct up the parts inside a filesytem, then expecting application
> developers to be able to connect the dots in a sane manner is
> bordering on fantasy....

Hopefully it is becoming clearer that we are not proposing anything
radically different than what is present for error recovery today
modulo thinking about the mechanisms to trap and recover a DAX read of
a bad media area via a DM/MD implementation.  DAX writes on the other
hand are more challenging in that we'd likely want to stage them and
wait to commit them until an explicit sync point.  However, this is
still consistent with the DAX programming model.  An application is
free to trade off raw access to the media for higher-order filesystem
and storage layer features provided by the kernel.

[toc] | [prev] | [next] | [standalone]

#1393916

On Tue, May 03, 2016 at 10:28:15AM -0700, Dan Williams wrote:
> On Mon, May 2, 2016 at 6:51 PM, Dave Chinner <david@fromorbit.com> wrote:
> > On Mon, May 02, 2016 at 04:25:51PM -0700, Dan Williams wrote:
> [..]
> > Yes, I know, and it doesn't answer any of the questions I just
> > asked. What you just told me is that there is something that is kept
> > three levels of abstraction away from a filesystem. So:
> 
> Ok, let's answer them.
> 
> A lot of your questions seem to assume the filesystem has a leading
> role to play with error recovery, that isn't the case with traditional
> disk errors and we're not looking to change that situation. 

*cough* BTRFS

New filesystems are mostly being designed with redundancy and
recovery mechanisms built into them. Hence the high level
/assumption/ that filesystems aren't going to play a significant
role in error recovery for pmem storage is, well, somewhat
disturbing....

> The
> filesystem can help with forensics after an error escapes the kernel
> and is communicated to userspace, but the ability to reverse map a
> sector to a file is just a convenience to identify potential data
> loss.

So working out what file got corrupted in your terabytes of pmem
storage is "just a convenience"? I suspect that a rather large
percentage of admins will disagree with you on this.

> For redundancy in the DAX case I can envision DAX-aware RAID that
> makes the potential exposure to bad blocks smaller, but it will always
> be the case that the array can be out-of-sync / degraded and has no
> choice but to communicate the error to userspace.  So, the answers
> below address what we do when we are in that state, and include some
> thoughts about follow-on enabling we can do at the DM/MD layer.
> 
> >         - What mechanism is to be used for the underlying block
> >           device to inform the filesytem that a new bad block was
> >           added to this list?
> 
> The filesystem doesn't need this notification and doesn't get it today
> from RAID.

Why doesn't the filesystem need this notification? Just because we
don't get it today from a RAID device does not mean we can't use it.

Indeed, think about the btrfs scrub operation - it validates
everything on it's individual block devices, and when it finds a
problem (e.g. a data CRC error) it notifies a different layer in the
btrfs code that goes and works out if it can repair the problem from
redundant copies/parity/mirrors/etc.

> It's handy for the bad block list to be available to
> fs/dax.c and the block layer, but I don't see ext4/xfs having a role
> to play with the list and certainly not care about "new error detected
> events". 

That's very short-sighted. Just because ext4/xfs don't *currently*
do this, it doesn't mean other filesystems (existing or new) can't
make use of notifications, nor that ext4/XFS can't ever make use of
it, either.

> For a DM/MD driver it also does not need to know about new
> errors because it will follow the traditional disk model where errors
> are handled on access, or discovered and scrubbed during a periodic
> array scan.
>
> That said, new errors may get added to the list by having the pmem
> driver trigger a rescan of the device whenever a latent error is
> discovered (i.e. memcpy_from_pmem() returns -EIO).  The update of the
> bad block list is asynchronous.  We also have a task on the todo list
> to allow the pmem rescan action to be triggered via sysfs.

IOWs, the pmem driver won't report errors to anyone who can correct
them until an access to that bad block is made? Even if it means the
error might go unreported and hence uncorrected for weeks or months
because no access is made to that bad data?

> >           What context comes along with that
> >           notification?
> 
> The only notification the file system gets is -EIO on access.
> However, assuming we had a DAX-aware RAID driver what infrastructure
> would we need to prevent SIGBUS from reaching the application if we
> happened to have a redundant copy of the data?

We'd need the same infrastructure at the filesystem layer would
require if it has a redundant copy of the data. I don't know what
that is, however, because I know very little about about MCEs and
signal delivery (which is why I asked this question).

[....]

> The in-kernel recovery path, assuming RAID is present, needs more
> thought especially considering the limited NMI context of a machine
> check notification and the need to trap back into driver code.

This is precisely the problem I am asking about - I know there is a
limited context, but how exactly is it limited and what can we
actually do from this context? e.g. Can we schedule recovery work on
other CPU cores and wait for it to complete in a MCE notification
handler?

> I see
> the code in fs/dax.c getting involved to translate a
> process-physical-address back to a sector, but otherwise the rest of
> the filesystem need not be involved.

More /assumptions/ about filesystems not containing or being able to
recover from redudant copies of data.

> 
> >         - how does the filesystem query the bad block list without
> >           adding layering violations?
> 
> Why does the file system need to read the list?
> Apologies for answering this question with a question, but these
> patches don't assume the filesystem will do anything with a bad block
> list.

People keep talking about FIEMAP reporting bad blocks in files! How
the fuck are we supposed to report bad blocks in a file via FIEMAP
if the filesystem can't access the bad block list?

> >         - Is the filesystem expectd to find the active application or
> >           address_space access that triggered the bad block
> >           notification to handle them correctly? (e.g. prevent a
> >           page fault from failing because we can recover from the
> >           error immediately)
> 
> With these patches no, but it would be nice to incrementally add that
> ability.  I.e. trap machine check faults on non-anonymous memory and
> send a request down the stack to recover the sector if the storage
> layer has a redundant copy.  Again, fs/dax.c would need extensions to
> do this coordination, but I don't foresee the filesystem getting
> involved beyond that point.

Again, the /assumption/ here is that only the block layer has the
ability to recover, and only sector mapping is required from the fs.

> >         - what exactly is the filesystem supposed to do with the bad
> >           block? e.g:
> >                 - is the block persistently bad until the filesystem
> >                   rewrites it? Over power cycles? Will we get
> >                   multiple notifications (e.g. once per boot)?
> 
> Bad blocks on persistent memory media remain bad after a reboot.  Per
> the ACPI spec the DIMM device tracks the errors and reports them in
> response to an "address range scrub" command.  Each boot the libnvdimm
> sub-system kicks off a scrub and populates the bad block list per pmem
> namespace.  As mentioned above, we want to add the ability to
> re-trigger this scrub on-demand, in response to a memcpy_from_pmem()
> discovering an error, or after a SIGBUS is communicated to userspace.
> 
> >                 - Is the filesystem supposed to intercept
> >                   reads/writes to bad blocks once it knows about
> >                   them?
> 
> No, the driver handles that.

So, -EIO will be returned to the filesystem on access? If -EIO, then
we'll have to check over the bad block list to determine if data
recovery operations are required, right? Perhaps we need a different
error here to tell the higher layers it's a specific type of error
(e.g. -EBADBLOCK)?

> >                 - how is the filesystem supposed to communicate that
> >                   there is a bad block in a file back to userspace?
> 
> -EIO on access.

So no consideration for proactive "data loss has occurred at offset X
in file /mnt/path/to/file, attempting recovery" messages when the
error is first detected by the lowest layers?

> >                   Or is userspace supposed to infer that there's a
> >                   bad block from EIO and so has to run FIEMAP to
> >                   determine if the error really was due to a bad
> >                   block?
> 
> The information is there to potentially do forensics on why an I/O
> encountered an error, but there is no expectation that userspace
> follow up on each -EIO with a FIEMAP.

Ok, so how is userspace driven error recovery supposed to work if it
can't differentiate the cause of an EIO error? If there's no
requirement for FIEMAP to report the bad blocks in a file that needs
recovery, then what is the app supposed to do with the EIO? Indeed,
what consideration has been given to ensuring the app knows aheadi
of time that the filesystem FIEMAP implementation will report bad
blocks if they exist?

Of course, the filesystem has to know about the bad blocks to be
able to do any of this with FIEMAP....

> >                 - what happens if there is no running application
> >                   that we can report the error to or will handle the
> >                   error (e.g. found error by a media scrub or during
> >                   boot)?
> 
> Same as RAID today, if the array is in sync the bad block will get
> re-written during the scrub hopefully in advance of when an
> application might discover it.  If no RAID is present then the only
> notification is an on-access error.

More /assumptions/ that only device level RAID will be able to
recover....

> >         - if the bad block is in filesystem free space, what should
> >           the filesystem do with it?
> 
> Nothing.  When the free space becomes allocated we rely on the fact
> that the filesystem will first zero the blocks.  That zeroing process
> will clear the media error.

The incorrect /assumption/ here is that all allocations will do
block zeroing first. That's simply wrong. We do that for *user data*
in XFS and ext4, but we do not do it for metadata as they are not
accessed by DAX and, being transactionally protected, don't need
zeroing to prevent stale data exposure.

Hence we have a problem here - the first write to such blocks may
be metadata writeback of some type and so the filesystem will see
EIO errors in metadata writes and they'll freak out. What
now - does this really mean that we'll have to add special IO
falback code for all internal IO paths to be able to clear pmem bad
block errors?

Oh, and just a thought: lots of people are pushing for selectable
FALLOC_FL_NO_HIDE_STALE behaviour which will skip zeroing
of data blocks on allocation. If this happens, it we also skip
the zeroing on allocation, so again there is no mechanism to clear
bad block status in this case.....

[...]

> > However, if I'm struggling to understand how I'm supposed to
> > connecct up the parts inside a filesytem, then expecting application
> > developers to be able to connect the dots in a sane manner is
> > bordering on fantasy....
> 
> Hopefully it is becoming clearer that we are not proposing anything
> radically different than what is present for error recovery today
> modulo thinking about the mechanisms to trap and recover a DAX read of
> a bad media area via a DM/MD implementation.

There is a radical difference - there is a pervasive /assumption/ in
what is being proposed that filesystems are incapable of storing
redundant information that can be used for error recovery. The
current IO stack makes no such assumptions, even if it doesn't
provide any infrastructure for such functionality.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

[toc] | [prev] | [next] | [standalone]

#1393950

On Tue, May 3, 2016 at 8:18 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Tue, May 03, 2016 at 10:28:15AM -0700, Dan Williams wrote:
>> On Mon, May 2, 2016 at 6:51 PM, Dave Chinner <david@fromorbit.com> wrote:
>> > On Mon, May 02, 2016 at 04:25:51PM -0700, Dan Williams wrote:
>> [..]
>> > Yes, I know, and it doesn't answer any of the questions I just
>> > asked. What you just told me is that there is something that is kept
>> > three levels of abstraction away from a filesystem. So:
>>
>> Ok, let's answer them.
>>
>> A lot of your questions seem to assume the filesystem has a leading
>> role to play with error recovery, that isn't the case with traditional
>> disk errors and we're not looking to change that situation.
>
> *cough* BTRFS
>
> New filesystems are mostly being designed with redundancy and
> recovery mechanisms built into them. Hence the high level
> /assumption/ that filesystems aren't going to play a significant
> role in error recovery for pmem storage is, well, somewhat
> disturbing....

It is unfortunate is that you cite the lack of pmem enabling in btrfs
as a reason to block patches that hookup the kernel's existing error
mechanisms for the DAX case.  I expect btrfs multi-device
redundancy-management for pmem to be a more a coherent solution than
what we can achieve with single-device-filesystems + RAID.  I'm trying
not to boil the ocean in this discussion, but Iet's go ahead and rope
in btrfs-devel into this thread so we can make progress on hooking up
SIGBUS notifications for DAX errors and bypassing dax_do_io() to clear
errors.

>> The
>> filesystem can help with forensics after an error escapes the kernel
>> and is communicated to userspace, but the ability to reverse map a
>> sector to a file is just a convenience to identify potential data
>> loss.
>
> So working out what file got corrupted in your terabytes of pmem
> storage is "just a convenience"? I suspect that a rather large
> percentage of admins will disagree with you on this.

Yes, I will point them to their file system maintainer to ask about
reverse mapping support.

>> For redundancy in the DAX case I can envision DAX-aware RAID that
>> makes the potential exposure to bad blocks smaller, but it will always
>> be the case that the array can be out-of-sync / degraded and has no
>> choice but to communicate the error to userspace.  So, the answers
>> below address what we do when we are in that state, and include some
>> thoughts about follow-on enabling we can do at the DM/MD layer.
>>
>> >         - What mechanism is to be used for the underlying block
>> >           device to inform the filesytem that a new bad block was
>> >           added to this list?
>>
>> The filesystem doesn't need this notification and doesn't get it today
>> from RAID.
>
> Why doesn't the filesystem need this notification? Just because we
> don't get it today from a RAID device does not mean we can't use it.

If xfs and ext4 had a use for error notification today we would hook into it.

> Indeed, think about the btrfs scrub operation - it validates
> everything on it's individual block devices, and when it finds a
> problem (e.g. a data CRC error) it notifies a different layer in the
> btrfs code that goes and works out if it can repair the problem from
> redundant copies/parity/mirrors/etc.

Yes, just like RAID, sounds like we should definitely keep that in
mind for the patch set that adds pmem support to btrfs, this isn't
that patch set.

>> It's handy for the bad block list to be available to
>> fs/dax.c and the block layer, but I don't see ext4/xfs having a role
>> to play with the list and certainly not care about "new error detected
>> events".
>
> That's very short-sighted. Just because ext4/xfs don't *currently*
> do this, it doesn't mean other filesystems (existing or new) can't
> make use of notifications, nor that ext4/XFS can't ever make use of
> it, either.

Did I say "can't ever make use of it", no, if you have a need for a
notification for xfs let's work on a notification mechanism.

>
>> For a DM/MD driver it also does not need to know about new
>> errors because it will follow the traditional disk model where errors
>> are handled on access, or discovered and scrubbed during a periodic
>> array scan.
>>
>> That said, new errors may get added to the list by having the pmem
>> driver trigger a rescan of the device whenever a latent error is
>> discovered (i.e. memcpy_from_pmem() returns -EIO).  The update of the
>> bad block list is asynchronous.  We also have a task on the todo list
>> to allow the pmem rescan action to be triggered via sysfs.
>
> IOWs, the pmem driver won't report errors to anyone who can correct
> them until an access to that bad block is made? Even if it means the
> error might go unreported and hence uncorrected for weeks or months
> because no access is made to that bad data?

RAID periodically polls for and fixes bad blocks.  The currently
implementation only polls for errors at driver load.  When we
implement userspace triggered bad blocks scans we could also have a
cron job to periodically kick off a scan, which follows the status quo
for RAID error scanning.

>
>> >           What context comes along with that
>> >           notification?
>>
>> The only notification the file system gets is -EIO on access.
>> However, assuming we had a DAX-aware RAID driver what infrastructure
>> would we need to prevent SIGBUS from reaching the application if we
>> happened to have a redundant copy of the data?
>
> We'd need the same infrastructure at the filesystem layer would
> require if it has a redundant copy of the data. I don't know what
> that is, however, because I know very little about about MCEs and
> signal delivery (which is why I asked this question).

Fair enough.

> [....]
>
>> The in-kernel recovery path, assuming RAID is present, needs more
>> thought especially considering the limited NMI context of a machine
>> check notification and the need to trap back into driver code.
>
> This is precisely the problem I am asking about - I know there is a
> limited context, but how exactly is it limited and what can we
> actually do from this context? e.g. Can we schedule recovery work on
> other CPU cores and wait for it to complete in a MCE notification
> handler?
>
>> I see
>> the code in fs/dax.c getting involved to translate a
>> process-physical-address back to a sector, but otherwise the rest of
>> the filesystem need not be involved.
>
> More /assumptions/ about filesystems not containing or being able to
> recover from redudant copies of data.
>
>>
>> >         - how does the filesystem query the bad block list without
>> >           adding layering violations?
>>
>> Why does the file system need to read the list?
>> Apologies for answering this question with a question, but these
>> patches don't assume the filesystem will do anything with a bad block
>> list.
>
> People keep talking about FIEMAP reporting bad blocks in files! How
> the fuck are we supposed to report bad blocks in a file via FIEMAP
> if the filesystem can't access the bad block list?

Compare FIEMAP results against the badblocks list, or arrange for
FIEMAP to return the errors in file list by parsing the list against
the badblocks list available in the gendisk.  Shall I send a patch?
It does assume we can do a inode to bdev lookup.

>> >         - Is the filesystem expectd to find the active application or
>> >           address_space access that triggered the bad block
>> >           notification to handle them correctly? (e.g. prevent a
>> >           page fault from failing because we can recover from the
>> >           error immediately)
>>
>> With these patches no, but it would be nice to incrementally add that
>> ability.  I.e. trap machine check faults on non-anonymous memory and
>> send a request down the stack to recover the sector if the storage
>> layer has a redundant copy.  Again, fs/dax.c would need extensions to
>> do this coordination, but I don't foresee the filesystem getting
>> involved beyond that point.
>
> Again, the /assumption/ here is that only the block layer has the
> ability to recover, and only sector mapping is required from the fs.

Presently yes, in the future, no.

>
>> >         - what exactly is the filesystem supposed to do with the bad
>> >           block? e.g:
>> >                 - is the block persistently bad until the filesystem
>> >                   rewrites it? Over power cycles? Will we get
>> >                   multiple notifications (e.g. once per boot)?
>>
>> Bad blocks on persistent memory media remain bad after a reboot.  Per
>> the ACPI spec the DIMM device tracks the errors and reports them in
>> response to an "address range scrub" command.  Each boot the libnvdimm
>> sub-system kicks off a scrub and populates the bad block list per pmem
>> namespace.  As mentioned above, we want to add the ability to
>> re-trigger this scrub on-demand, in response to a memcpy_from_pmem()
>> discovering an error, or after a SIGBUS is communicated to userspace.
>>
>> >                 - Is the filesystem supposed to intercept
>> >                   reads/writes to bad blocks once it knows about
>> >                   them?
>>
>> No, the driver handles that.
>
> So, -EIO will be returned to the filesystem on access? If -EIO, then
> we'll have to check over the bad block list to determine if data
> recovery operations are required, right? Perhaps we need a different
> error here to tell the higher layers it's a specific type of error
> (e.g. -EBADBLOCK)?

That sounds reasonable.

>> >                 - how is the filesystem supposed to communicate that
>> >                   there is a bad block in a file back to userspace?
>>
>> -EIO on access.
>
> So no consideration for proactive "data loss has occurred at offset X
> in file /mnt/path/to/file, attempting recovery" messages when the
> error is first detected by the lowest layers?

That sounds reasonable too, what does xfs report today for disk errors?

>> >                   Or is userspace supposed to infer that there's a
>> >                   bad block from EIO and so has to run FIEMAP to
>> >                   determine if the error really was due to a bad
>> >                   block?
>>
>> The information is there to potentially do forensics on why an I/O
>> encountered an error, but there is no expectation that userspace
>> follow up on each -EIO with a FIEMAP.
>
> Ok, so how is userspace driven error recovery supposed to work if it
> can't differentiate the cause of an EIO error? If there's no
> requirement for FIEMAP to report the bad blocks in a file that needs
> recovery, then what is the app supposed to do with the EIO? Indeed,
> what consideration has been given to ensuring the app knows aheadi
> of time that the filesystem FIEMAP implementation will report bad
> blocks if they exist?

So, having a new flavor of FIEMAP that reports bad blocks in a file,
or a new SEEK_BADBLOCK flag for lseek() was proposed for situations
where an application could not simply take a list of badblocks from
sysfs and do an intersection operation with typical FIEMAP results to
see if a file was impacted.

Indeed btrfs is an example where the per-block-device list is unusable
for an application to do this FIEMAP intersection operation, but an
application on a single-device-ext4 filesystem could make use of the
list.

> Of course, the filesystem has to know about the bad blocks to be
> able to do any of this with FIEMAP....

Is there some complication about looking up the gendisk from the inode
that I'm overlooking?  Is there a different location we need to place
the error list to make it easy for the fs to consume?

>> >                 - what happens if there is no running application
>> >                   that we can report the error to or will handle the
>> >                   error (e.g. found error by a media scrub or during
>> >                   boot)?
>>
>> Same as RAID today, if the array is in sync the bad block will get
>> re-written during the scrub hopefully in advance of when an
>> application might discover it.  If no RAID is present then the only
>> notification is an on-access error.
>
> More /assumptions/ that only device level RAID will be able to
> recover....

Presently yes, in the future, no.

>
>> >         - if the bad block is in filesystem free space, what should
>> >           the filesystem do with it?
>>
>> Nothing.  When the free space becomes allocated we rely on the fact
>> that the filesystem will first zero the blocks.  That zeroing process
>> will clear the media error.
>
> The incorrect /assumption/ here is that all allocations will do
> block zeroing first. That's simply wrong. We do that for *user data*
> in XFS and ext4, but we do not do it for metadata as they are not
> accessed by DAX and, being transactionally protected, don't need
> zeroing to prevent stale data exposure.

Ah, great point.  See, I knew if we kept talking, something productive
would come out of this discussion, but I think we're still ok, see
below.

> Hence we have a problem here - the first write to such blocks may
> be metadata writeback of some type and so the filesystem will see
> EIO errors in metadata writes and they'll freak out.

As long as the filesystem writes before it reads metadata we're mostly
ok, because a block error is cleared on write, we don't return -EIO
for them, however...

> What
> now - does this really mean that we'll have to add special IO
> falback code for all internal IO paths to be able to clear pmem bad
> block errors?

...we still have the case where a badblock can develop in metadata
after writing it, we can't do anything about it unless some layer
somewhere implements redundancy.

> Oh, and just a thought: lots of people are pushing for selectable
> FALLOC_FL_NO_HIDE_STALE behaviour which will skip zeroing
> of data blocks on allocation. If this happens, it we also skip
> the zeroing on allocation, so again there is no mechanism to clear
> bad block status in this case.....

Outside of rewriting the file, yes I was aware of this and it breaks
the "delete to clear media errors" recovery model.  My only answer
right now is that we would need to document FALLOC_FL_NO_HIDE_STALE
with notifications about the fact that we lose the side-effect of
clearing latent media errors when block zeroing / trimming on
reallocation is disabled.

> [...]
>
>> > However, if I'm struggling to understand how I'm supposed to
>> > connecct up the parts inside a filesytem, then expecting application
>> > developers to be able to connect the dots in a sane manner is
>> > bordering on fantasy....
>>
>> Hopefully it is becoming clearer that we are not proposing anything
>> radically different than what is present for error recovery today
>> modulo thinking about the mechanisms to trap and recover a DAX read of
>> a bad media area via a DM/MD implementation.
>
> There is a radical difference - there is a pervasive /assumption/ in
> what is being proposed that filesystems are incapable of storing
> redundant information that can be used for error recovery.

That's not an assumption, it is a fact that DAX enabled filesystems
don't account for data redundancy today outside of "let the storage
layer do it".  I'm all for making this situation better than it is.

> The
> current IO stack makes no such assumptions, even if it doesn't
> provide any infrastructure for such functionality.

Great, lets start to fill in that hole with some patches to return
SIGBUS on a DAX fault hitting a bad block, and bypassing dax_do_io()
for O_DIRECT writes so userspace I/O can clear errors instead of
receiving -EIO for a write.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web