Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1260263 > unrolled thread

Re: [RFC 00/11] DAX fsynx/msync support

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [RFC 00/11] DAX fsynx/msync support Dave Chinner <david@fromorbit.com> - 2015-11-02 00:40 +0100
    Re: [RFC 00/11] DAX fsynx/msync support Jeff Moyer <jmoyer@redhat.com> - 2015-11-02 15:30 +0100
      Re: [RFC 00/11] DAX fsynx/msync support Dave Chinner <david@fromorbit.com> - 2015-11-02 21:20 +0100
        Re: [RFC 00/11] DAX fsynx/msync support Jeff Moyer <jmoyer@redhat.com> - 2015-11-02 22:10 +0100
          Re: [RFC 00/11] DAX fsynx/msync support Jeff Moyer <jmoyer@redhat.com> - 2015-11-04 19:40 +0100
          Re: [RFC 00/11] DAX fsynx/msync support Dave Chinner <david@fromorbit.com> - 2015-11-05 09:40 +0100
            Re: [RFC 00/11] DAX fsynx/msync support Jeff Moyer <jmoyer@redhat.com> - 2015-11-05 20:50 +0100
            Re: [RFC 00/11] DAX fsynx/msync support Jens Axboe <axboe@kernel.dk> - 2015-11-05 22:00 +0100

#1260263 — Re: [RFC 00/11] DAX fsynx/msync support

On Fri, Oct 30, 2015 at 12:39:38PM -0600, Ross Zwisler wrote:
> On Fri, Oct 30, 2015 at 02:55:33PM +1100, Dave Chinner wrote:
> > On Thu, Oct 29, 2015 at 02:12:04PM -0600, Ross Zwisler wrote:
> > > This patch series adds support for fsync/msync to DAX.
> > > 
> > > Patches 1 through 8 add various utilities that the DAX code will eventually
> > > need, and the DAX code itself is added by patch 9.  Patches 10 and 11 are
> > > filesystem changes that are needed after the DAX code is added, but these
> > > patches may change slightly as the filesystem fault handling for DAX is
> > > being modified ([1] and [2]).
> > > 
> > > I've marked this series as RFC because I'm still testing, but I wanted to
> > > get this out there so people would see the direction I was going and
> > > hopefully comment on any big red flags sooner rather than later.
> > > 
> > > I realize that we are getting pretty dang close to the v4.4 merge window,
> > > but I think that if we can get this reviewed and working it's a much better
> > > solution than the "big hammer" approach that blindly flushes entire PMEM
> > > namespaces [3].
> > 
> > We need the "big hammer" regardless of fsync. If REQ_FLUSH and
> > REQ_FUA don't do the right thing when it comes to ordering journal
> > writes against other IO operations, then the filesystems are not
> > crash safe. i.e. we need REQ_FLUSH/REQ_FUA to commit all outstanding
> > changes back to stable storage, just like they do for existing
> > storage....
> 
> I think that what I've got here (when it's fully working) will protect all the
> cases that we need.
> 
> AFAIK there are three ways that data can be written to a PMEM namespace:
> 
> 1) Through the PMEM driver via either pmem_make_request(), pmem_rw_page() or
> pmem_rw_bytes().  All of these paths sync the newly written data durably to
> media before the I/O completes so they shouldn't have any reliance on
> REQ_FUA/REQ_FLUSH.

I suspect that not all future pmem devices will use this
driver/interface/semantics.

Further, REQ_FLUSH/REQ_FUA are more than just "put the data on stable
storage" commands. They are also IO barriers that affect scheduling
of IOs in progress and in the request queues.  A REQ_FLUSH/REQ_FUA
IO cannot be dispatched before all prior IO has been dispatched and
drained from the request queue, and IO submitted after a queued
REQ_FLUSH/REQ_FUA cannot be scheduled ahead of the queued
REQ_FLUSH/REQ_FUA operation.

IOWs, REQ_FUA/REQ_FLUSH not only guarantee data is on stable
storage, they also guarantee the order of IO dispatch and
completion when concurrent IO is in progress.


> 2) Through the DAX I/O path, dax_io().  As with PMEM we flush the newly
> written data durably to media before the I/O operation completes, so this path
> shouldn't have any reliance on REQ_FUA/REQ_FLUSH.

That's fine, but that's not the problem we need solved ;)

> 3) Through mmaps set up by DAX.  This is the path we are trying to protect
> with the dirty page tracking and flushing in this patch set, and I think that
> this is the only path that has reliance on REQ_FLUSH.

Quite possibly this is the case for the current intel pmem driver,
but I don't look at the functionality from that perspective.

Dirty page tracking is needed to enable "data writeback", whether it
be CPU cachelines via pcommit() or dirty pages via submit_bio(). How
the pages get dirty is irrelevant - the fact is they are dirty and
we need to do /something/ to ensure they are correctly written back
to the storage layer.

REQ_FLUSH is needed to guarantee all data that has been written back
to the storage layer is persistent in that layer.  How a /driver/
manages that is up to the driver - the actual implementation is
irrelevant to the higher layers. i.e. what we are concerned about at
the filesystem level is that:

	a) "data writeback" is started correctly;
	b) the "data writeback" is completed; and
	c) volatile caches are completely flushed before we write
	   the metadata changes that reference that data to the
	   journal via FUA

e.g. we could have pmem, but we are using buffered IO (i.e. non-DAX)
and a hardware driver that doesn't flush CPU cachelines in the
physical IO path. This requires that driver to flush CPU cachelines
and place memory barriers in REQ_FLUSH operations, as well as after
writing the data in REQ_FUA operations.  Yes, this is different to
the way the intel pmem drivers work (i.e.  as noted in 1) above),
but it is /not wrong/ as long as REQ_FLUSH/REQ_FUA also flush dirty
cpu cachelines.

IOWs, the high level code we write that implements fsync
for DAX needs to be generic enough so that when something slightly
different comes along we don't have to throw everything away and
start again. I think your code will end up being generic enough to
handle this, but let's make sure we don't implement something that
can only work with pmem hardware/drivers that do all IO as fully
synchronous to the stable domain...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1260679

Dave Chinner <david@fromorbit.com> writes:

> Further, REQ_FLUSH/REQ_FUA are more than just "put the data on stable
> storage" commands. They are also IO barriers that affect scheduling
> of IOs in progress and in the request queues.  A REQ_FLUSH/REQ_FUA
> IO cannot be dispatched before all prior IO has been dispatched and
> drained from the request queue, and IO submitted after a queued
> REQ_FLUSH/REQ_FUA cannot be scheduled ahead of the queued
> REQ_FLUSH/REQ_FUA operation.
>
> IOWs, REQ_FUA/REQ_FLUSH not only guarantee data is on stable
> storage, they also guarantee the order of IO dispatch and
> completion when concurrent IO is in progress.

This hasn't been the case for several years, now.  It used to work that
way, and that was deemed a big performance problem.  Since file systems
already issued and waited for all I/O before sending down a barrier, we
decided to get rid of the I/O ordering pieces of barriers (and stop
calling them barriers).

See commit 28e7d184521 (block: drop barrier ordering by queue draining).

Cheers,
Jeff
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1260931

On Mon, Nov 02, 2015 at 09:22:15AM -0500, Jeff Moyer wrote:
> Dave Chinner <david@fromorbit.com> writes:
> 
> > Further, REQ_FLUSH/REQ_FUA are more than just "put the data on stable
> > storage" commands. They are also IO barriers that affect scheduling
> > of IOs in progress and in the request queues.  A REQ_FLUSH/REQ_FUA
> > IO cannot be dispatched before all prior IO has been dispatched and
> > drained from the request queue, and IO submitted after a queued
> > REQ_FLUSH/REQ_FUA cannot be scheduled ahead of the queued
> > REQ_FLUSH/REQ_FUA operation.
> >
> > IOWs, REQ_FUA/REQ_FLUSH not only guarantee data is on stable
> > storage, they also guarantee the order of IO dispatch and
> > completion when concurrent IO is in progress.
> 
> This hasn't been the case for several years, now.  It used to work that
> way, and that was deemed a big performance problem.  Since file systems
> already issued and waited for all I/O before sending down a barrier, we
> decided to get rid of the I/O ordering pieces of barriers (and stop
> calling them barriers).
> 
> See commit 28e7d184521 (block: drop barrier ordering by queue draining).

Yes, I realise that, even if I wasn't very clear about how I wrote
it. ;)

Correct me if I'm wrong: AFAIA, dispatch ordering (i.e. the "IO
barrier") is still enforced by the scheduler via REQ_FUA|REQ_FLUSH
-> ELEVATOR_INSERT_FLUSH -> REQ_SOFTBARRIER and subsequent IO
scheduler calls to elv_dispatch_sort() that don't pass
REQ_SOFTBARRIER in the queue.

IOWs, if we queue a bunch of REQ_WRITE IOs followed by a
REQ_WRITE|REQ_FLUSH IO, all of the prior REQ_WRITE IOs will be
dispatched before the REQ_WRITE|REQ_FLUSH IO and hence be captured
by the cache flush.

Hence once the filesystem has waited on the REQ_WRITE|REQ_FLUSH IO
to complete, we know that all the earlier REQ_WRITE IOs are on
stable storage, too. Hence there's no need for the elevator to drain
the queue to guarantee completion ordering - the dispatch ordering
and flush/fua write semantics guarantee that when the flush/fua
completes, all the IOs dispatch prior to that flush/fua write are
also on stable storage...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1260973

Dave Chinner <david@fromorbit.com> writes:

> On Mon, Nov 02, 2015 at 09:22:15AM -0500, Jeff Moyer wrote:
>> Dave Chinner <david@fromorbit.com> writes:
>> 
>> > Further, REQ_FLUSH/REQ_FUA are more than just "put the data on stable
>> > storage" commands. They are also IO barriers that affect scheduling
>> > of IOs in progress and in the request queues.  A REQ_FLUSH/REQ_FUA
>> > IO cannot be dispatched before all prior IO has been dispatched and
>> > drained from the request queue, and IO submitted after a queued
>> > REQ_FLUSH/REQ_FUA cannot be scheduled ahead of the queued
>> > REQ_FLUSH/REQ_FUA operation.
>> >
>> > IOWs, REQ_FUA/REQ_FLUSH not only guarantee data is on stable
>> > storage, they also guarantee the order of IO dispatch and
>> > completion when concurrent IO is in progress.
>> 
>> This hasn't been the case for several years, now.  It used to work that
>> way, and that was deemed a big performance problem.  Since file systems
>> already issued and waited for all I/O before sending down a barrier, we
>> decided to get rid of the I/O ordering pieces of barriers (and stop
>> calling them barriers).
>> 
>> See commit 28e7d184521 (block: drop barrier ordering by queue draining).
>
> Yes, I realise that, even if I wasn't very clear about how I wrote
> it. ;)
>
> Correct me if I'm wrong: AFAIA, dispatch ordering (i.e. the "IO
> barrier") is still enforced by the scheduler via REQ_FUA|REQ_FLUSH
> -> ELEVATOR_INSERT_FLUSH -> REQ_SOFTBARRIER and subsequent IO
> scheduler calls to elv_dispatch_sort() that don't pass
> REQ_SOFTBARRIER in the queue.

This part is right.

> IOWs, if we queue a bunch of REQ_WRITE IOs followed by a
> REQ_WRITE|REQ_FLUSH IO, all of the prior REQ_WRITE IOs will be
> dispatched before the REQ_WRITE|REQ_FLUSH IO and hence be captured
> by the cache flush.

But this part is not.  It is up to the I/O scheduler to decide when to
dispatch requests.  It can hold on to them for a variety of reasons.
Flush requests, however, do not go through the I/O scheduler.  At the
very moment that the flush request is inserted, it goes directly to the
dispatch queue (assuming no other flush is in progress).  The prior
requests may still be waiting in the I/O scheduler's internal lists.

So, any newly dispatched I/Os will certainly not get past the REQ_FLUSH.
However, the REQ_FLUSH is very likely to jump ahead of prior I/Os in the
queue.

> Hence once the filesystem has waited on the REQ_WRITE|REQ_FLUSH IO
> to complete, we know that all the earlier REQ_WRITE IOs are on
> stable storage, too. Hence there's no need for the elevator to drain
> the queue to guarantee completion ordering - the dispatch ordering
> and flush/fua write semantics guarantee that when the flush/fua
> completes, all the IOs dispatch prior to that flush/fua write are
> also on stable storage...

Des xfs rely on this model for correctness?  If so, I'd say we've got a
problem.

Cheers,
Jeff
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1262497

Jeff Moyer <jmoyer@redhat.com> writes:

>> Hence once the filesystem has waited on the REQ_WRITE|REQ_FLUSH IO
>> to complete, we know that all the earlier REQ_WRITE IOs are on
>> stable storage, too. Hence there's no need for the elevator to drain
>> the queue to guarantee completion ordering - the dispatch ordering
>> and flush/fua write semantics guarantee that when the flush/fua
>> completes, all the IOs dispatch prior to that flush/fua write are
>> also on stable storage...
>
> Des xfs rely on this model for correctness?  If so, I'd say we've got a
> problem.

Dave?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1263005

[ sorry for slow response, been without an internet connection for
~36 hours ]

On Mon, Nov 02, 2015 at 04:02:48PM -0500, Jeff Moyer wrote:
> Dave Chinner <david@fromorbit.com> writes:
> 
> > On Mon, Nov 02, 2015 at 09:22:15AM -0500, Jeff Moyer wrote:
> >> Dave Chinner <david@fromorbit.com> writes:
> >> 
> >> > Further, REQ_FLUSH/REQ_FUA are more than just "put the data on stable
> >> > storage" commands. They are also IO barriers that affect scheduling
> >> > of IOs in progress and in the request queues.  A REQ_FLUSH/REQ_FUA
> >> > IO cannot be dispatched before all prior IO has been dispatched and
> >> > drained from the request queue, and IO submitted after a queued
> >> > REQ_FLUSH/REQ_FUA cannot be scheduled ahead of the queued
> >> > REQ_FLUSH/REQ_FUA operation.
> >> >
> >> > IOWs, REQ_FUA/REQ_FLUSH not only guarantee data is on stable
> >> > storage, they also guarantee the order of IO dispatch and
> >> > completion when concurrent IO is in progress.
> >> 
> >> This hasn't been the case for several years, now.  It used to work that
> >> way, and that was deemed a big performance problem.  Since file systems
> >> already issued and waited for all I/O before sending down a barrier, we
> >> decided to get rid of the I/O ordering pieces of barriers (and stop
> >> calling them barriers).
> >> 
> >> See commit 28e7d184521 (block: drop barrier ordering by queue draining).
> >
> > Yes, I realise that, even if I wasn't very clear about how I wrote
> > it. ;)
> >
> > Correct me if I'm wrong: AFAIA, dispatch ordering (i.e. the "IO
> > barrier") is still enforced by the scheduler via REQ_FUA|REQ_FLUSH
> > -> ELEVATOR_INSERT_FLUSH -> REQ_SOFTBARRIER and subsequent IO
> > scheduler calls to elv_dispatch_sort() that don't pass
> > REQ_SOFTBARRIER in the queue.
> 
> This part is right.
> 
> > IOWs, if we queue a bunch of REQ_WRITE IOs followed by a
> > REQ_WRITE|REQ_FLUSH IO, all of the prior REQ_WRITE IOs will be
> > dispatched before the REQ_WRITE|REQ_FLUSH IO and hence be captured
> > by the cache flush.
> 
> But this part is not.  It is up to the I/O scheduler to decide when to
> dispatch requests.  It can hold on to them for a variety of reasons.
> Flush requests, however, do not go through the I/O scheduler.  At the

That's pure REQ_FLUSH bios, right? Aren't data IOs with
REQ_FLUSH|REQ_FUA sorted like any other IO?

> very moment that the flush request is inserted, it goes directly to the
> dispatch queue (assuming no other flush is in progress).  The prior
> requests may still be waiting in the I/O scheduler's internal lists.
> 
> So, any newly dispatched I/Os will certainly not get past the REQ_FLUSH.
> However, the REQ_FLUSH is very likely to jump ahead of prior I/Os in the
> queue.

Uh, ok, that's different, and most definitely not the "IO barrier" I
was under the impression REQ_FLUSH|REQ_FUA gave us.

> > Hence once the filesystem has waited on the REQ_WRITE|REQ_FLUSH IO
> > to complete, we know that all the earlier REQ_WRITE IOs are on
> > stable storage, too. Hence there's no need for the elevator to drain
> > the queue to guarantee completion ordering - the dispatch ordering
> > and flush/fua write semantics guarantee that when the flush/fua
> > completes, all the IOs dispatch prior to that flush/fua write are
> > also on stable storage...
> 
> Des xfs rely on this model for correctness?  If so, I'd say we've got a
> problem

No, it doesn't. The XFS integrity model doesn't trust the IO layers
to tell the truth about IO ordering and completion or for it's
developers to fully understand how IO layer ordering works. :P

i.e. we wait for full completions of all dependent IO before issuing
flushes or log writes that use REQ_FLUSH|REQ_FUA semantics to ensure
the dependent IOs are fully caught by the cache flushes...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1263512

Dave Chinner <david@fromorbit.com> writes:

>> But this part is not.  It is up to the I/O scheduler to decide when to
>> dispatch requests.  It can hold on to them for a variety of reasons.
>> Flush requests, however, do not go through the I/O scheduler.  At the
>
> That's pure REQ_FLUSH bios, right? Aren't data IOs with
> REQ_FLUSH|REQ_FUA sorted like any other IO?

No, they also go through the flush machinery, and so short-circuit the
I/O scheduler.

>> Des xfs rely on this model for correctness?  If so, I'd say we've got a
>> problem
>
> No, it doesn't. The XFS integrity model doesn't trust the IO layers
> to tell the truth about IO ordering and completion or for it's
> developers to fully understand how IO layer ordering works. :P
>
> i.e. we wait for full completions of all dependent IO before issuing
> flushes or log writes that use REQ_FLUSH|REQ_FUA semantics to ensure
> the dependent IOs are fully caught by the cache flushes...

OK, phew!  ;-)

-Jeff
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1263552

On 11/05/2015 01:33 AM, Dave Chinner wrote:
>> Des xfs rely on this model for correctness?  If so, I'd say we've got a
>> problem
>
> No, it doesn't. The XFS integrity model doesn't trust the IO layers
> to tell the truth about IO ordering and completion or for it's
> developers to fully understand how IO layer ordering works. :P

That's good, because the storage developers simplified the model so that 
fs developers would be able to get and use it.

> i.e. we wait for full completions of all dependent IO before issuing
> flushes or log writes that use REQ_FLUSH|REQ_FUA semantics to ensure
> the dependent IOs are fully caught by the cache flushes...

... which is what you are supposed to do, that's how it works.

-- 
Jens Axboe

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web