Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1258986 > unrolled thread

[RFC 00/11] DAX fsynx/msync support

Back to article view | Back to linux.kernel

Contents

  [RFC 00/11] DAX fsynx/msync support Ross Zwisler <ross.zwisler@linux.intel.com> - 2015-10-29 21:30 +0100
    [RFC 02/11] mm: add pmd_mkclean() Ross Zwisler <ross.zwisler@linux.intel.com> - 2015-10-29 21:30 +0100
    Re: [RFC 00/11] DAX fsynx/msync support Ross Zwisler <ross.zwisler@linux.intel.com> - 2015-10-30 00:00 +0100
    Re: [RFC 00/11] DAX fsynx/msync support Dave Chinner <david@fromorbit.com> - 2015-10-30 05:00 +0100
      Re: [RFC 00/11] DAX fsynx/msync support Ross Zwisler <ross.zwisler@linux.intel.com> - 2015-10-30 19:40 +0100
    Re: [RFC 00/11] DAX fsynx/msync support Dan Williams <dan.j.williams@intel.com> - 2015-10-30 19:40 +0100
      Re: [RFC 00/11] DAX fsynx/msync support Ross Zwisler <ross.zwisler@linux.intel.com> - 2015-10-30 20:50 +0100
        Re: [RFC 00/11] DAX fsynx/msync support Dan Williams <dan.j.williams@intel.com> - 2015-10-30 21:00 +0100

#1258986 — [RFC 00/11] DAX fsynx/msync support

This patch series adds support for fsync/msync to DAX.

Patches 1 through 8 add various utilities that the DAX code will eventually
need, and the DAX code itself is added by patch 9.  Patches 10 and 11 are
filesystem changes that are needed after the DAX code is added, but these
patches may change slightly as the filesystem fault handling for DAX is
being modified ([1] and [2]).

I've marked this series as RFC because I'm still testing, but I wanted to
get this out there so people would see the direction I was going and
hopefully comment on any big red flags sooner rather than later.

I realize that we are getting pretty dang close to the v4.4 merge window,
but I think that if we can get this reviewed and working it's a much better
solution than the "big hammer" approach that blindly flushes entire PMEM
namespaces [3].

[1] http://oss.sgi.com/archives/xfs/2015-10/msg00523.html
[2] http://marc.info/?l=linux-ext4&m=144550211312472&w=2
[3] https://lists.01.org/pipermail/linux-nvdimm/2015-October/002614.html

Ross Zwisler (11):
  pmem: add wb_cache_pmem() to the PMEM API
  mm: add pmd_mkclean()
  pmem: enable REQ_FLUSH handling
  dax: support dirty DAX entries in radix tree
  mm: add follow_pte_pmd()
  mm: add pgoff_mkclean()
  mm: add find_get_entries_tag()
  fs: add get_block() to struct inode_operations
  dax: add support for fsync/sync
  xfs, ext2: call dax_pfn_mkwrite() on write fault
  ext4: add ext4_dax_pfn_mkwrite()

 arch/x86/include/asm/pgtable.h |   5 ++
 arch/x86/include/asm/pmem.h    |  11 +--
 drivers/nvdimm/pmem.c          |   3 +-
 fs/dax.c                       | 161 +++++++++++++++++++++++++++++++++++++++--
 fs/ext2/file.c                 |   5 +-
 fs/ext4/file.c                 |  23 +++++-
 fs/inode.c                     |   1 +
 fs/xfs/xfs_file.c              |   9 ++-
 fs/xfs/xfs_iops.c              |   1 +
 include/linux/dax.h            |   6 ++
 include/linux/fs.h             |   5 +-
 include/linux/mm.h             |   2 +
 include/linux/pagemap.h        |   3 +
 include/linux/pmem.h           |  22 +++++-
 include/linux/radix-tree.h     |   3 +
 include/linux/rmap.h           |   5 ++
 mm/filemap.c                   |  73 ++++++++++++++++++-
 mm/huge_memory.c               |  14 ++--
 mm/memory.c                    |  41 +++++++++--
 mm/page-writeback.c            |   9 +++
 mm/rmap.c                      |  53 ++++++++++++++
 mm/truncate.c                  |   5 +-
 22 files changed, 418 insertions(+), 42 deletions(-)

-- 
2.1.0

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1258990 — [RFC 02/11] mm: add pmd_mkclean()

Currently PMD pages can be dirtied via pmd_mkdirty(), but cannot be
cleaned.  For DAX mmap dirty page tracking we need to be able to clean PMD
pages when we flush them to media so that we get a new write fault the next
time the are written to.

Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com>
---
 arch/x86/include/asm/pgtable.h | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h
index 867da5b..c548e4c 100644
--- a/arch/x86/include/asm/pgtable.h
+++ b/arch/x86/include/asm/pgtable.h
@@ -277,6 +277,11 @@ static inline pmd_t pmd_mkdirty(pmd_t pmd)
 	return pmd_set_flags(pmd, _PAGE_DIRTY | _PAGE_SOFT_DIRTY);
 }
 
+static inline pmd_t pmd_mkclean(pmd_t pmd)
+{
+	return pmd_clear_flags(pmd, _PAGE_DIRTY | _PAGE_SOFT_DIRTY);
+}
+
 static inline pmd_t pmd_mkhuge(pmd_t pmd)
 {
 	return pmd_set_flags(pmd, _PAGE_PSE);
-- 
2.1.0

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1259075

On Thu, Oct 29, 2015 at 02:12:04PM -0600, Ross Zwisler wrote:
> This patch series adds support for fsync/msync to DAX.
> 
> Patches 1 through 8 add various utilities that the DAX code will eventually
> need, and the DAX code itself is added by patch 9.  Patches 10 and 11 are
> filesystem changes that are needed after the DAX code is added, but these
> patches may change slightly as the filesystem fault handling for DAX is
> being modified ([1] and [2]).
> 
> I've marked this series as RFC because I'm still testing, but I wanted to
> get this out there so people would see the direction I was going and
> hopefully comment on any big red flags sooner rather than later.
> 
> I realize that we are getting pretty dang close to the v4.4 merge window,
> but I think that if we can get this reviewed and working it's a much better
> solution than the "big hammer" approach that blindly flushes entire PMEM
> namespaces [3].
> 
> [1] http://oss.sgi.com/archives/xfs/2015-10/msg00523.html
> [2] http://marc.info/?l=linux-ext4&m=144550211312472&w=2
> [3] https://lists.01.org/pipermail/linux-nvdimm/2015-October/002614.html

Hmm...I think I may need to isolate the fsync/msync flushing against races
with truncate since we are calling into the filesystem directly with
get_block().  Dave (Chinner), does this sound right?

Also, one thing I forgot to mention is that these patches are built upon the
first version of Dave Chinner's XFS patches and my ext2 patches that deal with
the truncate races with DAX.  A snapshot of my development tree with these
patches applied can be found here:

https://github.com/01org/prd/tree/fsync_rfc
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1259193

On Thu, Oct 29, 2015 at 02:12:04PM -0600, Ross Zwisler wrote:
> This patch series adds support for fsync/msync to DAX.
> 
> Patches 1 through 8 add various utilities that the DAX code will eventually
> need, and the DAX code itself is added by patch 9.  Patches 10 and 11 are
> filesystem changes that are needed after the DAX code is added, but these
> patches may change slightly as the filesystem fault handling for DAX is
> being modified ([1] and [2]).
> 
> I've marked this series as RFC because I'm still testing, but I wanted to
> get this out there so people would see the direction I was going and
> hopefully comment on any big red flags sooner rather than later.
> 
> I realize that we are getting pretty dang close to the v4.4 merge window,
> but I think that if we can get this reviewed and working it's a much better
> solution than the "big hammer" approach that blindly flushes entire PMEM
> namespaces [3].

We need the "big hammer" regardless of fsync. If REQ_FLUSH and
REQ_FUA don't do the right thing when it comes to ordering journal
writes against other IO operations, then the filesystems are not
crash safe. i.e. we need REQ_FLUSH/REQ_FUA to commit all outstanding
changes back to stable storage, just like they do for existing
storage....

> [1] http://oss.sgi.com/archives/xfs/2015-10/msg00523.html
> [2] http://marc.info/?l=linux-ext4&m=144550211312472&w=2
> [3] https://lists.01.org/pipermail/linux-nvdimm/2015-October/002614.html
> 
> Ross Zwisler (11):
>   pmem: add wb_cache_pmem() to the PMEM API
>   mm: add pmd_mkclean()
>   pmem: enable REQ_FLUSH handling
>   dax: support dirty DAX entries in radix tree
>   mm: add follow_pte_pmd()
>   mm: add pgoff_mkclean()
>   mm: add find_get_entries_tag()
>   fs: add get_block() to struct inode_operations

I don't think this is the right thing to do - it propagates the use
of bufferheads as a mapping structure into places where we do not
want bufferheads. We've recently added a similar block mapping
interface to the export operations structure for PNFS and that uses
a "struct iomap" which is far more suited to being an inode
operation this.

We have plans to move this to the inode operations for various
reasons. e.g: multipage write, adding interfaces that support proper
mapping of holes, etc:

https://www.redhat.com/archives/cluster-devel/2014-October/msg00167.html

So after many years of saying no to moving getblocks to the inode
operations it seems like the wrong thing to do now considering I
want to convert all the DAX code to use iomaps while only 2/3
filesystems are supported...

>   dax: add support for fsync/sync

Why put the dax_flush_mapping() in do_writepages()? Why not call it
directly from the filesystem ->fsync() implementations where a
getblocks callback could also be provided?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1259700

On Fri, Oct 30, 2015 at 02:55:33PM +1100, Dave Chinner wrote:
> On Thu, Oct 29, 2015 at 02:12:04PM -0600, Ross Zwisler wrote:
> > This patch series adds support for fsync/msync to DAX.
> > 
> > Patches 1 through 8 add various utilities that the DAX code will eventually
> > need, and the DAX code itself is added by patch 9.  Patches 10 and 11 are
> > filesystem changes that are needed after the DAX code is added, but these
> > patches may change slightly as the filesystem fault handling for DAX is
> > being modified ([1] and [2]).
> > 
> > I've marked this series as RFC because I'm still testing, but I wanted to
> > get this out there so people would see the direction I was going and
> > hopefully comment on any big red flags sooner rather than later.
> > 
> > I realize that we are getting pretty dang close to the v4.4 merge window,
> > but I think that if we can get this reviewed and working it's a much better
> > solution than the "big hammer" approach that blindly flushes entire PMEM
> > namespaces [3].
> 
> We need the "big hammer" regardless of fsync. If REQ_FLUSH and
> REQ_FUA don't do the right thing when it comes to ordering journal
> writes against other IO operations, then the filesystems are not
> crash safe. i.e. we need REQ_FLUSH/REQ_FUA to commit all outstanding
> changes back to stable storage, just like they do for existing
> storage....

I think that what I've got here (when it's fully working) will protect all the
cases that we need.

AFAIK there are three ways that data can be written to a PMEM namespace:

1) Through the PMEM driver via either pmem_make_request(), pmem_rw_page() or
pmem_rw_bytes().  All of these paths sync the newly written data durably to
media before the I/O completes so they shouldn't have any reliance on
REQ_FUA/REQ_FLUSH.

2) Through the DAX I/O path, dax_io().  As with PMEM we flush the newly
written data durably to media before the I/O operation completes, so this path
shouldn't have any reliance on REQ_FUA/REQ_FLUSH.

3) Through mmaps set up by DAX.  This is the path we are trying to protect
with the dirty page tracking and flushing in this patch set, and I think that
this is the only path that has reliance on REQ_FLUSH.

The goal of this set is to have the cache writeback all happen as part of the
fsync/msync handling, and then have the REQ_FLUSH just provide the trailing
wmb_pmem().

My guess is that XFS metadata writes happen via path 1), down through the PMEM
driver.  Am I missing anything, or should we be good to go?

> > [1] http://oss.sgi.com/archives/xfs/2015-10/msg00523.html
> > [2] http://marc.info/?l=linux-ext4&m=144550211312472&w=2
> > [3] https://lists.01.org/pipermail/linux-nvdimm/2015-October/002614.html
> > 
> > Ross Zwisler (11):
> >   pmem: add wb_cache_pmem() to the PMEM API
> >   mm: add pmd_mkclean()
> >   pmem: enable REQ_FLUSH handling
> >   dax: support dirty DAX entries in radix tree
> >   mm: add follow_pte_pmd()
> >   mm: add pgoff_mkclean()
> >   mm: add find_get_entries_tag()
> >   fs: add get_block() to struct inode_operations
> 
> I don't think this is the right thing to do - it propagates the use
> of bufferheads as a mapping structure into places where we do not
> want bufferheads. We've recently added a similar block mapping
> interface to the export operations structure for PNFS and that uses
> a "struct iomap" which is far more suited to being an inode
> operation this.
> 
> We have plans to move this to the inode operations for various
> reasons. e.g: multipage write, adding interfaces that support proper
> mapping of holes, etc:
> 
> https://www.redhat.com/archives/cluster-devel/2014-October/msg00167.html
> 
> So after many years of saying no to moving getblocks to the inode
> operations it seems like the wrong thing to do now considering I
> want to convert all the DAX code to use iomaps while only 2/3
> filesystems are supported...

Okay, I'll take a look at this interface.  I also think that we may need to
flow through the filesystem before going into the DAX code so that we can
serialize our flushing with respect to extent manipulation, as we had to do
with our DAX fault paths.  

> >   dax: add support for fsync/sync
> 
> Why put the dax_flush_mapping() in do_writepages()? Why not call it
> directly from the filesystem ->fsync() implementations where a
> getblocks callback could also be provided?

Because that's where you put it in your example. :)

https://lists.01.org/pipermail/linux-nvdimm/2015-March/000341.html

Moving it into the filesystem where we know about get_block() is probably the
right thing to do - I'll check it out.  Thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1259708

On Thu, Oct 29, 2015 at 1:12 PM, Ross Zwisler
<ross.zwisler@linux.intel.com> wrote:
> This patch series adds support for fsync/msync to DAX.
>
> Patches 1 through 8 add various utilities that the DAX code will eventually
> need, and the DAX code itself is added by patch 9.  Patches 10 and 11 are
> filesystem changes that are needed after the DAX code is added, but these
> patches may change slightly as the filesystem fault handling for DAX is
> being modified ([1] and [2]).
>
> I've marked this series as RFC because I'm still testing, but I wanted to
> get this out there so people would see the direction I was going and
> hopefully comment on any big red flags sooner rather than later.
>
> I realize that we are getting pretty dang close to the v4.4 merge window,
> but I think that if we can get this reviewed and working it's a much better
> solution than the "big hammer" approach that blindly flushes entire PMEM
> namespaces [3].
>
> [1] http://oss.sgi.com/archives/xfs/2015-10/msg00523.html
> [2] http://marc.info/?l=linux-ext4&m=144550211312472&w=2
> [3] https://lists.01.org/pipermail/linux-nvdimm/2015-October/002614.html
>
> Ross Zwisler (11):
>   pmem: add wb_cache_pmem() to the PMEM API
>   mm: add pmd_mkclean()
>   pmem: enable REQ_FLUSH handling
>   dax: support dirty DAX entries in radix tree
>   mm: add follow_pte_pmd()
>   mm: add pgoff_mkclean()
>   mm: add find_get_entries_tag()
>   fs: add get_block() to struct inode_operations
>   dax: add support for fsync/sync
>   xfs, ext2: call dax_pfn_mkwrite() on write fault
>   ext4: add ext4_dax_pfn_mkwrite()

This is great to have when the flush-the-world solution ends up
killing performance.  However, there are a couple mitigating options
for workloads that dirty small amounts and flush often that we need to
collect data on:

1/ Using cache management and pcommit from userspace to skip calls to
msync / fsync.  Although, this does not eliminate all calls to
blkdev_issue_flush as the fs may invoke it for other reasons.  I
suspect turning on REQ_FUA support eliminates a number of those
invocations, and pmem already satisfies REQ_FUA semantics by default.

2/ Turn off DAX and use the page cache.  As Dave mentions [1] we
should enable this control on a per-inode basis.  I'm folding in this
capability as a blkdev_ioctl for the next version of the raw block DAX
support patch.

It's entirely possible these mitigations won't eliminate the need for
a mechanism like this, but I think we have a bit more work to do to
find out how bad this is in practice as well as the crossover point
where walking the radix becomes prohibitive.

We also have the option of tracking open DAX extents in the driver.
Even at coarse granularities I'd be surprised if we can't mitigate
most of the overhead.

[1]: https://lists.01.org/pipermail/linux-nvdimm/2015-October/002598.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1259750

On Fri, Oct 30, 2015 at 11:34:07AM -0700, Dan Williams wrote:
> On Thu, Oct 29, 2015 at 1:12 PM, Ross Zwisler
> <ross.zwisler@linux.intel.com> wrote:
> > This patch series adds support for fsync/msync to DAX.
> >
> > Patches 1 through 8 add various utilities that the DAX code will eventually
> > need, and the DAX code itself is added by patch 9.  Patches 10 and 11 are
> > filesystem changes that are needed after the DAX code is added, but these
> > patches may change slightly as the filesystem fault handling for DAX is
> > being modified ([1] and [2]).
> >
> > I've marked this series as RFC because I'm still testing, but I wanted to
> > get this out there so people would see the direction I was going and
> > hopefully comment on any big red flags sooner rather than later.
> >
> > I realize that we are getting pretty dang close to the v4.4 merge window,
> > but I think that if we can get this reviewed and working it's a much better
> > solution than the "big hammer" approach that blindly flushes entire PMEM
> > namespaces [3].
> >
> > [1] http://oss.sgi.com/archives/xfs/2015-10/msg00523.html
> > [2] http://marc.info/?l=linux-ext4&m=144550211312472&w=2
> > [3] https://lists.01.org/pipermail/linux-nvdimm/2015-October/002614.html
> >
> > Ross Zwisler (11):
> >   pmem: add wb_cache_pmem() to the PMEM API
> >   mm: add pmd_mkclean()
> >   pmem: enable REQ_FLUSH handling
> >   dax: support dirty DAX entries in radix tree
> >   mm: add follow_pte_pmd()
> >   mm: add pgoff_mkclean()
> >   mm: add find_get_entries_tag()
> >   fs: add get_block() to struct inode_operations
> >   dax: add support for fsync/sync
> >   xfs, ext2: call dax_pfn_mkwrite() on write fault
> >   ext4: add ext4_dax_pfn_mkwrite()
> 
> This is great to have when the flush-the-world solution ends up
> killing performance.  However, there are a couple mitigating options
> for workloads that dirty small amounts and flush often that we need to
> collect data on:
> 
> 1/ Using cache management and pcommit from userspace to skip calls to
> msync / fsync.  Although, this does not eliminate all calls to
> blkdev_issue_flush as the fs may invoke it for other reasons.  I
> suspect turning on REQ_FUA support eliminates a number of those
> invocations, and pmem already satisfies REQ_FUA semantics by default.

Sure, I'll turn on REQ_FUA in addition to REQ_FLUSH - I agree that PMEM
already handles the requirements of REQ_FUA, but I didn't realize that it
might reduce the number of REQ_FLUSH bios we receive.

> 2/ Turn off DAX and use the page cache.  As Dave mentions [1] we
> should enable this control on a per-inode basis.  I'm folding in this
> capability as a blkdev_ioctl for the next version of the raw block DAX
> support patch.

Umm...I think you just said "the way to avoid this delay is to just not use
DAX".  :)  I don't think this is where we want to go - we are trying to make
DAX better, not abandon it.

> It's entirely possible these mitigations won't eliminate the need for
> a mechanism like this, but I think we have a bit more work to do to
> find out how bad this is in practice as well as the crossover point
> where walking the radix becomes prohibitive.

I'm guessing a single run through xfstests will be enough to convince you that
the "big hammer" approach is untenable.  Tests that used to take a second now
take several minutes, at least in my VM testing environment...  And that's
only using a tiny 4GiB namespace.

Yes, we can distribute the cost over multiple CPUs, but that just distributes
the problem and doesn't reduce the overall work that needs to be done.
Ultimately I think that looping through multiple GiB or even TiB of cache
lines and blindly writing them back individually on every REQ_FLUSH is going
to be a deal breaker.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1259756

On Fri, Oct 30, 2015 at 12:43 PM, Ross Zwisler
<ross.zwisler@linux.intel.com> wrote:
> On Fri, Oct 30, 2015 at 11:34:07AM -0700, Dan Williams wrote:
>> On Thu, Oct 29, 2015 at 1:12 PM, Ross Zwisler
>> <ross.zwisler@linux.intel.com> wrote:
>> > This patch series adds support for fsync/msync to DAX.
>> >
>> > Patches 1 through 8 add various utilities that the DAX code will eventually
>> > need, and the DAX code itself is added by patch 9.  Patches 10 and 11 are
>> > filesystem changes that are needed after the DAX code is added, but these
>> > patches may change slightly as the filesystem fault handling for DAX is
>> > being modified ([1] and [2]).
>> >
>> > I've marked this series as RFC because I'm still testing, but I wanted to
>> > get this out there so people would see the direction I was going and
>> > hopefully comment on any big red flags sooner rather than later.
>> >
>> > I realize that we are getting pretty dang close to the v4.4 merge window,
>> > but I think that if we can get this reviewed and working it's a much better
>> > solution than the "big hammer" approach that blindly flushes entire PMEM
>> > namespaces [3].
>> >
>> > [1] http://oss.sgi.com/archives/xfs/2015-10/msg00523.html
>> > [2] http://marc.info/?l=linux-ext4&m=144550211312472&w=2
>> > [3] https://lists.01.org/pipermail/linux-nvdimm/2015-October/002614.html
>> >
>> > Ross Zwisler (11):
>> >   pmem: add wb_cache_pmem() to the PMEM API
>> >   mm: add pmd_mkclean()
>> >   pmem: enable REQ_FLUSH handling
>> >   dax: support dirty DAX entries in radix tree
>> >   mm: add follow_pte_pmd()
>> >   mm: add pgoff_mkclean()
>> >   mm: add find_get_entries_tag()
>> >   fs: add get_block() to struct inode_operations
>> >   dax: add support for fsync/sync
>> >   xfs, ext2: call dax_pfn_mkwrite() on write fault
>> >   ext4: add ext4_dax_pfn_mkwrite()
>>
>> This is great to have when the flush-the-world solution ends up
>> killing performance.  However, there are a couple mitigating options
>> for workloads that dirty small amounts and flush often that we need to
>> collect data on:
>>
>> 1/ Using cache management and pcommit from userspace to skip calls to
>> msync / fsync.  Although, this does not eliminate all calls to
>> blkdev_issue_flush as the fs may invoke it for other reasons.  I
>> suspect turning on REQ_FUA support eliminates a number of those
>> invocations, and pmem already satisfies REQ_FUA semantics by default.
>
> Sure, I'll turn on REQ_FUA in addition to REQ_FLUSH - I agree that PMEM
> already handles the requirements of REQ_FUA, but I didn't realize that it
> might reduce the number of REQ_FLUSH bios we receive.

I'll let Dave chime in, but a lot of the flush requirements come from
guaranteeing the state of the metadata, if metadata updates can be
done with REQ_FUA then there is no subsequent need to flush.

>> 2/ Turn off DAX and use the page cache.  As Dave mentions [1] we
>> should enable this control on a per-inode basis.  I'm folding in this
>> capability as a blkdev_ioctl for the next version of the raw block DAX
>> support patch.
>
> Umm...I think you just said "the way to avoid this delay is to just not use
> DAX".  :)  I don't think this is where we want to go - we are trying to make
> DAX better, not abandon it.

That's a bit of an exaggeration.  Avoiding DAX where it is not
necessary is not "abandoning DAX", it's using the right tool for the
job.  Page cache is fine for many cases.

>> It's entirely possible these mitigations won't eliminate the need for
>> a mechanism like this, but I think we have a bit more work to do to
>> find out how bad this is in practice as well as the crossover point
>> where walking the radix becomes prohibitive.
>
> I'm guessing a single run through xfstests will be enough to convince you that
> the "big hammer" approach is untenable.  Tests that used to take a second now
> take several minutes, at least in my VM testing environment...  And that's
> only using a tiny 4GiB namespace.
>
> Yes, we can distribute the cost over multiple CPUs, but that just distributes
> the problem and doesn't reduce the overall work that needs to be done.
> Ultimately I think that looping through multiple GiB or even TiB of cache
> lines and blindly writing them back individually on every REQ_FLUSH is going
> to be a deal breaker.

Right, part of the problem is that the driver doesn't know which
blocks are actively DAX mapped.  I think we can incrementally fix that
without requiring DAX specific fsync/msync handling code for each fs
that supports DAX.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web