Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1563272 > unrolled thread

[PATCH 00/13] dax, pmem: move cpu cache maintenance to libnvdimm

Back to article view | Back to linux.kernel

Contents

  [PATCH 00/13] dax, pmem: move cpu cache maintenance to libnvdimm Dan Williams <dan.j.williams@intel.com> - 2017-01-20 05:00 +0100
    [PATCH 08/13] x86, libnvdimm, dax: stop abusing __copy_user_nocache Dan Williams <dan.j.williams@intel.com> - 2017-01-20 05:00 +0100
    [PATCH 10/13] libnvdimm, pmem: fix persistence warning Dan Williams <dan.j.williams@intel.com> - 2017-01-20 05:00 +0100
    [PATCH 07/13] x86, libnvdimm,  pmem: move arch_invalidate_pmem() to libnvdimm Dan Williams <dan.j.williams@intel.com> - 2017-01-20 05:00 +0100
    [PATCH 01/13] x86, dax,  pmem: remove indirection around memcpy_from_pmem() Dan Williams <dan.j.williams@intel.com> - 2017-01-20 05:00 +0100
    [PATCH 05/13] x86,  dax: replace clear_pmem() with open coded memset + dax_ops->flush Dan Williams <dan.j.williams@intel.com> - 2017-01-20 05:00 +0100
      Re: [PATCH 05/13] x86, dax: replace clear_pmem() with open coded  memset + dax_ops->flush Jan Kara <jack@suse.cz> - 2017-01-20 11:50 +0100
        Re: [PATCH 05/13] x86, dax: replace clear_pmem() with open coded  memset + dax_ops->flush Dan Williams <dan.j.williams@intel.com> - 2017-01-20 16:40 +0100
    [PATCH 03/13] x86, dax,  pmem: introduce 'copy_from_iter' dax operation Dan Williams <dan.j.williams@intel.com> - 2017-01-20 05:00 +0100
    [PATCH 13/13] libnvdimm,  pmem: disable dax flushing for 'cache flush on fail' platforms Dan Williams <dan.j.williams@intel.com> - 2017-01-20 05:00 +0100
    [PATCH 09/13] libnvdimm,  pmem: implement cache bypass for all copy_from_iter() operations Dan Williams <dan.j.williams@intel.com> - 2017-01-20 05:00 +0100
    [PATCH 04/13] dax, pmem: introduce an optional 'flush' dax operation Dan Williams <dan.j.williams@intel.com> - 2017-01-20 05:00 +0100
    [PATCH 06/13] x86, dax, libnvdimm: move wb_cache_pmem() to libnvdimm Dan Williams <dan.j.williams@intel.com> - 2017-01-20 05:00 +0100
    RE: [PATCH 00/13] dax, pmem: move cpu cache maintenance to libnvdimm Matthew Wilcox <mawilcox@microsoft.com> - 2017-01-21 17:30 +0100
      Re: [PATCH 00/13] dax, pmem: move cpu cache maintenance to         libnvdimm Christoph Hellwig <hch@lst.de> - 2017-01-21 19:00 +0100
        RE: [PATCH 00/13] dax, pmem: move cpu cache maintenance to     libnvdimm Matthew Wilcox <mawilcox@microsoft.com> - 2017-01-22 16:50 +0100
          Re: [PATCH 00/13] dax, pmem: move cpu cache maintenance to         libnvdimm Christoph Hellwig <hch@lst.de> - 2017-01-22 17:40 +0100
            RE: [PATCH 00/13] dax, pmem: move cpu cache maintenance to     libnvdimm Matthew Wilcox <mawilcox@microsoft.com> - 2017-01-22 19:20 +0100
              Re: [PATCH 00/13] dax, pmem: move cpu cache maintenance to         libnvdimm Christoph Hellwig <hch@lst.de> - 2017-01-22 19:40 +0100
                RE: [PATCH 00/13] dax, pmem: move cpu cache maintenance to     libnvdimm Matthew Wilcox <mawilcox@microsoft.com> - 2017-01-22 19:40 +0100
                  Re: [PATCH 00/13] dax, pmem: move cpu cache maintenance to         libnvdimm Christoph Hellwig <hch@lst.de> - 2017-01-22 19:50 +0100
                    RE: [PATCH 00/13] dax, pmem: move cpu cache maintenance to     libnvdimm Matthew Wilcox <mawilcox@microsoft.com> - 2017-01-23 07:40 +0100
                      Re: [PATCH 00/13] dax, pmem: move cpu cache maintenance to libnvdimm Dan Williams <dan.j.williams@intel.com> - 2017-01-23 08:20 +0100
                        Re: [PATCH 00/13] dax, pmem: move cpu cache maintenance to         libnvdimm Christoph Hellwig <hch@lst.de> - 2017-01-23 17:10 +0100
                          Re: [PATCH 00/13] dax, pmem: move cpu cache maintenance to libnvdimm Dan Williams <dan.j.williams@intel.com> - 2017-01-23 18:20 +0100
                            Re: [PATCH 00/13] dax, pmem: move cpu cache maintenance to         libnvdimm Christoph Hellwig <hch@lst.de> - 2017-01-23 19:10 +0100
                              Re: [PATCH 00/13] dax, pmem: move cpu cache maintenance to libnvdimm Dan Williams <dan.j.williams@intel.com> - 2017-01-23 19:40 +0100
                      Re: [PATCH 00/13] dax, pmem: move cpu cache maintenance to         libnvdimm Christoph Hellwig <hch@lst.de> - 2017-01-23 17:00 +0100
        Re: [PATCH 00/13] dax, pmem: move cpu cache maintenance to libnvdimm Dan Williams <dan.j.williams@intel.com> - 2017-01-22 18:40 +0100
          Re: [PATCH 00/13] dax, pmem: move cpu cache maintenance to         libnvdimm Christoph Hellwig <hch@lst.de> - 2017-01-23 17:10 +0100

Page 2 of 2 — ← Prev page 1 [2]

#1564549 — Re: [PATCH 00/13] dax, pmem: move cpu cache maintenance to libnvdimm

On Sun, Jan 22, 2017 at 06:39:28PM +0000, Matthew Wilcox wrote:
> Two guests on the same physical machine (or a guest and a host) have access to the same set of physical addresses.  This might be an NV-DIMM, or it might just be DRAM (for the purposes of reducing guest overhead).  The network filesystem has been enhanced with a call to allow the client to ask the server "What is the physical address for this range of bytes in this file?"
> 
> We don't want to use the guest pagecache here.  That's antithetical to the second usage, and it's inefficient for the first usage.

And the answer is that you need a dax device for whatever memoery exposed
in this way, as it needs to show up in the memory map for example.

[toc] | [prev] | [next] | [standalone]

#1564708 — RE: [PATCH 00/13] dax, pmem: move cpu cache maintenance to libnvdimm

From: Christoph Hellwig [mailto:hch@lst.de]
> On Sun, Jan 22, 2017 at 06:39:28PM +0000, Matthew Wilcox wrote:
> > Two guests on the same physical machine (or a guest and a host) have access
> > to the same set of physical addresses.  This might be an NV-DIMM, or it might
> > just be DRAM (for the purposes of reducing guest overhead).  The network
> > filesystem has been enhanced with a call to allow the client to ask the server
> > "What is the physical address for this range of bytes in this file?"
> >
> > We don't want to use the guest pagecache here.  That's antithetical to the
> > second usage, and it's inefficient for the first usage.
> 
> And the answer is that you need a dax device for whatever memoery exposed
> in this way, as it needs to show up in the memory map for example.

Wow, DAX devices look painful and awful.  I certainly don't want to be exposing the memory fronted by my network filesystem to userspace to access.  That just seems like a world of pain and bad experiences.  Absolutely the filesystem (or perhaps better, the ACPI tables) need to mark that chunk of memory as reserved, but it's definitely not available for anyone to access without the filesystem being aware.

Even if we let the filesystem create a DAX device that doesn't show up in /dev (for example), Dan's patches don't give us a way to go from a file on the filesystem to a set of dax_ops.  And it does need to be a per-file operation, eg to support a file on an XFS volume which might be on a RT device or a normal device.  That was why I leaned towards an address_space operation, but I'd be happy to see an inode_operation instead. 

[toc] | [prev] | [next] | [standalone]

#1564720

On Sun, Jan 22, 2017 at 10:37 PM, Matthew Wilcox <mawilcox@microsoft.com> wrote:
> From: Christoph Hellwig [mailto:hch@lst.de]
>> On Sun, Jan 22, 2017 at 06:39:28PM +0000, Matthew Wilcox wrote:
>> > Two guests on the same physical machine (or a guest and a host) have access
>> > to the same set of physical addresses.  This might be an NV-DIMM, or it might
>> > just be DRAM (for the purposes of reducing guest overhead).  The network
>> > filesystem has been enhanced with a call to allow the client to ask the server
>> > "What is the physical address for this range of bytes in this file?"
>> >
>> > We don't want to use the guest pagecache here.  That's antithetical to the
>> > second usage, and it's inefficient for the first usage.
>>
>> And the answer is that you need a dax device for whatever memoery exposed
>> in this way, as it needs to show up in the memory map for example.
>
> Wow, DAX devices look painful and awful.  I certainly don't want to be exposing the memory fronted by my network filesystem to userspace to access.  That just seems like a world of pain and bad experiences.  Absolutely the filesystem (or perhaps better, the ACPI tables) need to mark that chunk of memory as reserved, but it's definitely not available for anyone to access without the filesystem being aware.
>
> Even if we let the filesystem create a DAX device that doesn't show up in /dev (for example), Dan's patches don't give us a way to go from a file on the filesystem to a set of dax_ops.  And it does need to be a per-file operation, eg to support a file on an XFS volume which might be on a RT device or a normal device.  That was why I leaned towards an address_space operation, but I'd be happy to see an inode_operation instead.

How about we solve the copy_from_user() abuse first before we hijack
this thread for some future feature that afaics has no patches posted
yet.

An incremental step towards disentangling filesystem-dax from
block_devices is a lookup mechanism to go from a block_device to a dax
object that holds dax_ops. When this brave new filesystem enabling
appears it can grow a mechanism to lookup, or mount on, the dax object
directly.

One idea is to just hang a pointer to this dax object off of
bdev_inode, set at bdev open() time.

[toc] | [prev] | [next] | [standalone]

#1565084 — Re: [PATCH 00/13] dax, pmem: move cpu cache maintenance to libnvdimm

On Sun, Jan 22, 2017 at 11:10:04PM -0800, Dan Williams wrote:
> How about we solve the copy_from_user() abuse first before we hijack
> this thread for some future feature that afaics has no patches posted
> yet.

Solving copy_from_user abuse first sounds perfectly fine to me.  But
please do so without abusing the block layer for persistent memory
access.  Given that we don't have use cases for different pmem access
methods in a single OS image yet let's avoid introducing new ops
for now and just remove the copy_from_user abuse.

In the longer run I like your dax_operations, but they need to be
separate from the block layer.

[toc] | [prev] | [next] | [standalone]

#1565166

On Mon, Jan 23, 2017 at 8:00 AM, Christoph Hellwig <hch@lst.de> wrote:
> On Sun, Jan 22, 2017 at 11:10:04PM -0800, Dan Williams wrote:
>> How about we solve the copy_from_user() abuse first before we hijack
>> this thread for some future feature that afaics has no patches posted
>> yet.
>
> Solving copy_from_user abuse first sounds perfectly fine to me.  But
> please do so without abusing the block layer for persistent memory
> access.  Given that we don't have use cases for different pmem access
> methods in a single OS image yet let's avoid introducing new ops
> for now and just remove the copy_from_user abuse.

The use case that we have now is distinguishing volatile vs persistent
memory (brd vs pmem).

I took a look at mtd layering approach and the main difference is that
layers above the block layer do not appear to know anything about mtd
specifics. For fs/dax.c we currently need some path to retrieve a dax
anchor object through the block device.

> In the longer run I like your dax_operations, but they need to be
> separate from the block layer.

I'll move them from block_device_operations to dax data hanging off of
the bdev_inode, or is there a better way to go from bdev-to-dax?

[toc] | [prev] | [next] | [standalone]

#1565201 — Re: [PATCH 00/13] dax, pmem: move cpu cache maintenance to libnvdimm

On Mon, Jan 23, 2017 at 09:14:04AM -0800, Dan Williams wrote:
> The use case that we have now is distinguishing volatile vs persistent
> memory (brd vs pmem).

brd is a development tool, so until we have other reasons for this
abstraction (which I'm pretty sure will show up rather sooner than later)
I would not worry about it too much.

> I took a look at mtd layering approach and the main difference is that
> layers above the block layer do not appear to know anything about mtd
> specifics.

Or the block layer itself for that matter.  And that's exactly where
I want DAX to be in the future.

> For fs/dax.c we currently need some path to retrieve a dax
> anchor object through the block device.

We have a need to retreiver the anchor object.  We currently do it
though the block layer for historical reasons, but it doesn't have
to be that way.

> > In the longer run I like your dax_operations, but they need to be
> > separate from the block layer.
> 
> I'll move them from block_device_operations to dax data hanging off of
> the bdev_inode, or is there a better way to go from bdev-to-dax?

I don't think that's any better.  What we really want is a way
to find the underlying persistent memory / DAX / whatever we call
it node without going through a block device.  E.g. a library function
to give that object for a given path name, where the path name could
be either that of the /dev/pmemN or the /dev/daxN device.

If the file system for now still needs a block device as well it
will only accept the /dev/pmemN name, and open both the low-level
pmem device and the block device.  Once that file system doesn't
need block code (and I think we could do that easily for XFS,
nevermind any new FS) it won't have to deal with the block
device at all.

pmem.c then becomes a consumer of the dax_ops just like the file system.

[toc] | [prev] | [next] | [standalone]

#1565234

On Mon, Jan 23, 2017 at 10:03 AM, Christoph Hellwig <hch@lst.de> wrote:
> On Mon, Jan 23, 2017 at 09:14:04AM -0800, Dan Williams wrote:
>> The use case that we have now is distinguishing volatile vs persistent
>> memory (brd vs pmem).
>
> brd is a development tool, so until we have other reasons for this
> abstraction (which I'm pretty sure will show up rather sooner than later)
> I would not worry about it too much.

By "volatile" I also meant cases where pmem is fronting volatile
memory, or more importantly when the platform has otherwise arranged
for cpu caches to be flushed on a power loss event like I believe some
existing storage appliances do.

>> I took a look at mtd layering approach and the main difference is that
>> layers above the block layer do not appear to know anything about mtd
>> specifics.
>
> Or the block layer itself for that matter.  And that's exactly where
> I want DAX to be in the future.
>
>> For fs/dax.c we currently need some path to retrieve a dax
>> anchor object through the block device.
>
> We have a need to retreiver the anchor object.  We currently do it
> though the block layer for historical reasons, but it doesn't have
> to be that way.
>
>> > In the longer run I like your dax_operations, but they need to be
>> > separate from the block layer.
>>
>> I'll move them from block_device_operations to dax data hanging off of
>> the bdev_inode, or is there a better way to go from bdev-to-dax?
>
> I don't think that's any better.  What we really want is a way
> to find the underlying persistent memory / DAX / whatever we call
> it node without going through a block device.  E.g. a library function
> to give that object for a given path name, where the path name could
> be either that of the /dev/pmemN or the /dev/daxN device.
>
> If the file system for now still needs a block device as well it
> will only accept the /dev/pmemN name, and open both the low-level
> pmem device and the block device.  Once that file system doesn't
> need block code (and I think we could do that easily for XFS,
> nevermind any new FS) it won't have to deal with the block
> device at all.
>
> pmem.c then becomes a consumer of the dax_ops just like the file system.

Ah ok, I'll take a look at a dax_by_path() capability.

[toc] | [prev] | [next] | [standalone]

#1565080 — Re: [PATCH 00/13] dax, pmem: move cpu cache maintenance to libnvdimm

On Mon, Jan 23, 2017 at 06:37:18AM +0000, Matthew Wilcox wrote:
> Wow, DAX devices look painful and awful.  I certainly don't want to be
> exposing the memory fronted by my network filesystem to userspace to
> access.  That just seems like a world of pain and bad experiences.

So what is your interest in using DAX for your file system then instead
of a private mechanisms?

> Absolutely the filesystem (or perhaps better, the ACPI tables) need to
> mark that chunk of memory as reserved, but it's definitely not available
> for anyone to access without the filesystem being aware.

That does sounds like a massive special case all over the stack.
But until we see it I think we should simply ignore this case and
concentrate on what we have right now.

> Even if we let the filesystem create a DAX device that doesn't show
> up in /dev (for example), Dan's patches don't give us a way to go
> from a file on the filesystem to a set of dax_ops.

Which doesn't make sense any way.  The entry points into the file system
are read + write and mmap, and the file system might then use libraries
to implement different types of I/O, such as the page cache or DAX.

> And it does need to be a per-file operation, eg to support a file on
> an XFS volume which might be on a RT device or a normal device.
> That was why I leaned towards an address_space operation, but I'd be
> happy to see an inode_operation instead. 

Again, no.  The layers above the file system have absolutely no business
to even know if we're using DAX or pagecache access, nevermind how
in detail they are used.  Assuming you want to use DAX-like semantics
it's up to the lower level to expose the correct operations for
a given memory region.  Right now these would just be intel nfit or
legacy 820 + ADR for regions marked such in the memory map.  If say
a hypervisor wants to expose a region that needs a special flush
call or even has requirements on the type of memcpy it needs to
provide operations for this memory region.  The user of this region
(DAX-native file system pmem driver or device DAX) then needs
to use these methods.

And those pretty much are the methods Dan proposes here - it's
just that we should not tie them to block device operations, at
least not in the long run.

[toc] | [prev] | [next] | [standalone]

#1564476

On Sat, Jan 21, 2017 at 9:52 AM, Christoph Hellwig <hch@lst.de> wrote:
> On Sat, Jan 21, 2017 at 04:28:52PM +0000, Matthew Wilcox wrote:
>> Of course, there may not be a backing device either!
>
> s/backing device/block device/ ?  If so fully agreed.  I like the dax_ops
> scheme, but we should go all the way and detangle it from the block
> device.  I already brought up this issue with the fallback to direct I/O
> on I/O error series.
>
>> I see two possible routes here:
>>
>> 1. Add a new address_space_operation:
>>
>>       const struct dax_operations *(*get_dax_ops)(struct address_space *);
>>
>> 2. Add two of the dax_operations to address_space_operations:
>>
>>       size_t (*copy_from_iter)(struct address_space *, void *, size_t, struct iov_iter *);
>>       void (*flush)(struct address_space *, void *, size_t);
>> (we won't need ->direct_access as an address_space op because that'll be handled a different way in the brave new world that supports non-bdev-based filesystems)
>
> And both of them are wrong.  The write_begin/write_end mistake
> notwithstanding address_space ops are operations the VM can call without
> knowing things like fs locking contexts.  The above on the other hand
> are device operations provided by the low-level driver, similar to
> block_device operations.  So what we need is to have a way to mount
> a dax device as a file system, similar to how we support that for block
> or MTD devices and can then call methods on it.  For now this will
> be a bit complicated because all current DAX-aware file systems also
> still need block device for the metadata path, so we can't just say
> you mount either a DAX or block device.  But I think we should aim
> for mounting a DAX device as the primary use case, and then deal
> with block device emulation as a generic DAX layer thing, similarly
> how we implement (bad in the rw case) block devices on top of MTD.

So are you saying we need a way to go from a block_device inode to a
dax_device inode and then look up the dax_operations from there?

A filesystem, if it so chooses, could mount on top of the dax_device
inode directly?

I did add a dax_superblock for the device-dax character device
representation I could refactor that so the block_device presentation
of a namespace and a character device presentation are just different
layers on top of the base-level dax inode.

...or am I not tracking what you are suggesting?

[toc] | [prev] | [next] | [standalone]

#1565092 — Re: [PATCH 00/13] dax, pmem: move cpu cache maintenance to libnvdimm

On Sun, Jan 22, 2017 at 09:30:23AM -0800, Dan Williams wrote:
> So are you saying we need a way to go from a block_device inode to a
> dax_device inode and then look up the dax_operations from there?
> 
> A filesystem, if it so chooses, could mount on top of the dax_device
> inode directly?

Sentence 1: maybe if we have to.  Sentence 2: absolutely.

> I did add a dax_superblock for the device-dax character device
> representation I could refactor that so the block_device presentation
> of a namespace and a character device presentation are just different
> layers on top of the base-level dax inode.

That's a good start.

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web