Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax, dma-to-storage, and swap

From	Dan Williams <dan.j.williams@intel.com>
Newsgroups	linux.kernel
Subject	Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax, dma-to-storage, and swap
Date	2017-08-12 00:30 +0200
Message-ID	<udqBl-Ku-29@gated-at.bofh.it> (permalink)
References	<uaAGR-Pk-3@gated-at.bofh.it> <uaAGS-Pk-17@gated-at.bofh.it> <ub42e-3iq-9@gated-at.bofh.it> <ubyWl-5Tz-3@gated-at.bofh.it> <udfFT-2pX-9@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

On Fri, Aug 11, 2017 at 3:44 AM, Christoph Hellwig <hch@lst.de> wrote:
> On Sun, Aug 06, 2017 at 11:51:50AM -0700, Dan Williams wrote:
>> Of course it's a useful API. An application already needs to worry
>> about the block map, that's why we have fallocate, msync, fiemap
>> and...
>
> Fallocate and msync do not expose the block map in any way.  Proof:
> they work just fine over say nfs.

Right, but they let userspace make inferences about the state of
metadata relative to I/O to a given storage address. In this regard
S_IOMAP_IMMUTABLE is no different than MAP_SYNC, but 'immutable' goes
a step further to let an application infer that the storage address is
stable. This enables applications that MAP_SYNC does not, see below.

> fiemap does indeed expose the block map, which is the whole point.
> But it's a debug tool that we don't event have a man page for.  And
> it's not usable for anything else, if only for the fact that it doesn't
> tell you what device your returned extents are relative to.

True, one couldn't just use immutable + fiemap and expect to have the
right storage device.

>
>> > We've been through this a few times but let me repeat it:  The only
>> > sensible API gurantee is one that is observable and usable.
>>
>> I'm missing how block-map immutable files violate this observable and
>> usable constraint?
>
> What is the observable behavior of an extent map change?  How can you
> describe your immutable extent map behavior so that when I violate
> them by e.g. moving one extent to a different place on disk you can
> observe that in userspace?

The violation is blocked, it's immutable. Using this feature means the
application is taking away some of the kernel's freedom. That is a
valid / safe tradeoff for the set of applications that would otherwise
resort to raw device access.

>
>> This immutable approach should also go in, it solves the same problem
>> without the the latency drawback,
>
> How is your latency going to be any different from MAP_SYNC on
> a fully allocated and pre-zeroed file?

So, I went back and read Jan's patches, and in the pre-allocated case
I don't think we can get stuck behind a backlog of dirty metada
flushing since the implementation only seems to take the synchronous
fault path if the fault dirtied the block map.

>> Beyond flush from userspace it also
>> can be used to solve the swapfile problems you highlighted
>
> Which swapfile problem?

The TOCTOU problem of enabling swap vs reflink that you mentioned in
your criticism of the daxctl syscall, but now that I look your
comments were based on the *general* case use of bmap(), However, xfs
in particular as of commits:

   eb5e248d502b xfs: don't allow bmap on rt files
   db1327b16c2b xfs: report shared extent mappings to userspace correctly

...doesn't appear to have this problem. That said Dave's idea to use
immutable + unwritten extents for swap makes sense to me. That's a
feature, not a bug fix, but I went ahead and appended a
proof-of-concept implementation to the v3 posting.

>> and it
>> allows safe ongoing dma to a filesystem-dax mapping beyond what we can
>> already do with direct-I/O.
>
> Please explain how this interface allows for any sort of safe userspace
> DMA.

So this is where I continue to see S_IOMAP_IMMUTABLE being able to
support applications that MAP_SYNC does not. Dave mentioned userspace
pNFS4 servers, but there's also Samba and other protocols that want to
negotiate a direct path to pmem outside the kernel. Xen support has
thus far not been able to follow in the footsteps of KVM enabling due
to a dependence on static M2P tables that assume a static
guest-physical to host-physical relationship [1]. Immutable files
would allow Xen to follow the same "mmap a file" semantic as KVM.

Applications that just want flush from userspace can use MAP_SYNC,
those that need to temporarily pin the block for RDMA can use the
in-kernel pNFS server, and those that need to coordinate both from
userspace can use S_IOMAP_IMMUTABLE. It's a continuum, not a
competition.

[1]: https://lists.xen.org/archives/html/xen-devel/2017-04/msg00427.html

Thread

[PATCH v2 0/5] fs, xfs: block map immutable files for dax,  dma-to-storage, and swap Dan Williams <dan.j.williams@intel.com> - 2017-08-04 04:40 +0200
  [PATCH v2 2/5] fs, xfs: introduce FALLOC_FL_SEAL_BLOCK_MAP Dan Williams <dan.j.williams@intel.com> - 2017-08-04 04:40 +0200
    Re: [PATCH v2 2/5] fs, xfs: introduce FALLOC_FL_SEAL_BLOCK_MAP "Darrick J. Wong" <darrick.wong@oracle.com> - 2017-08-04 21:50 +0200
      Re: [PATCH v2 2/5] fs, xfs: introduce FALLOC_FL_SEAL_BLOCK_MAP Dan Williams <dan.j.williams@intel.com> - 2017-08-04 22:00 +0200
    Re: [PATCH v2 2/5] fs, xfs: introduce FALLOC_FL_SEAL_BLOCK_MAP Dave Chinner <david@fromorbit.com> - 2017-08-05 01:40 +0200
      Re: [PATCH v2 2/5] fs, xfs: introduce FALLOC_FL_SEAL_BLOCK_MAP Dan Williams <dan.j.williams@intel.com> - 2017-08-05 01:50 +0200
        Re: [PATCH v2 2/5] fs, xfs: introduce FALLOC_FL_SEAL_BLOCK_MAP Dave Chinner <david@fromorbit.com> - 2017-08-05 02:10 +0200
  [PATCH v2 1/5] fs, xfs: introduce S_IOMAP_IMMUTABLE Dan Williams <dan.j.williams@intel.com> - 2017-08-04 04:40 +0200
    Re: [PATCH v2 1/5] fs, xfs: introduce S_IOMAP_IMMUTABLE "Darrick J. Wong" <darrick.wong@oracle.com> - 2017-08-04 22:10 +0200
      Re: [PATCH v2 1/5] fs, xfs: introduce S_IOMAP_IMMUTABLE Dan Williams <dan.j.williams@intel.com> - 2017-08-04 22:40 +0200
    Re: [PATCH v2 1/5] fs, xfs: introduce S_IOMAP_IMMUTABLE Christoph Hellwig <hch@lst.de> - 2017-08-05 11:50 +0200
      Re: [PATCH v2 1/5] fs, xfs: introduce S_IOMAP_IMMUTABLE Dave Chinner <david@fromorbit.com> - 2017-08-07 02:30 +0200
        Re: [PATCH v2 1/5] fs, xfs: introduce S_IOMAP_IMMUTABLE Christoph Hellwig <hch@lst.de> - 2017-08-11 12:40 +0200
  Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,  dma-to-storage, and swap Dan Williams <dan.j.williams@intel.com> - 2017-08-04 04:40 +0200
    Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,         dma-to-storage, and swap Christoph Hellwig <hch@lst.de> - 2017-08-05 12:00 +0200
      Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,  dma-to-storage, and swap Dan Williams <dan.j.williams@intel.com> - 2017-08-06 21:00 +0200
        Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,         dma-to-storage, and swap Christoph Hellwig <hch@lst.de> - 2017-08-11 12:50 +0200
          Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,  dma-to-storage, and swap Dan Williams <dan.j.williams@intel.com> - 2017-08-12 00:30 +0200
            Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,  dma-to-storage, and swap Andy Lutomirski <luto@kernel.org> - 2017-08-12 06:00 +0200
              Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,  dma-to-storage, and swap Dan Williams <dan.j.williams@intel.com> - 2017-08-12 06:50 +0200
              Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,         dma-to-storage, and swap Christoph Hellwig <hch@lst.de> - 2017-08-12 09:40 +0200
            Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,         dma-to-storage, and swap Christoph Hellwig <hch@lst.de> - 2017-08-12 09:40 +0200
              Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,  dma-to-storage, and swap Dan Williams <dan.j.williams@intel.com> - 2017-08-12 21:30 +0200
                Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,         dma-to-storage, and swap Christoph Hellwig <hch@lst.de> - 2017-08-13 11:30 +0200
                Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,  dma-to-storage, and swap Dan Williams <dan.j.williams@intel.com> - 2017-08-13 22:40 +0200
                Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,  dma-to-storage, and swap Jan Kara <jack@suse.cz> - 2017-08-14 14:50 +0200
                Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,  dma-to-storage, and swap Dan Williams <dan.j.williams@intel.com> - 2017-08-14 18:20 +0200
                Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,  dma-to-storage, and swap Jan Kara <jack@suse.cz> - 2017-08-15 10:40 +0200
                Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,  dma-to-storage, and swap Dan Williams <dan.j.williams@intel.com> - 2017-08-16 02:00 +0200
                Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,  dma-to-storage, and swap Jan Kara <jack@suse.cz> - 2017-08-16 16:00 +0200
                Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,  dma-to-storage, and swap Peter Zijlstra <peterz@infradead.org> - 2017-08-21 11:20 +0200
                Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,  dma-to-storage, and swap "Darrick J. Wong" <darrick.wong@oracle.com> - 2017-08-14 23:50 +0200
                Re: [PATCH v2 0/5] fs, xfs: block map immutable files for dax,  dma-to-storage, and swap Dave Chinner <david@fromorbit.com> - 2017-08-14 01:50 +0200
  [PATCH v2 5/5] xfs: toggle XFS_DIFLAG2_IOMAP_IMMUTABLE in response  to fallocate Dan Williams <dan.j.williams@intel.com> - 2017-08-04 04:40 +0200
    Re: [PATCH v2 5/5] xfs: toggle XFS_DIFLAG2_IOMAP_IMMUTABLE in  response to fallocate "Darrick J. Wong" <darrick.wong@oracle.com> - 2017-08-04 22:20 +0200
      Re: [PATCH v2 5/5] xfs: toggle XFS_DIFLAG2_IOMAP_IMMUTABLE in  response to fallocate Dan Williams <dan.j.williams@intel.com> - 2017-08-04 22:50 +0200
        Re: [PATCH v2 5/5] xfs: toggle XFS_DIFLAG2_IOMAP_IMMUTABLE in  response to fallocate Dan Williams <dan.j.williams@intel.com> - 2017-08-04 23:00 +0200
        Re: [PATCH v2 5/5] xfs: toggle XFS_DIFLAG2_IOMAP_IMMUTABLE in  response to fallocate "Darrick J. Wong" <darrick.wong@oracle.com> - 2017-08-04 23:00 +0200
  [PATCH v2 4/5] xfs: introduce XFS_DIFLAG2_IOMAP_IMMUTABLE Dan Williams <dan.j.williams@intel.com> - 2017-08-04 04:40 +0200
    Re: [PATCH v2 4/5] xfs: introduce XFS_DIFLAG2_IOMAP_IMMUTABLE "Darrick J. Wong" <darrick.wong@oracle.com> - 2017-08-04 22:40 +0200
      Re: [PATCH v2 4/5] xfs: introduce XFS_DIFLAG2_IOMAP_IMMUTABLE Dan Williams <dan.j.williams@intel.com> - 2017-08-04 22:50 +0200
      Re: [PATCH v2 4/5] xfs: introduce XFS_DIFLAG2_IOMAP_IMMUTABLE Dave Chinner <david@fromorbit.com> - 2017-08-05 01:50 +0200
        Re: [PATCH v2 4/5] xfs: introduce XFS_DIFLAG2_IOMAP_IMMUTABLE "Darrick J. Wong" <darrick.wong@oracle.com> - 2017-08-05 02:00 +0200

csiph-web