Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1496794 > unrolled thread

Re: [RFC PATCH-tip v4 02/10] locking/rwsem: Stop active read lock ASAP

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [RFC PATCH-tip v4 02/10] locking/rwsem: Stop active read lock  ASAP Davidlohr Bueso <dave@stgolabs.net> - 2016-10-06 20:20 +0200
    Re: [RFC PATCH-tip v4 02/10] locking/rwsem: Stop active read lock  ASAP Dave Chinner <david@fromorbit.com> - 2016-10-06 23:50 +0200
      Re: [RFC PATCH-tip v4 02/10] locking/rwsem: Stop active read lock  ASAP Davidlohr Bueso <dave@stgolabs.net> - 2016-10-07 01:00 +0200
      Re: [RFC PATCH-tip v4 02/10] locking/rwsem: Stop active read lock  ASAP Christoph Hellwig <hch@infradead.org> - 2016-10-09 17:20 +0200
        Re: [RFC PATCH-tip v4 02/10] locking/rwsem: Stop active read lock  ASAP Dave Chinner <david@fromorbit.com> - 2016-10-10 08:10 +0200
          Re: [RFC PATCH-tip v4 02/10] locking/rwsem: Stop active read lock  ASAP Christoph Hellwig <hch@infradead.org> - 2016-10-10 12:20 +0200
            Re: [RFC PATCH-tip v4 02/10] locking/rwsem: Stop active read lock  ASAP Dave Chinner <david@fromorbit.com> - 2016-10-11 23:10 +0200

#1496794 — Re: [RFC PATCH-tip v4 02/10] locking/rwsem: Stop active read lock ASAP

On Thu, 18 Aug 2016, Waiman Long wrote:

>Currently, when down_read() fails, the active read locking isn't undone
>until the rwsem_down_read_failed() function grabs the wait_lock. If the
>wait_lock is contended, it may takes a while to get the lock. During
>that period, writer lock stealing will be disabled because of the
>active read lock.
>
>This patch will release the active read lock ASAP so that writer lock
>stealing can happen sooner. The only downside is when the reader is
>the first one in the wait queue as it has to issue another atomic
>operation to update the count.
>
>On a 4-socket Haswell machine running on a 4.7-rc1 tip-based kernel,
>the fio test with multithreaded randrw and randwrite tests on the
>same file on a XFS partition on top of a NVDIMM with DAX were run,
>the aggregated bandwidths before and after the patch were as follows:
>
>  Test      BW before patch     BW after patch  % change
>  ----      ---------------     --------------  --------
>  randrw        1210 MB/s          1352 MB/s      +12%
>  randwrite     1622 MB/s          1710 MB/s      +5.4%

Yeah, this is really a bad workload to make decisions on locking
heuristics imo - if I'm thinking of the same workload. Mainly because
concurrent buffered io to the same file isn't very realistic and you
end up pathologically pounding on i_rwsem (which used to be until
recently i_mutex until Al's parallel lookup/readdir). Obviously write
lock stealing wins in this case.

Thanks,
Davidlohr

[toc] | [next] | [standalone]

#1496876

On Thu, Oct 06, 2016 at 11:17:18AM -0700, Davidlohr Bueso wrote:
> On Thu, 18 Aug 2016, Waiman Long wrote:
> 
> >Currently, when down_read() fails, the active read locking isn't undone
> >until the rwsem_down_read_failed() function grabs the wait_lock. If the
> >wait_lock is contended, it may takes a while to get the lock. During
> >that period, writer lock stealing will be disabled because of the
> >active read lock.
> >
> >This patch will release the active read lock ASAP so that writer lock
> >stealing can happen sooner. The only downside is when the reader is
> >the first one in the wait queue as it has to issue another atomic
> >operation to update the count.
> >
> >On a 4-socket Haswell machine running on a 4.7-rc1 tip-based kernel,
> >the fio test with multithreaded randrw and randwrite tests on the
> >same file on a XFS partition on top of a NVDIMM with DAX were run,
> >the aggregated bandwidths before and after the patch were as follows:
> >
> > Test      BW before patch     BW after patch  % change
> > ----      ---------------     --------------  --------
> > randrw        1210 MB/s          1352 MB/s      +12%
> > randwrite     1622 MB/s          1710 MB/s      +5.4%
> 
> Yeah, this is really a bad workload to make decisions on locking
> heuristics imo - if I'm thinking of the same workload. Mainly because
> concurrent buffered io to the same file isn't very realistic and you
> end up pathologically pounding on i_rwsem (which used to be until
> recently i_mutex until Al's parallel lookup/readdir). Obviously write
> lock stealing wins in this case.

Except that it's DAX, and in 4.7-rc1 that used shared locking at the
XFS level and never took exclusive locks.

*However*, the DAX IO path locking in XFS  has changed in 4.9-rc1 to
match the buffered IO single writer POSIX semantics - the test is a
bad test based on the fact it exercised a path that is under heavy
development and so can't be used as a regression test across
multiple kernels.

If you want to stress concurrent access to a single file, please
use direct IO, not DAX or buffered IO.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

[toc] | [prev] | [next] | [standalone]

#1496908

On Fri, 07 Oct 2016, Dave Chinner wrote:

>Except that it's DAX

Duh, of course; silly me.

Thanks,
Davidlohr

[toc] | [prev] | [next] | [standalone]

#1497933

On Fri, Oct 07, 2016 at 08:47:51AM +1100, Dave Chinner wrote:
> Except that it's DAX, and in 4.7-rc1 that used shared locking at the
> XFS level and never took exclusive locks.
> 
> *However*, the DAX IO path locking in XFS  has changed in 4.9-rc1 to
> match the buffered IO single writer POSIX semantics - the test is a
> bad test based on the fact it exercised a path that is under heavy
> development and so can't be used as a regression test across
> multiple kernels.

That being said - I wonder if we should allow the shared lock on DAX
files IFF the user is specifying O_DIRECT in the open mode..

[toc] | [prev] | [next] | [standalone]

#1498116

On Sun, Oct 09, 2016 at 08:17:48AM -0700, Christoph Hellwig wrote:
> On Fri, Oct 07, 2016 at 08:47:51AM +1100, Dave Chinner wrote:
> > Except that it's DAX, and in 4.7-rc1 that used shared locking at the
> > XFS level and never took exclusive locks.
> > 
> > *However*, the DAX IO path locking in XFS  has changed in 4.9-rc1 to
> > match the buffered IO single writer POSIX semantics - the test is a
> > bad test based on the fact it exercised a path that is under heavy
> > development and so can't be used as a regression test across
> > multiple kernels.
> 
> That being said - I wonder if we should allow the shared lock on DAX
> files IFF the user is specifying O_DIRECT in the open mode..

It should do - if it doesn't then we screwed up the IO path
selection logic in XFS and we'll need to fix it.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

[toc] | [prev] | [next] | [standalone]

#1498215

On Mon, Oct 10, 2016 at 05:07:45PM +1100, Dave Chinner wrote:
> > > *However*, the DAX IO path locking in XFS  has changed in 4.9-rc1 to
> > > match the buffered IO single writer POSIX semantics - the test is a
> > > bad test based on the fact it exercised a path that is under heavy
> > > development and so can't be used as a regression test across
> > > multiple kernels.
> > 
> > That being said - I wonder if we should allow the shared lock on DAX
> > files IFF the user is specifying O_DIRECT in the open mode..
> 
> It should do - if it doesn't then we screwed up the IO path
> selection logic in XFS and we'll need to fix it.

Depends on your defintion of "we".  The DAX code has always abused the
direct I/O path, and that abuse is ingrained in the VFS path in a way that
we can't easily undo it in XFS, e.g. take a look at io_is_direct and
iocb_flags in include/linux/fs.h, which ensure that DAX I/O will always
appear as IOCB_DIRECT to the fs.  It will take some time to untagle
this, but it's on my todo list once the last file system (ext4)
untangles the DAX and direct I/O path.

[toc] | [prev] | [next] | [standalone]

#1499207

On Mon, Oct 10, 2016 at 02:34:34AM -0700, Christoph Hellwig wrote:
> On Mon, Oct 10, 2016 at 05:07:45PM +1100, Dave Chinner wrote:
> > > > *However*, the DAX IO path locking in XFS  has changed in 4.9-rc1 to
> > > > match the buffered IO single writer POSIX semantics - the test is a
> > > > bad test based on the fact it exercised a path that is under heavy
> > > > development and so can't be used as a regression test across
> > > > multiple kernels.
> > > 
> > > That being said - I wonder if we should allow the shared lock on DAX
> > > files IFF the user is specifying O_DIRECT in the open mode..
> > 
> > It should do - if it doesn't then we screwed up the IO path
> > selection logic in XFS and we'll need to fix it.
> 
> Depends on your defintion of "we".  The DAX code has always abused the
> direct I/O path, and that abuse is ingrained in the VFS path in a way that
> we can't easily undo it in XFS, e.g. take a look at io_is_direct and
> iocb_flags in include/linux/fs.h, which ensure that DAX I/O will always
> appear as IOCB_DIRECT to the fs. 

Um, I seem to have completely missed that change - when did that
happen and why?

Oh, it was part of the misguided "enable DAX on block devices"
changes - it was supposed to fix a problem with block device access
using buffered IO instead of DAX (commit 65f87ee71852 "fs, block:
force direct-I/O for dax-enabled block devices")).  The original
block device DAX code was reverted soon after this because it was
badly flawed, but this change was not removed at the same time
(probably because it was forgotton about).

Hence I'd suggest that DAX check in io_is_direct() should be removed
ASAP; the filesystems don't need it as they check the inode DAX
state directly, and the code it "fixed" is no longer in the tree.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web