Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1441860 > unrolled thread

Re: dm stripe: add DAX support

Started by"Kani, Toshimitsu" <toshi.kani@hpe.com>
First post2016-07-13 01:00 +0200
Last post2016-07-21 05:10 +0200
Articles 5 — 3 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: dm stripe: add DAX support "Kani, Toshimitsu" <toshi.kani@hpe.com> - 2016-07-13 01:00 +0200
    Re: dm stripe: add DAX support Mike Snitzer <snitzer@redhat.com> - 2016-07-13 04:10 +0200
      Re: dm stripe: add DAX support "Kani, Toshimitsu" <toshi.kani@hpe.com> - 2016-07-13 17:40 +0200
        Re: dm stripe: add DAX support Mike Snitzer <snitzer@redhat.com> - 2016-07-21 02:10 +0200
          Re: dm stripe: add DAX support Jens Axboe <axboe@kernel.dk> - 2016-07-21 05:10 +0200

#1441860 — Re: dm stripe: add DAX support

From"Kani, Toshimitsu" <toshi.kani@hpe.com>
Date2016-07-13 01:00 +0200
SubjectRe: dm stripe: add DAX support
Message-ID<rUeOJ-2RS-5@gated-at.bofh.it>
On Fri, 2016-06-24 at 14:29 -0400, Mike Snitzer wrote:
> 
> BTW, if in your testing you could evaluate/quantify any extra overhead
> from DM that'd be useful to share.  It could be there are bottlenecks
> that need to be fixed, etc.

Here are some results from fio benchmark.  The test is single-threaded and is
bound to one CPU.

 DAX  LVM   IOPS   NOTE
 ---------------------------------------
  Y    N    790K
  Y    Y    754K   5% overhead with LVM
  N    N    567K
  N    Y    457K   20% overhead with LVM

 DAX: Y: mount -o dax,noatime, N: mount -o noatime
 LVM: Y: dm-linear on pmem0 device, N: pmem0 device
 fio: bs=4k, size=2G, direct=1, rw=randread, numjobs=1

Among the 5% overhead with DAX/LVM, the new DM direct_access interfaces
account for less than 0.5%.

 dm_blk_direct_access 0.28%
 linear_direct_access 0.17%

The average latency increases slightly from 0.93us to 0.95us.  I think most of
the overhead comes from the submit_bio() path, which is used only for
accessing metadata with DAX.  I believe this is due to cloning bio for each
request in DM.  There is 12% more L2 miss in total.

Without DAX, 20% overhead is observed with LVM.  Average latency increases
from 1.39us to 1.82us.  Without DAX, bio is cloned for both data and metadata.

Thanks,
-Toshi

[toc] | [next] | [standalone]

#1441958

FromMike Snitzer <snitzer@redhat.com>
Date2016-07-13 04:10 +0200
Message-ID<rUhMB-4Y0-3@gated-at.bofh.it>
In reply to#1441860
On Tue, Jul 12 2016 at  6:22pm -0400,
Kani, Toshimitsu <toshi.kani@hpe.com> wrote:

> On Fri, 2016-06-24 at 14:29 -0400, Mike Snitzer wrote:
> > 
> > BTW, if in your testing you could evaluate/quantify any extra overhead
> > from DM that'd be useful to share.  It could be there are bottlenecks
> > that need to be fixed, etc.
> 
> Here are some results from fio benchmark.  The test is single-threaded and is
> bound to one CPU.
> 
>  DAX  LVM   IOPS   NOTE
>  ---------------------------------------
>   Y    N    790K
>   Y    Y    754K   5% overhead with LVM
>   N    N    567K
>   N    Y    457K   20% overhead with LVM
> 
>  DAX: Y: mount -o dax,noatime, N: mount -o noatime
>  LVM: Y: dm-linear on pmem0 device, N: pmem0 device
>  fio: bs=4k, size=2G, direct=1, rw=randread, numjobs=1
> 
> Among the 5% overhead with DAX/LVM, the new DM direct_access interfaces
> account for less than 0.5%.
> 
>  dm_blk_direct_access 0.28%
>  linear_direct_access 0.17%
> 
> The average latency increases slightly from 0.93us to 0.95us.  I think most of
> the overhead comes from the submit_bio() path, which is used only for
> accessing metadata with DAX.  I believe this is due to cloning bio for each
> request in DM.  There is 12% more L2 miss in total.
> 
> Without DAX, 20% overhead is observed with LVM.  Average latency increases
> from 1.39us to 1.82us.  Without DAX, bio is cloned for both data and metadata.

Thanks for putting this summary together.  Unfortunately none of the DM
changes can be queued for 4.8 until Jens takes the 2 block core patches:
https://patchwork.kernel.org/patch/9196021/
https://patchwork.kernel.org/patch/9196019/

Not sure what the hold up and/or issue is with them.  But I've asked
twice (and implicilty a 3rd time here).  Hopefully they land in time for
4.8.

Mike

[toc] | [prev] | [next] | [standalone]

#1442558

From"Kani, Toshimitsu" <toshi.kani@hpe.com>
Date2016-07-13 17:40 +0200
Message-ID<rUuqt-4Vy-19@gated-at.bofh.it>
In reply to#1441958
On Tue, 2016-07-12 at 22:01 -0400, Mike Snitzer wrote:
> On Tue, Jul 12 2016 at  6:22pm -0400,
> Kani, Toshimitsu <toshi.kani@hpe.com> wrote:
> > On Fri, 2016-06-24 at 14:29 -0400, Mike Snitzer wrote:
> > >  
 :
> Thanks for putting this summary together.  Unfortunately none of the DM
> changes can be queued for 4.8 until Jens takes the 2 block core patches:
> https://patchwork.kernel.org/patch/9196021/
> https://patchwork.kernel.org/patch/9196019/
> 
> Not sure what the hold up and/or issue is with them.  But I've asked
> twice (and implicilty a 3rd time here).  Hopefully they land in time for
> 4.8.

Hi Jens,

Can you take the two patches above?  These patches add QUEUE_FLAG_DAX and its
sysfs show func.  They allow device-mapper to set dax-capability based on its
configuration, and also allow user applications to check dax-capability via
sysfs.

Thanks,
-Toshi 

[toc] | [prev] | [next] | [standalone]

#1447544

FromMike Snitzer <snitzer@redhat.com>
Date2016-07-21 02:10 +0200
Message-ID<rX9IR-8qH-7@gated-at.bofh.it>
In reply to#1442558
On Wed, Jul 13 2016 at 11:03am -0400,
Kani, Toshimitsu <toshi.kani@hpe.com> wrote:

> On Tue, 2016-07-12 at 22:01 -0400, Mike Snitzer wrote:
> > On Tue, Jul 12 2016 at  6:22pm -0400,
> > Kani, Toshimitsu <toshi.kani@hpe.com> wrote:
> > > On Fri, 2016-06-24 at 14:29 -0400, Mike Snitzer wrote:
> > > >  
>  :
> > Thanks for putting this summary together.  Unfortunately none of the DM
> > changes can be queued for 4.8 until Jens takes the 2 block core patches:
> > https://patchwork.kernel.org/patch/9196021/
> > https://patchwork.kernel.org/patch/9196019/
> > 
> > Not sure what the hold up and/or issue is with them.  But I've asked
> > twice (and implicilty a 3rd time here).  Hopefully they land in time for
> > 4.8.
> 
> Hi Jens,
> 
> Can you take the two patches above?  These patches add QUEUE_FLAG_DAX and its
> sysfs show func.  They allow device-mapper to set dax-capability based on its
> configuration, and also allow user applications to check dax-capability via
> sysfs.

Hi Jens,

As I shared with you before, here are the 2 patches I staged in
linux-next via linux-dm.git's 'for-next' (same as in the patchwork
references above) -- these were staged just so we had linux-next
coverage until you picked them up.

Please pick these up for 4.8 -- the changes are straightforward, are
required for DM's DAX support, and Dan Williams has also acked them:

https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=6c00531c6affa42b165df73a1eac3289bc45f4c4
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=51d1a5abcd0e88cb4528f35643245ea59cf234f1

Feel free to pick them up from patchwork or cherry-pick from
linux-dm.git

Thanks!
Mike

[toc] | [prev] | [next] | [standalone]

#1447589

FromJens Axboe <axboe@kernel.dk>
Date2016-07-21 05:10 +0200
Message-ID<rXcx3-1PH-7@gated-at.bofh.it>
In reply to#1447544
On 07/20/2016 06:01 PM, Mike Snitzer wrote:
> On Wed, Jul 13 2016 at 11:03am -0400,
> Kani, Toshimitsu <toshi.kani@hpe.com> wrote:
>
>> On Tue, 2016-07-12 at 22:01 -0400, Mike Snitzer wrote:
>>> On Tue, Jul 12 2016 at  6:22pm -0400,
>>> Kani, Toshimitsu <toshi.kani@hpe.com> wrote:
>>>> On Fri, 2016-06-24 at 14:29 -0400, Mike Snitzer wrote:
>>>>>
>>  :
>>> Thanks for putting this summary together.  Unfortunately none of the DM
>>> changes can be queued for 4.8 until Jens takes the 2 block core patches:
>>> https://patchwork.kernel.org/patch/9196021/
>>> https://patchwork.kernel.org/patch/9196019/
>>>
>>> Not sure what the hold up and/or issue is with them.  But I've asked
>>> twice (and implicilty a 3rd time here).  Hopefully they land in time for
>>> 4.8.
>>
>> Hi Jens,
>>
>> Can you take the two patches above?  These patches add QUEUE_FLAG_DAX and its
>> sysfs show func.  They allow device-mapper to set dax-capability based on its
>> configuration, and also allow user applications to check dax-capability via
>> sysfs.
>
> Hi Jens,
>
> As I shared with you before, here are the 2 patches I staged in
> linux-next via linux-dm.git's 'for-next' (same as in the patchwork
> references above) -- these were staged just so we had linux-next
> coverage until you picked them up.
>
> Please pick these up for 4.8 -- the changes are straightforward, are
> required for DM's DAX support, and Dan Williams has also acked them:
>
> https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=6c00531c6affa42b165df73a1eac3289bc45f4c4
> https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=51d1a5abcd0e88cb4528f35643245ea59cf234f1
>
> Feel free to pick them up from patchwork or cherry-pick from
> linux-dm.git

Added for 4.8, thanks!

-- 
Jens Axboe

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web