Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1204167 > unrolled thread
| Started by | Rafal Mielniczuk <rafal.mielniczuk@citrix.com> |
|---|---|
| First post | 2015-08-10 13:10 +0200 |
| Last post | 2015-08-18 11:50 +0200 |
| Articles | 11 — 3 participants |
Back to article view | Back to linux.kernel
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
Re: [Xen-devel] [PATCH RFC v2 0/5] Multi-queue support for xen-blkfront and xen-blkback Rafal Mielniczuk <rafal.mielniczuk@citrix.com> - 2015-08-10 13:10 +0200
Re: [Xen-devel] [PATCH RFC v2 0/5] Multi-queue support for xen-blkfront and xen-blkback Bob Liu <bob.liu@oracle.com> - 2015-08-10 13:20 +0200
Re: [Xen-devel] [PATCH RFC v2 0/5] Multi-queue support for xen-blkfront and xen-blkback Jens Axboe <axboe@fb.com> - 2015-08-10 18:00 +0200
Re: [Xen-devel] [PATCH RFC v2 0/5] Multi-queue support for xen-blkfront and xen-blkback Bob Liu <bob.liu@oracle.com> - 2015-08-11 08:10 +0200
Re: [Xen-devel] [PATCH RFC v2 0/5] Multi-queue support for xen-blkfront and xen-blkback Rafal Mielniczuk <rafal.mielniczuk@citrix.com> - 2015-08-11 11:50 +0200
Re: [Xen-devel] [PATCH RFC v2 0/5] Multi-queue support for xen-blkfront and xen-blkback Jens Axboe <axboe@fb.com> - 2015-08-11 19:40 +0200
Re: [Xen-devel] [PATCH RFC v2 0/5] Multi-queue support for xen-blkfront and xen-blkback Bob Liu <bob.liu@oracle.com> - 2015-08-12 12:20 +0200
Re: [Xen-devel] [PATCH RFC v2 0/5] Multi-queue support for xen-blkfront and xen-blkback Rafal Mielniczuk <rafal.mielniczuk@citrix.com> - 2015-08-12 19:20 +0200
Re: [Xen-devel] [PATCH RFC v2 0/5] Multi-queue support for xen-blkfront and xen-blkback Bob Liu <bob.liu@oracle.com> - 2015-08-14 10:40 +0200
Re: [Xen-devel] [PATCH RFC v2 0/5] Multi-queue support for xen-blkfront and xen-blkback Rafal Mielniczuk <rafal.mielniczuk@citrix.com> - 2015-08-14 14:40 +0200
Re: [Xen-devel] [PATCH RFC v2 0/5] Multi-queue support for xen-blkfront and xen-blkback Rafal Mielniczuk <rafal.mielniczuk@citrix.com> - 2015-08-18 11:50 +0200
| From | Rafal Mielniczuk <rafal.mielniczuk@citrix.com> |
|---|---|
| Date | 2015-08-10 13:10 +0200 |
| Subject | Re: [Xen-devel] [PATCH RFC v2 0/5] Multi-queue support for xen-blkfront and xen-blkback |
| Message-ID | <pVT7Q-4oP-33@gated-at.bofh.it> |
On 01/07/15 04:03, Jens Axboe wrote:
> On 06/30/2015 08:21 AM, Marcus Granado wrote:
>> Hi,
>>
>> Our measurements for the multiqueue patch indicate a clear improvement
>> in iops when more queues are used.
>>
>> The measurements were obtained under the following conditions:
>>
>> - using blkback as the dom0 backend with the multiqueue patch applied to
>> a dom0 kernel 4.0 on 8 vcpus.
>>
>> - using a recent Ubuntu 15.04 kernel 3.19 with multiqueue frontend
>> applied to be used as a guest on 4 vcpus
>>
>> - using a micron RealSSD P320h as the underlying local storage on a Dell
>> PowerEdge R720 with 2 Xeon E5-2643 v2 cpus.
>>
>> - fio 2.2.7-22-g36870 as the generator of synthetic loads in the guest.
>> We used direct_io to skip caching in the guest and ran fio for 60s
>> reading a number of block sizes ranging from 512 bytes to 4MiB. Queue
>> depth of 32 for each queue was used to saturate individual vcpus in the
>> guest.
>>
>> We were interested in observing storage iops for different values of
>> block sizes. Our expectation was that iops would improve when increasing
>> the number of queues, because both the guest and dom0 would be able to
>> make use of more vcpus to handle these requests.
>>
>> These are the results (as aggregate iops for all the fio threads) that
>> we got for the conditions above with sequential reads:
>>
>> fio_threads io_depth block_size 1-queue_iops 8-queue_iops
>> 8 32 512 158K 264K
>> 8 32 1K 157K 260K
>> 8 32 2K 157K 258K
>> 8 32 4K 148K 257K
>> 8 32 8K 124K 207K
>> 8 32 16K 84K 105K
>> 8 32 32K 50K 54K
>> 8 32 64K 24K 27K
>> 8 32 128K 11K 13K
>>
>> 8-queue iops was better than single queue iops for all the block sizes.
>> There were very good improvements as well for sequential writes with
>> block size 4K (from 80K iops with single queue to 230K iops with 8
>> queues), and no regressions were visible in any measurement performed.
> Great results! And I don't know why this code has lingered for so long,
> so thanks for helping get some attention to this again.
>
> Personally I'd be really interested in the results for the same set of
> tests, but without the blk-mq patches. Do you have them, or could you
> potentially run them?
>
Hello,
We rerun the tests for sequential reads with the identical settings but with Bob Liu's multiqueue patches reverted from dom0 and guest kernels.
The results we obtained were *better* than the results we got with multiqueue patches applied:
fio_threads io_depth block_size 1-queue_iops 8-queue_iops *no-mq-patches_iops*
8 32 512 158K 264K 321K
8 32 1K 157K 260K 328K
8 32 2K 157K 258K 336K
8 32 4K 148K 257K 308K
8 32 8K 124K 207K 188K
8 32 16K 84K 105K 82K
8 32 32K 50K 54K 36K
8 32 64K 24K 27K 16K
8 32 128K 11K 13K 11K
We noticed that the requests are not merged by the guest when the multiqueue patches are applied,
which results in a regression for small block sizes (RealSSD P320h's optimal block size is around 32-64KB).
We observed similar regression for the Dell MZ-5EA1000-0D3 100 GB 2.5" Internal SSD
As I understand blk-mq layer bypasses I/O scheduler which also effectively disables merges.
Could you explain why it is difficult to enable merging in the blk-mq layer?
That could help closing the performance gap we observed.
Otherwise, the tests shows that the multiqueue patches does not improve the performance,
at least when it comes to sequential read/writes operations.
Rafal
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [next] | [standalone]
| From | Bob Liu <bob.liu@oracle.com> |
|---|---|
| Date | 2015-08-10 13:20 +0200 |
| Subject | Re: [Xen-devel] [PATCH RFC v2 0/5] Multi-queue support for xen-blkfront and xen-blkback |
| Message-ID | <pVThw-4Hc-11@gated-at.bofh.it> |
| In reply to | #1204167 |
On 08/10/2015 07:03 PM, Rafal Mielniczuk wrote: > On 01/07/15 04:03, Jens Axboe wrote: >> On 06/30/2015 08:21 AM, Marcus Granado wrote: >>> Hi, >>> >>> Our measurements for the multiqueue patch indicate a clear improvement >>> in iops when more queues are used. >>> >>> The measurements were obtained under the following conditions: >>> >>> - using blkback as the dom0 backend with the multiqueue patch applied to >>> a dom0 kernel 4.0 on 8 vcpus. >>> >>> - using a recent Ubuntu 15.04 kernel 3.19 with multiqueue frontend >>> applied to be used as a guest on 4 vcpus >>> >>> - using a micron RealSSD P320h as the underlying local storage on a Dell >>> PowerEdge R720 with 2 Xeon E5-2643 v2 cpus. >>> >>> - fio 2.2.7-22-g36870 as the generator of synthetic loads in the guest. >>> We used direct_io to skip caching in the guest and ran fio for 60s >>> reading a number of block sizes ranging from 512 bytes to 4MiB. Queue >>> depth of 32 for each queue was used to saturate individual vcpus in the >>> guest. >>> >>> We were interested in observing storage iops for different values of >>> block sizes. Our expectation was that iops would improve when increasing >>> the number of queues, because both the guest and dom0 would be able to >>> make use of more vcpus to handle these requests. >>> >>> These are the results (as aggregate iops for all the fio threads) that >>> we got for the conditions above with sequential reads: >>> >>> fio_threads io_depth block_size 1-queue_iops 8-queue_iops >>> 8 32 512 158K 264K >>> 8 32 1K 157K 260K >>> 8 32 2K 157K 258K >>> 8 32 4K 148K 257K >>> 8 32 8K 124K 207K >>> 8 32 16K 84K 105K >>> 8 32 32K 50K 54K >>> 8 32 64K 24K 27K >>> 8 32 128K 11K 13K >>> >>> 8-queue iops was better than single queue iops for all the block sizes. >>> There were very good improvements as well for sequential writes with >>> block size 4K (from 80K iops with single queue to 230K iops with 8 >>> queues), and no regressions were visible in any measurement performed. >> Great results! And I don't know why this code has lingered for so long, >> so thanks for helping get some attention to this again. >> >> Personally I'd be really interested in the results for the same set of >> tests, but without the blk-mq patches. Do you have them, or could you >> potentially run them? >> > Hello, > > We rerun the tests for sequential reads with the identical settings but with Bob Liu's multiqueue patches reverted from dom0 and guest kernels. > The results we obtained were *better* than the results we got with multiqueue patches applied: > > fio_threads io_depth block_size 1-queue_iops 8-queue_iops *no-mq-patches_iops* > 8 32 512 158K 264K 321K > 8 32 1K 157K 260K 328K > 8 32 2K 157K 258K 336K > 8 32 4K 148K 257K 308K > 8 32 8K 124K 207K 188K > 8 32 16K 84K 105K 82K > 8 32 32K 50K 54K 36K > 8 32 64K 24K 27K 16K > 8 32 128K 11K 13K 11K > > We noticed that the requests are not merged by the guest when the multiqueue patches are applied, > which results in a regression for small block sizes (RealSSD P320h's optimal block size is around 32-64KB). > > We observed similar regression for the Dell MZ-5EA1000-0D3 100 GB 2.5" Internal SSD > Which block scheduler was used in domU? Please try to "cat /sys/block/sdxxx/queue/scheduler". How about the result if using "noop" scheduler? Thanks, Bob Liu > As I understand blk-mq layer bypasses I/O scheduler which also effectively disables merges. > Could you explain why it is difficult to enable merging in the blk-mq layer? > That could help closing the performance gap we observed. > > Otherwise, the tests shows that the multiqueue patches does not improve the performance, > at least when it comes to sequential read/writes operations. > > Rafal > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Jens Axboe <axboe@fb.com> |
|---|---|
| Date | 2015-08-10 18:00 +0200 |
| Subject | Re: [Xen-devel] [PATCH RFC v2 0/5] Multi-queue support for xen-blkfront and xen-blkback |
| Message-ID | <pVXEv-2ob-37@gated-at.bofh.it> |
| In reply to | #1204167 |
On 08/10/2015 05:03 AM, Rafal Mielniczuk wrote: > On 01/07/15 04:03, Jens Axboe wrote: >> On 06/30/2015 08:21 AM, Marcus Granado wrote: >>> Hi, >>> >>> Our measurements for the multiqueue patch indicate a clear improvement >>> in iops when more queues are used. >>> >>> The measurements were obtained under the following conditions: >>> >>> - using blkback as the dom0 backend with the multiqueue patch applied to >>> a dom0 kernel 4.0 on 8 vcpus. >>> >>> - using a recent Ubuntu 15.04 kernel 3.19 with multiqueue frontend >>> applied to be used as a guest on 4 vcpus >>> >>> - using a micron RealSSD P320h as the underlying local storage on a Dell >>> PowerEdge R720 with 2 Xeon E5-2643 v2 cpus. >>> >>> - fio 2.2.7-22-g36870 as the generator of synthetic loads in the guest. >>> We used direct_io to skip caching in the guest and ran fio for 60s >>> reading a number of block sizes ranging from 512 bytes to 4MiB. Queue >>> depth of 32 for each queue was used to saturate individual vcpus in the >>> guest. >>> >>> We were interested in observing storage iops for different values of >>> block sizes. Our expectation was that iops would improve when increasing >>> the number of queues, because both the guest and dom0 would be able to >>> make use of more vcpus to handle these requests. >>> >>> These are the results (as aggregate iops for all the fio threads) that >>> we got for the conditions above with sequential reads: >>> >>> fio_threads io_depth block_size 1-queue_iops 8-queue_iops >>> 8 32 512 158K 264K >>> 8 32 1K 157K 260K >>> 8 32 2K 157K 258K >>> 8 32 4K 148K 257K >>> 8 32 8K 124K 207K >>> 8 32 16K 84K 105K >>> 8 32 32K 50K 54K >>> 8 32 64K 24K 27K >>> 8 32 128K 11K 13K >>> >>> 8-queue iops was better than single queue iops for all the block sizes. >>> There were very good improvements as well for sequential writes with >>> block size 4K (from 80K iops with single queue to 230K iops with 8 >>> queues), and no regressions were visible in any measurement performed. >> Great results! And I don't know why this code has lingered for so long, >> so thanks for helping get some attention to this again. >> >> Personally I'd be really interested in the results for the same set of >> tests, but without the blk-mq patches. Do you have them, or could you >> potentially run them? >> > Hello, > > We rerun the tests for sequential reads with the identical settings but with Bob Liu's multiqueue patches reverted from dom0 and guest kernels. > The results we obtained were *better* than the results we got with multiqueue patches applied: > > fio_threads io_depth block_size 1-queue_iops 8-queue_iops *no-mq-patches_iops* > 8 32 512 158K 264K 321K > 8 32 1K 157K 260K 328K > 8 32 2K 157K 258K 336K > 8 32 4K 148K 257K 308K > 8 32 8K 124K 207K 188K > 8 32 16K 84K 105K 82K > 8 32 32K 50K 54K 36K > 8 32 64K 24K 27K 16K > 8 32 128K 11K 13K 11K > > We noticed that the requests are not merged by the guest when the multiqueue patches are applied, > which results in a regression for small block sizes (RealSSD P320h's optimal block size is around 32-64KB). > > We observed similar regression for the Dell MZ-5EA1000-0D3 100 GB 2.5" Internal SSD > > As I understand blk-mq layer bypasses I/O scheduler which also effectively disables merges. > Could you explain why it is difficult to enable merging in the blk-mq layer? > That could help closing the performance gap we observed. > > Otherwise, the tests shows that the multiqueue patches does not improve the performance, > at least when it comes to sequential read/writes operations. blk-mq still provides merging, there should be no difference there. Does the xen patches set BLK_MQ_F_SHOULD_MERGE? -- Jens Axboe -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Bob Liu <bob.liu@oracle.com> |
|---|---|
| Date | 2015-08-11 08:10 +0200 |
| Subject | Re: [Xen-devel] [PATCH RFC v2 0/5] Multi-queue support for xen-blkfront and xen-blkback |
| Message-ID | <pWaV3-4VF-1@gated-at.bofh.it> |
| In reply to | #1204393 |
On 08/10/2015 11:52 PM, Jens Axboe wrote: > On 08/10/2015 05:03 AM, Rafal Mielniczuk wrote: >> On 01/07/15 04:03, Jens Axboe wrote: >>> On 06/30/2015 08:21 AM, Marcus Granado wrote: >>>> Hi, >>>> >>>> Our measurements for the multiqueue patch indicate a clear improvement >>>> in iops when more queues are used. >>>> >>>> The measurements were obtained under the following conditions: >>>> >>>> - using blkback as the dom0 backend with the multiqueue patch applied to >>>> a dom0 kernel 4.0 on 8 vcpus. >>>> >>>> - using a recent Ubuntu 15.04 kernel 3.19 with multiqueue frontend >>>> applied to be used as a guest on 4 vcpus >>>> >>>> - using a micron RealSSD P320h as the underlying local storage on a Dell >>>> PowerEdge R720 with 2 Xeon E5-2643 v2 cpus. >>>> >>>> - fio 2.2.7-22-g36870 as the generator of synthetic loads in the guest. >>>> We used direct_io to skip caching in the guest and ran fio for 60s >>>> reading a number of block sizes ranging from 512 bytes to 4MiB. Queue >>>> depth of 32 for each queue was used to saturate individual vcpus in the >>>> guest. >>>> >>>> We were interested in observing storage iops for different values of >>>> block sizes. Our expectation was that iops would improve when increasing >>>> the number of queues, because both the guest and dom0 would be able to >>>> make use of more vcpus to handle these requests. >>>> >>>> These are the results (as aggregate iops for all the fio threads) that >>>> we got for the conditions above with sequential reads: >>>> >>>> fio_threads io_depth block_size 1-queue_iops 8-queue_iops >>>> 8 32 512 158K 264K >>>> 8 32 1K 157K 260K >>>> 8 32 2K 157K 258K >>>> 8 32 4K 148K 257K >>>> 8 32 8K 124K 207K >>>> 8 32 16K 84K 105K >>>> 8 32 32K 50K 54K >>>> 8 32 64K 24K 27K >>>> 8 32 128K 11K 13K >>>> >>>> 8-queue iops was better than single queue iops for all the block sizes. >>>> There were very good improvements as well for sequential writes with >>>> block size 4K (from 80K iops with single queue to 230K iops with 8 >>>> queues), and no regressions were visible in any measurement performed. >>> Great results! And I don't know why this code has lingered for so long, >>> so thanks for helping get some attention to this again. >>> >>> Personally I'd be really interested in the results for the same set of >>> tests, but without the blk-mq patches. Do you have them, or could you >>> potentially run them? >>> >> Hello, >> >> We rerun the tests for sequential reads with the identical settings but with Bob Liu's multiqueue patches reverted from dom0 and guest kernels. >> The results we obtained were *better* than the results we got with multiqueue patches applied: >> >> fio_threads io_depth block_size 1-queue_iops 8-queue_iops *no-mq-patches_iops* >> 8 32 512 158K 264K 321K >> 8 32 1K 157K 260K 328K >> 8 32 2K 157K 258K 336K >> 8 32 4K 148K 257K 308K >> 8 32 8K 124K 207K 188K >> 8 32 16K 84K 105K 82K >> 8 32 32K 50K 54K 36K >> 8 32 64K 24K 27K 16K >> 8 32 128K 11K 13K 11K >> >> We noticed that the requests are not merged by the guest when the multiqueue patches are applied, >> which results in a regression for small block sizes (RealSSD P320h's optimal block size is around 32-64KB). >> >> We observed similar regression for the Dell MZ-5EA1000-0D3 100 GB 2.5" Internal SSD >> >> As I understand blk-mq layer bypasses I/O scheduler which also effectively disables merges. >> Could you explain why it is difficult to enable merging in the blk-mq layer? >> That could help closing the performance gap we observed. >> >> Otherwise, the tests shows that the multiqueue patches does not improve the performance, >> at least when it comes to sequential read/writes operations. > > blk-mq still provides merging, there should be no difference there. Does the xen patches set BLK_MQ_F_SHOULD_MERGE? > Yes. Is it possible that xen-blkfront driver dequeue requests too fast after we have multiple hardware queues? Because new requests don't have the chance merging with old requests which were already dequeued and issued. -- Regards, -Bob -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Rafal Mielniczuk <rafal.mielniczuk@citrix.com> |
|---|---|
| Date | 2015-08-11 11:50 +0200 |
| Message-ID | <pWem0-1jK-57@gated-at.bofh.it> |
| In reply to | #1204767 |
On 11/08/15 07:08, Bob Liu wrote:
> On 08/10/2015 11:52 PM, Jens Axboe wrote:
>> On 08/10/2015 05:03 AM, Rafal Mielniczuk wrote:
>>> On 01/07/15 04:03, Jens Axboe wrote:
>>>> On 06/30/2015 08:21 AM, Marcus Granado wrote:
>>>>> Hi,
>>>>>
>>>>> Our measurements for the multiqueue patch indicate a clear improvement
>>>>> in iops when more queues are used.
>>>>>
>>>>> The measurements were obtained under the following conditions:
>>>>>
>>>>> - using blkback as the dom0 backend with the multiqueue patch applied to
>>>>> a dom0 kernel 4.0 on 8 vcpus.
>>>>>
>>>>> - using a recent Ubuntu 15.04 kernel 3.19 with multiqueue frontend
>>>>> applied to be used as a guest on 4 vcpus
>>>>>
>>>>> - using a micron RealSSD P320h as the underlying local storage on a Dell
>>>>> PowerEdge R720 with 2 Xeon E5-2643 v2 cpus.
>>>>>
>>>>> - fio 2.2.7-22-g36870 as the generator of synthetic loads in the guest.
>>>>> We used direct_io to skip caching in the guest and ran fio for 60s
>>>>> reading a number of block sizes ranging from 512 bytes to 4MiB. Queue
>>>>> depth of 32 for each queue was used to saturate individual vcpus in the
>>>>> guest.
>>>>>
>>>>> We were interested in observing storage iops for different values of
>>>>> block sizes. Our expectation was that iops would improve when increasing
>>>>> the number of queues, because both the guest and dom0 would be able to
>>>>> make use of more vcpus to handle these requests.
>>>>>
>>>>> These are the results (as aggregate iops for all the fio threads) that
>>>>> we got for the conditions above with sequential reads:
>>>>>
>>>>> fio_threads io_depth block_size 1-queue_iops 8-queue_iops
>>>>> 8 32 512 158K 264K
>>>>> 8 32 1K 157K 260K
>>>>> 8 32 2K 157K 258K
>>>>> 8 32 4K 148K 257K
>>>>> 8 32 8K 124K 207K
>>>>> 8 32 16K 84K 105K
>>>>> 8 32 32K 50K 54K
>>>>> 8 32 64K 24K 27K
>>>>> 8 32 128K 11K 13K
>>>>>
>>>>> 8-queue iops was better than single queue iops for all the block sizes.
>>>>> There were very good improvements as well for sequential writes with
>>>>> block size 4K (from 80K iops with single queue to 230K iops with 8
>>>>> queues), and no regressions were visible in any measurement performed.
>>>> Great results! And I don't know why this code has lingered for so long,
>>>> so thanks for helping get some attention to this again.
>>>>
>>>> Personally I'd be really interested in the results for the same set of
>>>> tests, but without the blk-mq patches. Do you have them, or could you
>>>> potentially run them?
>>>>
>>> Hello,
>>>
>>> We rerun the tests for sequential reads with the identical settings but with Bob Liu's multiqueue patches reverted from dom0 and guest kernels.
>>> The results we obtained were *better* than the results we got with multiqueue patches applied:
>>>
>>> fio_threads io_depth block_size 1-queue_iops 8-queue_iops *no-mq-patches_iops*
>>> 8 32 512 158K 264K 321K
>>> 8 32 1K 157K 260K 328K
>>> 8 32 2K 157K 258K 336K
>>> 8 32 4K 148K 257K 308K
>>> 8 32 8K 124K 207K 188K
>>> 8 32 16K 84K 105K 82K
>>> 8 32 32K 50K 54K 36K
>>> 8 32 64K 24K 27K 16K
>>> 8 32 128K 11K 13K 11K
>>>
>>> We noticed that the requests are not merged by the guest when the multiqueue patches are applied,
>>> which results in a regression for small block sizes (RealSSD P320h's optimal block size is around 32-64KB).
>>>
>>> We observed similar regression for the Dell MZ-5EA1000-0D3 100 GB 2.5" Internal SSD
>>>
>>> As I understand blk-mq layer bypasses I/O scheduler which also effectively disables merges.
>>> Could you explain why it is difficult to enable merging in the blk-mq layer?
>>> That could help closing the performance gap we observed.
>>>
>>> Otherwise, the tests shows that the multiqueue patches does not improve the performance,
>>> at least when it comes to sequential read/writes operations.
>> blk-mq still provides merging, there should be no difference there. Does the xen patches set BLK_MQ_F_SHOULD_MERGE?
>>
> Yes.
> Is it possible that xen-blkfront driver dequeue requests too fast after we have multiple hardware queues?
> Because new requests don't have the chance merging with old requests which were already dequeued and issued.
>
For some reason we don't see merges even when we set multiqueue to 1.
Below are some stats from the guest system when doing sequential 4KB reads:
$ fio --name=test --ioengine=libaio --direct=1 --rw=read --numjobs=8
--iodepth=32 --time_based=1 --runtime=300 --bs=4KB
--filename=/dev/xvdb
$ iostat -xt 5 /dev/xvdb
avg-cpu: %user %nice %system %iowait %steal %idle
0.50 0.00 2.73 85.14 2.00 9.63
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdb 0.00 0.00 156926.00 0.00 627704.00 0.00
8.00 30.06 0.19 0.19 0.00 0.01 100.48
$ cat /sys/block/xvdb/queue/scheduler
none
$ cat /sys/block/xvdb/queue/nomerges
0
Relevant bits from the xenstore configuration on the dom0:
/local/domain/0/backend/vbd/2/51728/dev = "xvdb"
/local/domain/0/backend/vbd/2/51728/backend-kind = "vbd"
/local/domain/0/backend/vbd/2/51728/type = "phy"
/local/domain/0/backend/vbd/2/51728/multi-queue-max-queues = "1"
/local/domain/2/device/vbd/51728/multi-queue-num-queues = "1"
/local/domain/2/device/vbd/51728/ring-ref = "9"
/local/domain/2/device/vbd/51728/event-channel = "60"
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Jens Axboe <axboe@fb.com> |
|---|---|
| Date | 2015-08-11 19:40 +0200 |
| Subject | Re: [Xen-devel] [PATCH RFC v2 0/5] Multi-queue support for xen-blkfront and xen-blkback |
| Message-ID | <pWlGO-3z2-27@gated-at.bofh.it> |
| In reply to | #1204961 |
On 08/11/2015 03:45 AM, Rafal Mielniczuk wrote: > On 11/08/15 07:08, Bob Liu wrote: >> On 08/10/2015 11:52 PM, Jens Axboe wrote: >>> On 08/10/2015 05:03 AM, Rafal Mielniczuk wrote: >>>> On 01/07/15 04:03, Jens Axboe wrote: >>>>> On 06/30/2015 08:21 AM, Marcus Granado wrote: >>>>>> Hi, >>>>>> >>>>>> Our measurements for the multiqueue patch indicate a clear improvement >>>>>> in iops when more queues are used. >>>>>> >>>>>> The measurements were obtained under the following conditions: >>>>>> >>>>>> - using blkback as the dom0 backend with the multiqueue patch applied to >>>>>> a dom0 kernel 4.0 on 8 vcpus. >>>>>> >>>>>> - using a recent Ubuntu 15.04 kernel 3.19 with multiqueue frontend >>>>>> applied to be used as a guest on 4 vcpus >>>>>> >>>>>> - using a micron RealSSD P320h as the underlying local storage on a Dell >>>>>> PowerEdge R720 with 2 Xeon E5-2643 v2 cpus. >>>>>> >>>>>> - fio 2.2.7-22-g36870 as the generator of synthetic loads in the guest. >>>>>> We used direct_io to skip caching in the guest and ran fio for 60s >>>>>> reading a number of block sizes ranging from 512 bytes to 4MiB. Queue >>>>>> depth of 32 for each queue was used to saturate individual vcpus in the >>>>>> guest. >>>>>> >>>>>> We were interested in observing storage iops for different values of >>>>>> block sizes. Our expectation was that iops would improve when increasing >>>>>> the number of queues, because both the guest and dom0 would be able to >>>>>> make use of more vcpus to handle these requests. >>>>>> >>>>>> These are the results (as aggregate iops for all the fio threads) that >>>>>> we got for the conditions above with sequential reads: >>>>>> >>>>>> fio_threads io_depth block_size 1-queue_iops 8-queue_iops >>>>>> 8 32 512 158K 264K >>>>>> 8 32 1K 157K 260K >>>>>> 8 32 2K 157K 258K >>>>>> 8 32 4K 148K 257K >>>>>> 8 32 8K 124K 207K >>>>>> 8 32 16K 84K 105K >>>>>> 8 32 32K 50K 54K >>>>>> 8 32 64K 24K 27K >>>>>> 8 32 128K 11K 13K >>>>>> >>>>>> 8-queue iops was better than single queue iops for all the block sizes. >>>>>> There were very good improvements as well for sequential writes with >>>>>> block size 4K (from 80K iops with single queue to 230K iops with 8 >>>>>> queues), and no regressions were visible in any measurement performed. >>>>> Great results! And I don't know why this code has lingered for so long, >>>>> so thanks for helping get some attention to this again. >>>>> >>>>> Personally I'd be really interested in the results for the same set of >>>>> tests, but without the blk-mq patches. Do you have them, or could you >>>>> potentially run them? >>>>> >>>> Hello, >>>> >>>> We rerun the tests for sequential reads with the identical settings but with Bob Liu's multiqueue patches reverted from dom0 and guest kernels. >>>> The results we obtained were *better* than the results we got with multiqueue patches applied: >>>> >>>> fio_threads io_depth block_size 1-queue_iops 8-queue_iops *no-mq-patches_iops* >>>> 8 32 512 158K 264K 321K >>>> 8 32 1K 157K 260K 328K >>>> 8 32 2K 157K 258K 336K >>>> 8 32 4K 148K 257K 308K >>>> 8 32 8K 124K 207K 188K >>>> 8 32 16K 84K 105K 82K >>>> 8 32 32K 50K 54K 36K >>>> 8 32 64K 24K 27K 16K >>>> 8 32 128K 11K 13K 11K >>>> >>>> We noticed that the requests are not merged by the guest when the multiqueue patches are applied, >>>> which results in a regression for small block sizes (RealSSD P320h's optimal block size is around 32-64KB). >>>> >>>> We observed similar regression for the Dell MZ-5EA1000-0D3 100 GB 2.5" Internal SSD >>>> >>>> As I understand blk-mq layer bypasses I/O scheduler which also effectively disables merges. >>>> Could you explain why it is difficult to enable merging in the blk-mq layer? >>>> That could help closing the performance gap we observed. >>>> >>>> Otherwise, the tests shows that the multiqueue patches does not improve the performance, >>>> at least when it comes to sequential read/writes operations. >>> blk-mq still provides merging, there should be no difference there. Does the xen patches set BLK_MQ_F_SHOULD_MERGE? >>> >> Yes. >> Is it possible that xen-blkfront driver dequeue requests too fast after we have multiple hardware queues? >> Because new requests don't have the chance merging with old requests which were already dequeued and issued. >> > > For some reason we don't see merges even when we set multiqueue to 1. > Below are some stats from the guest system when doing sequential 4KB reads: > > $ fio --name=test --ioengine=libaio --direct=1 --rw=read --numjobs=8 > --iodepth=32 --time_based=1 --runtime=300 --bs=4KB > --filename=/dev/xvdb > > $ iostat -xt 5 /dev/xvdb > avg-cpu: %user %nice %system %iowait %steal %idle > 0.50 0.00 2.73 85.14 2.00 9.63 > > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > avgrq-sz avgqu-sz await r_await w_await svctm %util > xvdb 0.00 0.00 156926.00 0.00 627704.00 0.00 > 8.00 30.06 0.19 0.19 0.00 0.01 100.48 > > $ cat /sys/block/xvdb/queue/scheduler > none > > $ cat /sys/block/xvdb/queue/nomerges > 0 > > Relevant bits from the xenstore configuration on the dom0: > > /local/domain/0/backend/vbd/2/51728/dev = "xvdb" > /local/domain/0/backend/vbd/2/51728/backend-kind = "vbd" > /local/domain/0/backend/vbd/2/51728/type = "phy" > /local/domain/0/backend/vbd/2/51728/multi-queue-max-queues = "1" > > /local/domain/2/device/vbd/51728/multi-queue-num-queues = "1" > /local/domain/2/device/vbd/51728/ring-ref = "9" > /local/domain/2/device/vbd/51728/event-channel = "60" If you add --iodepth-batch=16 to that fio command line? Both mq and non-mq relies on plugging to get batching in the use case above, otherwise IO is dispatched immediately. O_DIRECT is immediate. I'd be more interested in seeing a test case with buffered IO of a file system on top of the xvdb device, if we're missing merging for that case, then that's a much bigger issue. -- Jens Axboe -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Bob Liu <bob.liu@oracle.com> |
|---|---|
| Date | 2015-08-12 12:20 +0200 |
| Subject | Re: [Xen-devel] [PATCH RFC v2 0/5] Multi-queue support for xen-blkfront and xen-blkback |
| Message-ID | <pWBiy-1oV-15@gated-at.bofh.it> |
| In reply to | #1205325 |
On 08/12/2015 01:32 AM, Jens Axboe wrote: > On 08/11/2015 03:45 AM, Rafal Mielniczuk wrote: >> On 11/08/15 07:08, Bob Liu wrote: >>> On 08/10/2015 11:52 PM, Jens Axboe wrote: >>>> On 08/10/2015 05:03 AM, Rafal Mielniczuk wrote: ... >>>>> Hello, >>>>> >>>>> We rerun the tests for sequential reads with the identical settings but with Bob Liu's multiqueue patches reverted from dom0 and guest kernels. >>>>> The results we obtained were *better* than the results we got with multiqueue patches applied: >>>>> >>>>> fio_threads io_depth block_size 1-queue_iops 8-queue_iops *no-mq-patches_iops* >>>>> 8 32 512 158K 264K 321K >>>>> 8 32 1K 157K 260K 328K >>>>> 8 32 2K 157K 258K 336K >>>>> 8 32 4K 148K 257K 308K >>>>> 8 32 8K 124K 207K 188K >>>>> 8 32 16K 84K 105K 82K >>>>> 8 32 32K 50K 54K 36K >>>>> 8 32 64K 24K 27K 16K >>>>> 8 32 128K 11K 13K 11K >>>>> >>>>> We noticed that the requests are not merged by the guest when the multiqueue patches are applied, >>>>> which results in a regression for small block sizes (RealSSD P320h's optimal block size is around 32-64KB). >>>>> >>>>> We observed similar regression for the Dell MZ-5EA1000-0D3 100 GB 2.5" Internal SSD >>>>> >>>>> As I understand blk-mq layer bypasses I/O scheduler which also effectively disables merges. >>>>> Could you explain why it is difficult to enable merging in the blk-mq layer? >>>>> That could help closing the performance gap we observed. >>>>> >>>>> Otherwise, the tests shows that the multiqueue patches does not improve the performance, >>>>> at least when it comes to sequential read/writes operations. >>>> blk-mq still provides merging, there should be no difference there. Does the xen patches set BLK_MQ_F_SHOULD_MERGE? >>>> >>> Yes. >>> Is it possible that xen-blkfront driver dequeue requests too fast after we have multiple hardware queues? >>> Because new requests don't have the chance merging with old requests which were already dequeued and issued. >>> >> >> For some reason we don't see merges even when we set multiqueue to 1. >> Below are some stats from the guest system when doing sequential 4KB reads: >> >> $ fio --name=test --ioengine=libaio --direct=1 --rw=read --numjobs=8 >> --iodepth=32 --time_based=1 --runtime=300 --bs=4KB >> --filename=/dev/xvdb >> >> $ iostat -xt 5 /dev/xvdb >> avg-cpu: %user %nice %system %iowait %steal %idle >> 0.50 0.00 2.73 85.14 2.00 9.63 >> >> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s >> avgrq-sz avgqu-sz await r_await w_await svctm %util >> xvdb 0.00 0.00 156926.00 0.00 627704.00 0.00 >> 8.00 30.06 0.19 0.19 0.00 0.01 100.48 >> >> $ cat /sys/block/xvdb/queue/scheduler >> none >> >> $ cat /sys/block/xvdb/queue/nomerges >> 0 >> >> Relevant bits from the xenstore configuration on the dom0: >> >> /local/domain/0/backend/vbd/2/51728/dev = "xvdb" >> /local/domain/0/backend/vbd/2/51728/backend-kind = "vbd" >> /local/domain/0/backend/vbd/2/51728/type = "phy" >> /local/domain/0/backend/vbd/2/51728/multi-queue-max-queues = "1" >> >> /local/domain/2/device/vbd/51728/multi-queue-num-queues = "1" >> /local/domain/2/device/vbd/51728/ring-ref = "9" >> /local/domain/2/device/vbd/51728/event-channel = "60" > > If you add --iodepth-batch=16 to that fio command line? Both mq and non-mq relies on plugging to get > batching in the use case above, otherwise IO is dispatched immediately. O_DIRECT is immediate. > I'd be more interested in seeing a test case with buffered IO of a file system on top of the xvdb device, > if we're missing merging for that case, then that's a much bigger issue. > I was using the null block driver for xen blk-mq test. There were not merges happen any more even after patch: https://lkml.org/lkml/2015/7/13/185 (Which just converted xen block driver to use blk-mq apis) Will try a file system soon. -- Regards, -Bob -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Rafal Mielniczuk <rafal.mielniczuk@citrix.com> |
|---|---|
| Date | 2015-08-12 19:20 +0200 |
| Message-ID | <pWHR0-2rD-5@gated-at.bofh.it> |
| In reply to | #1205884 |
On 12/08/15 11:17, Bob Liu wrote:
> On 08/12/2015 01:32 AM, Jens Axboe wrote:
>> On 08/11/2015 03:45 AM, Rafal Mielniczuk wrote:
>>> On 11/08/15 07:08, Bob Liu wrote:
>>>> On 08/10/2015 11:52 PM, Jens Axboe wrote:
>>>>> On 08/10/2015 05:03 AM, Rafal Mielniczuk wrote:
> ...
>>>>>> Hello,
>>>>>>
>>>>>> We rerun the tests for sequential reads with the identical settings but with Bob Liu's multiqueue patches reverted from dom0 and guest kernels.
>>>>>> The results we obtained were *better* than the results we got with multiqueue patches applied:
>>>>>>
>>>>>> fio_threads io_depth block_size 1-queue_iops 8-queue_iops *no-mq-patches_iops*
>>>>>> 8 32 512 158K 264K 321K
>>>>>> 8 32 1K 157K 260K 328K
>>>>>> 8 32 2K 157K 258K 336K
>>>>>> 8 32 4K 148K 257K 308K
>>>>>> 8 32 8K 124K 207K 188K
>>>>>> 8 32 16K 84K 105K 82K
>>>>>> 8 32 32K 50K 54K 36K
>>>>>> 8 32 64K 24K 27K 16K
>>>>>> 8 32 128K 11K 13K 11K
>>>>>>
>>>>>> We noticed that the requests are not merged by the guest when the multiqueue patches are applied,
>>>>>> which results in a regression for small block sizes (RealSSD P320h's optimal block size is around 32-64KB).
>>>>>>
>>>>>> We observed similar regression for the Dell MZ-5EA1000-0D3 100 GB 2.5" Internal SSD
>>>>>>
>>>>>> As I understand blk-mq layer bypasses I/O scheduler which also effectively disables merges.
>>>>>> Could you explain why it is difficult to enable merging in the blk-mq layer?
>>>>>> That could help closing the performance gap we observed.
>>>>>>
>>>>>> Otherwise, the tests shows that the multiqueue patches does not improve the performance,
>>>>>> at least when it comes to sequential read/writes operations.
>>>>> blk-mq still provides merging, there should be no difference there. Does the xen patches set BLK_MQ_F_SHOULD_MERGE?
>>>>>
>>>> Yes.
>>>> Is it possible that xen-blkfront driver dequeue requests too fast after we have multiple hardware queues?
>>>> Because new requests don't have the chance merging with old requests which were already dequeued and issued.
>>>>
>>> For some reason we don't see merges even when we set multiqueue to 1.
>>> Below are some stats from the guest system when doing sequential 4KB reads:
>>>
>>> $ fio --name=test --ioengine=libaio --direct=1 --rw=read --numjobs=8
>>> --iodepth=32 --time_based=1 --runtime=300 --bs=4KB
>>> --filename=/dev/xvdb
>>>
>>> $ iostat -xt 5 /dev/xvdb
>>> avg-cpu: %user %nice %system %iowait %steal %idle
>>> 0.50 0.00 2.73 85.14 2.00 9.63
>>>
>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
>>> xvdb 0.00 0.00 156926.00 0.00 627704.00 0.00
>>> 8.00 30.06 0.19 0.19 0.00 0.01 100.48
>>>
>>> $ cat /sys/block/xvdb/queue/scheduler
>>> none
>>>
>>> $ cat /sys/block/xvdb/queue/nomerges
>>> 0
>>>
>>> Relevant bits from the xenstore configuration on the dom0:
>>>
>>> /local/domain/0/backend/vbd/2/51728/dev = "xvdb"
>>> /local/domain/0/backend/vbd/2/51728/backend-kind = "vbd"
>>> /local/domain/0/backend/vbd/2/51728/type = "phy"
>>> /local/domain/0/backend/vbd/2/51728/multi-queue-max-queues = "1"
>>>
>>> /local/domain/2/device/vbd/51728/multi-queue-num-queues = "1"
>>> /local/domain/2/device/vbd/51728/ring-ref = "9"
>>> /local/domain/2/device/vbd/51728/event-channel = "60"
>> If you add --iodepth-batch=16 to that fio command line? Both mq and non-mq relies on plugging to get
>> batching in the use case above, otherwise IO is dispatched immediately. O_DIRECT is immediate.
>> I'd be more interested in seeing a test case with buffered IO of a file system on top of the xvdb device,
>> if we're missing merging for that case, then that's a much bigger issue.
>>
>
> I was using the null block driver for xen blk-mq test.
>
> There were not merges happen any more even after patch:
> https://lkml.org/lkml/2015/7/13/185
> (Which just converted xen block driver to use blk-mq apis)
>
> Will try a file system soon.
>
I have more results for the guest with and without the patch
https://lkml.org/lkml/2015/7/13/185
applied to the latest stable kernel (4.1.5).
Command line used was:
fio --name=test --ioengine=libaio --rw=read --numjobs=8 \
--iodepth=32 --time_based=1 --runtime=300 --bs=4KB \
--filename=/dev/xvdb --direct=(0 and 1) --iodepth_batch=16
without patch (--direct=1):
xvdb: ios=18696304/0, merge=75763177/0, ticks=11323872/0, in_queue=11344352, util=100.00%
with patch (--direct=1):
xvdb: ios=43709976/0, merge=97/0, ticks=8851972/0, in_queue=8902928, util=100.00%
without patch buffered (--direct=0):
xvdb: ios=1079051/0, merge=76/0, ticks=749364/0, in_queue=748840, util=94.60
with patch buffered (--direct=0):
xvdb: ios=1132932/0, merge=0/0, ticks=689108/0, in_queue=688488, util=93.32%
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Bob Liu <bob.liu@oracle.com> |
|---|---|
| Date | 2015-08-14 10:40 +0200 |
| Subject | Re: [Xen-devel] [PATCH RFC v2 0/5] Multi-queue support for xen-blkfront and xen-blkback |
| Message-ID | <pXiGS-5dY-19@gated-at.bofh.it> |
| In reply to | #1206286 |
On 08/13/2015 12:46 AM, Rafal Mielniczuk wrote: > On 12/08/15 11:17, Bob Liu wrote: >> On 08/12/2015 01:32 AM, Jens Axboe wrote: >>> On 08/11/2015 03:45 AM, Rafal Mielniczuk wrote: >>>> On 11/08/15 07:08, Bob Liu wrote: >>>>> On 08/10/2015 11:52 PM, Jens Axboe wrote: >>>>>> On 08/10/2015 05:03 AM, Rafal Mielniczuk wrote: >> ... >>>>>>> Hello, >>>>>>> >>>>>>> We rerun the tests for sequential reads with the identical settings but with Bob Liu's multiqueue patches reverted from dom0 and guest kernels. >>>>>>> The results we obtained were *better* than the results we got with multiqueue patches applied: >>>>>>> >>>>>>> fio_threads io_depth block_size 1-queue_iops 8-queue_iops *no-mq-patches_iops* >>>>>>> 8 32 512 158K 264K 321K >>>>>>> 8 32 1K 157K 260K 328K >>>>>>> 8 32 2K 157K 258K 336K >>>>>>> 8 32 4K 148K 257K 308K >>>>>>> 8 32 8K 124K 207K 188K >>>>>>> 8 32 16K 84K 105K 82K >>>>>>> 8 32 32K 50K 54K 36K >>>>>>> 8 32 64K 24K 27K 16K >>>>>>> 8 32 128K 11K 13K 11K >>>>>>> >>>>>>> We noticed that the requests are not merged by the guest when the multiqueue patches are applied, >>>>>>> which results in a regression for small block sizes (RealSSD P320h's optimal block size is around 32-64KB). >>>>>>> >>>>>>> We observed similar regression for the Dell MZ-5EA1000-0D3 100 GB 2.5" Internal SSD >>>>>>> >>>>>>> As I understand blk-mq layer bypasses I/O scheduler which also effectively disables merges. >>>>>>> Could you explain why it is difficult to enable merging in the blk-mq layer? >>>>>>> That could help closing the performance gap we observed. >>>>>>> >>>>>>> Otherwise, the tests shows that the multiqueue patches does not improve the performance, >>>>>>> at least when it comes to sequential read/writes operations. >>>>>> blk-mq still provides merging, there should be no difference there. Does the xen patches set BLK_MQ_F_SHOULD_MERGE? >>>>>> >>>>> Yes. >>>>> Is it possible that xen-blkfront driver dequeue requests too fast after we have multiple hardware queues? >>>>> Because new requests don't have the chance merging with old requests which were already dequeued and issued. >>>>> >>>> For some reason we don't see merges even when we set multiqueue to 1. >>>> Below are some stats from the guest system when doing sequential 4KB reads: >>>> >>>> $ fio --name=test --ioengine=libaio --direct=1 --rw=read --numjobs=8 >>>> --iodepth=32 --time_based=1 --runtime=300 --bs=4KB >>>> --filename=/dev/xvdb >>>> >>>> $ iostat -xt 5 /dev/xvdb >>>> avg-cpu: %user %nice %system %iowait %steal %idle >>>> 0.50 0.00 2.73 85.14 2.00 9.63 >>>> >>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s >>>> avgrq-sz avgqu-sz await r_await w_await svctm %util >>>> xvdb 0.00 0.00 156926.00 0.00 627704.00 0.00 >>>> 8.00 30.06 0.19 0.19 0.00 0.01 100.48 >>>> >>>> $ cat /sys/block/xvdb/queue/scheduler >>>> none >>>> >>>> $ cat /sys/block/xvdb/queue/nomerges >>>> 0 >>>> >>>> Relevant bits from the xenstore configuration on the dom0: >>>> >>>> /local/domain/0/backend/vbd/2/51728/dev = "xvdb" >>>> /local/domain/0/backend/vbd/2/51728/backend-kind = "vbd" >>>> /local/domain/0/backend/vbd/2/51728/type = "phy" >>>> /local/domain/0/backend/vbd/2/51728/multi-queue-max-queues = "1" >>>> >>>> /local/domain/2/device/vbd/51728/multi-queue-num-queues = "1" >>>> /local/domain/2/device/vbd/51728/ring-ref = "9" >>>> /local/domain/2/device/vbd/51728/event-channel = "60" >>> If you add --iodepth-batch=16 to that fio command line? Both mq and non-mq relies on plugging to get >>> batching in the use case above, otherwise IO is dispatched immediately. O_DIRECT is immediate. >>> I'd be more interested in seeing a test case with buffered IO of a file system on top of the xvdb device, >>> if we're missing merging for that case, then that's a much bigger issue. >>> >> >> I was using the null block driver for xen blk-mq test. >> >> There were not merges happen any more even after patch: >> https://lkml.org/lkml/2015/7/13/185 >> (Which just converted xen block driver to use blk-mq apis) >> >> Will try a file system soon. >> > I have more results for the guest with and without the patch > https://lkml.org/lkml/2015/7/13/185 > applied to the latest stable kernel (4.1.5). > Thank you. > Command line used was: > fio --name=test --ioengine=libaio --rw=read --numjobs=8 \ > --iodepth=32 --time_based=1 --runtime=300 --bs=4KB \ > --filename=/dev/xvdb --direct=(0 and 1) --iodepth_batch=16 > > without patch (--direct=1): > xvdb: ios=18696304/0, merge=75763177/0, ticks=11323872/0, in_queue=11344352, util=100.00% > > with patch (--direct=1): > xvdb: ios=43709976/0, merge=97/0, ticks=8851972/0, in_queue=8902928, util=100.00% > So request merge can happen just more difficult to be triggered. How about the iops of both cases? > without patch buffered (--direct=0): > xvdb: ios=1079051/0, merge=76/0, ticks=749364/0, in_queue=748840, util=94.60 > > with patch buffered (--direct=0): > xvdb: ios=1132932/0, merge=0/0, ticks=689108/0, in_queue=688488, util=93.32% > -- Regards, -Bob -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Rafal Mielniczuk <rafal.mielniczuk@citrix.com> |
|---|---|
| Date | 2015-08-14 14:40 +0200 |
| Message-ID | <pXmr8-28w-15@gated-at.bofh.it> |
| In reply to | #1207406 |
On 14/08/15 09:31, Bob Liu wrote:
> On 08/13/2015 12:46 AM, Rafal Mielniczuk wrote:
>> On 12/08/15 11:17, Bob Liu wrote:
>>> On 08/12/2015 01:32 AM, Jens Axboe wrote:
>>>> On 08/11/2015 03:45 AM, Rafal Mielniczuk wrote:
>>>>> On 11/08/15 07:08, Bob Liu wrote:
>>>>>> On 08/10/2015 11:52 PM, Jens Axboe wrote:
>>>>>>> On 08/10/2015 05:03 AM, Rafal Mielniczuk wrote:
>>> ...
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> We rerun the tests for sequential reads with the identical settings but with Bob Liu's multiqueue patches reverted from dom0 and guest kernels.
>>>>>>>> The results we obtained were *better* than the results we got with multiqueue patches applied:
>>>>>>>>
>>>>>>>> fio_threads io_depth block_size 1-queue_iops 8-queue_iops *no-mq-patches_iops*
>>>>>>>> 8 32 512 158K 264K 321K
>>>>>>>> 8 32 1K 157K 260K 328K
>>>>>>>> 8 32 2K 157K 258K 336K
>>>>>>>> 8 32 4K 148K 257K 308K
>>>>>>>> 8 32 8K 124K 207K 188K
>>>>>>>> 8 32 16K 84K 105K 82K
>>>>>>>> 8 32 32K 50K 54K 36K
>>>>>>>> 8 32 64K 24K 27K 16K
>>>>>>>> 8 32 128K 11K 13K 11K
>>>>>>>>
>>>>>>>> We noticed that the requests are not merged by the guest when the multiqueue patches are applied,
>>>>>>>> which results in a regression for small block sizes (RealSSD P320h's optimal block size is around 32-64KB).
>>>>>>>>
>>>>>>>> We observed similar regression for the Dell MZ-5EA1000-0D3 100 GB 2.5" Internal SSD
>>>>>>>>
>>>>>>>> As I understand blk-mq layer bypasses I/O scheduler which also effectively disables merges.
>>>>>>>> Could you explain why it is difficult to enable merging in the blk-mq layer?
>>>>>>>> That could help closing the performance gap we observed.
>>>>>>>>
>>>>>>>> Otherwise, the tests shows that the multiqueue patches does not improve the performance,
>>>>>>>> at least when it comes to sequential read/writes operations.
>>>>>>> blk-mq still provides merging, there should be no difference there. Does the xen patches set BLK_MQ_F_SHOULD_MERGE?
>>>>>>>
>>>>>> Yes.
>>>>>> Is it possible that xen-blkfront driver dequeue requests too fast after we have multiple hardware queues?
>>>>>> Because new requests don't have the chance merging with old requests which were already dequeued and issued.
>>>>>>
>>>>> For some reason we don't see merges even when we set multiqueue to 1.
>>>>> Below are some stats from the guest system when doing sequential 4KB reads:
>>>>>
>>>>> $ fio --name=test --ioengine=libaio --direct=1 --rw=read --numjobs=8
>>>>> --iodepth=32 --time_based=1 --runtime=300 --bs=4KB
>>>>> --filename=/dev/xvdb
>>>>>
>>>>> $ iostat -xt 5 /dev/xvdb
>>>>> avg-cpu: %user %nice %system %iowait %steal %idle
>>>>> 0.50 0.00 2.73 85.14 2.00 9.63
>>>>>
>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
>>>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
>>>>> xvdb 0.00 0.00 156926.00 0.00 627704.00 0.00
>>>>> 8.00 30.06 0.19 0.19 0.00 0.01 100.48
>>>>>
>>>>> $ cat /sys/block/xvdb/queue/scheduler
>>>>> none
>>>>>
>>>>> $ cat /sys/block/xvdb/queue/nomerges
>>>>> 0
>>>>>
>>>>> Relevant bits from the xenstore configuration on the dom0:
>>>>>
>>>>> /local/domain/0/backend/vbd/2/51728/dev = "xvdb"
>>>>> /local/domain/0/backend/vbd/2/51728/backend-kind = "vbd"
>>>>> /local/domain/0/backend/vbd/2/51728/type = "phy"
>>>>> /local/domain/0/backend/vbd/2/51728/multi-queue-max-queues = "1"
>>>>>
>>>>> /local/domain/2/device/vbd/51728/multi-queue-num-queues = "1"
>>>>> /local/domain/2/device/vbd/51728/ring-ref = "9"
>>>>> /local/domain/2/device/vbd/51728/event-channel = "60"
>>>> If you add --iodepth-batch=16 to that fio command line? Both mq and non-mq relies on plugging to get
>>>> batching in the use case above, otherwise IO is dispatched immediately. O_DIRECT is immediate.
>>>> I'd be more interested in seeing a test case with buffered IO of a file system on top of the xvdb device,
>>>> if we're missing merging for that case, then that's a much bigger issue.
>>>>
>>>
>>> I was using the null block driver for xen blk-mq test.
>>>
>>> There were not merges happen any more even after patch:
>>> https://lkml.org/lkml/2015/7/13/185
>>> (Which just converted xen block driver to use blk-mq apis)
>>>
>>> Will try a file system soon.
>>>
>> I have more results for the guest with and without the patch
>> https://lkml.org/lkml/2015/7/13/185
>> applied to the latest stable kernel (4.1.5).
>>
> Thank you.
>
>> Command line used was:
>> fio --name=test --ioengine=libaio --rw=read --numjobs=8 \
>> --iodepth=32 --time_based=1 --runtime=300 --bs=4KB \
>> --filename=/dev/xvdb --direct=(0 and 1) --iodepth_batch=16
>>
>> without patch (--direct=1):
>> xvdb: ios=18696304/0, merge=75763177/0, ticks=11323872/0, in_queue=11344352, util=100.00%
>>
>> with patch (--direct=1):
>> xvdb: ios=43709976/0, merge=97/0, ticks=8851972/0, in_queue=8902928, util=100.00%
>>
> So request merge can happen just more difficult to be triggered.
> How about the iops of both cases?
Without the patch it is 318Kiops, with the patch 146Kiops
>> without patch buffered (--direct=0):
>> xvdb: ios=1079051/0, merge=76/0, ticks=749364/0, in_queue=748840, util=94.60
>>
>> with patch buffered (--direct=0):
>> xvdb: ios=1132932/0, merge=0/0, ticks=689108/0, in_queue=688488, util=93.32%
>>
There seems to be very little difference when we measure buffered
sequential reads.
Although iostat shows that there are almost no merges happening for both
cases,
the avgrq-sz is around 250 sectors (125KB). Does that mean that the
merges are actually happening
but on some other layer, not visible to the iostat?
There is a big discrepancy for direct sequential reads and small block
sizes,
where we are missing merges that were happening in the version before
the patch.
It looks like the request does not reside in the queue for long enough
to get merged.
One thing I noticed is that in block/blk-mq.c in function
bool blk_mq_attempt_merge(struct request_queue *q,
struct blk_mq_ctx *ctx, struct bio *bio)
The ctx->rq_list queue is mostly empty, the for loop inside the body
of the function is almost never executed.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Rafal Mielniczuk <rafal.mielniczuk@citrix.com> |
|---|---|
| Date | 2015-08-18 11:50 +0200 |
| Message-ID | <pYLGO-2oQ-19@gated-at.bofh.it> |
| In reply to | #1207514 |
On 14/08/15 13:30, Rafal Mielniczuk wrote:
> On 14/08/15 09:31, Bob Liu wrote:
>> On 08/13/2015 12:46 AM, Rafal Mielniczuk wrote:
>>> On 12/08/15 11:17, Bob Liu wrote:
>>>> On 08/12/2015 01:32 AM, Jens Axboe wrote:
>>>>> On 08/11/2015 03:45 AM, Rafal Mielniczuk wrote:
>>>>>> On 11/08/15 07:08, Bob Liu wrote:
>>>>>>> On 08/10/2015 11:52 PM, Jens Axboe wrote:
>>>>>>>> On 08/10/2015 05:03 AM, Rafal Mielniczuk wrote:
>>>> ...
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> We rerun the tests for sequential reads with the identical settings but with Bob Liu's multiqueue patches reverted from dom0 and guest kernels.
>>>>>>>>> The results we obtained were *better* than the results we got with multiqueue patches applied:
>>>>>>>>>
>>>>>>>>> fio_threads io_depth block_size 1-queue_iops 8-queue_iops *no-mq-patches_iops*
>>>>>>>>> 8 32 512 158K 264K 321K
>>>>>>>>> 8 32 1K 157K 260K 328K
>>>>>>>>> 8 32 2K 157K 258K 336K
>>>>>>>>> 8 32 4K 148K 257K 308K
>>>>>>>>> 8 32 8K 124K 207K 188K
>>>>>>>>> 8 32 16K 84K 105K 82K
>>>>>>>>> 8 32 32K 50K 54K 36K
>>>>>>>>> 8 32 64K 24K 27K 16K
>>>>>>>>> 8 32 128K 11K 13K 11K
>>>>>>>>>
>>>>>>>>> We noticed that the requests are not merged by the guest when the multiqueue patches are applied,
>>>>>>>>> which results in a regression for small block sizes (RealSSD P320h's optimal block size is around 32-64KB).
>>>>>>>>>
>>>>>>>>> We observed similar regression for the Dell MZ-5EA1000-0D3 100 GB 2.5" Internal SSD
>>>>>>>>>
>>>>>>>>> As I understand blk-mq layer bypasses I/O scheduler which also effectively disables merges.
>>>>>>>>> Could you explain why it is difficult to enable merging in the blk-mq layer?
>>>>>>>>> That could help closing the performance gap we observed.
>>>>>>>>>
>>>>>>>>> Otherwise, the tests shows that the multiqueue patches does not improve the performance,
>>>>>>>>> at least when it comes to sequential read/writes operations.
>>>>>>>> blk-mq still provides merging, there should be no difference there. Does the xen patches set BLK_MQ_F_SHOULD_MERGE?
>>>>>>>>
>>>>>>> Yes.
>>>>>>> Is it possible that xen-blkfront driver dequeue requests too fast after we have multiple hardware queues?
>>>>>>> Because new requests don't have the chance merging with old requests which were already dequeued and issued.
>>>>>>>
>>>>>> For some reason we don't see merges even when we set multiqueue to 1.
>>>>>> Below are some stats from the guest system when doing sequential 4KB reads:
>>>>>>
>>>>>> $ fio --name=test --ioengine=libaio --direct=1 --rw=read --numjobs=8
>>>>>> --iodepth=32 --time_based=1 --runtime=300 --bs=4KB
>>>>>> --filename=/dev/xvdb
>>>>>>
>>>>>> $ iostat -xt 5 /dev/xvdb
>>>>>> avg-cpu: %user %nice %system %iowait %steal %idle
>>>>>> 0.50 0.00 2.73 85.14 2.00 9.63
>>>>>>
>>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
>>>>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
>>>>>> xvdb 0.00 0.00 156926.00 0.00 627704.00 0.00
>>>>>> 8.00 30.06 0.19 0.19 0.00 0.01 100.48
>>>>>>
>>>>>> $ cat /sys/block/xvdb/queue/scheduler
>>>>>> none
>>>>>>
>>>>>> $ cat /sys/block/xvdb/queue/nomerges
>>>>>> 0
>>>>>>
>>>>>> Relevant bits from the xenstore configuration on the dom0:
>>>>>>
>>>>>> /local/domain/0/backend/vbd/2/51728/dev = "xvdb"
>>>>>> /local/domain/0/backend/vbd/2/51728/backend-kind = "vbd"
>>>>>> /local/domain/0/backend/vbd/2/51728/type = "phy"
>>>>>> /local/domain/0/backend/vbd/2/51728/multi-queue-max-queues = "1"
>>>>>>
>>>>>> /local/domain/2/device/vbd/51728/multi-queue-num-queues = "1"
>>>>>> /local/domain/2/device/vbd/51728/ring-ref = "9"
>>>>>> /local/domain/2/device/vbd/51728/event-channel = "60"
>>>>> If you add --iodepth-batch=16 to that fio command line? Both mq and non-mq relies on plugging to get
>>>>> batching in the use case above, otherwise IO is dispatched immediately. O_DIRECT is immediate.
>>>>> I'd be more interested in seeing a test case with buffered IO of a file system on top of the xvdb device,
>>>>> if we're missing merging for that case, then that's a much bigger issue.
>>>>>
>>>>
>>>> I was using the null block driver for xen blk-mq test.
>>>>
>>>> There were not merges happen any more even after patch:
>>>> https://lkml.org/lkml/2015/7/13/185
>>>> (Which just converted xen block driver to use blk-mq apis)
>>>>
>>>> Will try a file system soon.
>>>>
>>> I have more results for the guest with and without the patch
>>> https://lkml.org/lkml/2015/7/13/185
>>> applied to the latest stable kernel (4.1.5).
>>>
>> Thank you.
>>
>>> Command line used was:
>>> fio --name=test --ioengine=libaio --rw=read --numjobs=8 \
>>> --iodepth=32 --time_based=1 --runtime=300 --bs=4KB \
>>> --filename=/dev/xvdb --direct=(0 and 1) --iodepth_batch=16
>>>
>>> without patch (--direct=1):
>>> xvdb: ios=18696304/0, merge=75763177/0, ticks=11323872/0, in_queue=11344352, util=100.00%
>>>
>>> with patch (--direct=1):
>>> xvdb: ios=43709976/0, merge=97/0, ticks=8851972/0, in_queue=8902928, util=100.00%
>>>
>> So request merge can happen just more difficult to be triggered.
>> How about the iops of both cases?
> Without the patch it is 318Kiops, with the patch 146Kiops
>
>>> without patch buffered (--direct=0):
>>> xvdb: ios=1079051/0, merge=76/0, ticks=749364/0, in_queue=748840, util=94.60
>>>
>>> with patch buffered (--direct=0):
>>> xvdb: ios=1132932/0, merge=0/0, ticks=689108/0, in_queue=688488, util=93.32%
>>>
> There seems to be very little difference when we measure buffered
> sequential reads.
> Although iostat shows that there are almost no merges happening for both
> cases,
> the avgrq-sz is around 250 sectors (125KB). Does that mean that the
> merges are actually happening
> but on some other layer, not visible to the iostat?
>
> There is a big discrepancy for direct sequential reads and small block
> sizes,
> where we are missing merges that were happening in the version before
> the patch.
> It looks like the request does not reside in the queue for long enough
> to get merged.
>
> One thing I noticed is that in block/blk-mq.c in function
>
> bool blk_mq_attempt_merge(struct request_queue *q,
> struct blk_mq_ctx *ctx, struct bio *bio)
>
> The ctx->rq_list queue is mostly empty, the for loop inside the body
> of the function is almost never executed.
>
Hi,
I was able to reproduce Bob's results with nullblk device with default module parameters.
Also, when I increased the completion time of the requests,
I could see merges happening in the version without the patch, which resulted in greater throughput.
Could it be because request had time to accumulate in the queue and had a chance to be merged?
Why merges did not happen in the version with the patch then? Is the patch missing plugging Jens mentioned,
or is it a problem in blk-mq itself?
fio --name=test --ioengine=libaio --rw=read --numjobs=8 --iodepth=32 \
--time_based=1 --runtime=30 --bs=4KB --filename=/dev/xvdb \
--direct=1 --group_reporting=1 --iodepth_batch=16
========================================================================
modprobe null_blk
========================================================================
------------------------------------------------------------------------
*no patch* (avgrq-sz = 8.00 avgqu-sz=5.00)
------------------------------------------------------------------------
READ: io=10655MB, aggrb=363694KB/s, minb=363694KB/s, maxb=363694KB/s, mint=30001msec, maxt=30001msec
Disk stats (read/write):
xvdb: ios=2715852/0, merge=1089/0, ticks=126572/0, in_queue=127456, util=100.00%
------------------------------------------------------------------------
*with patch* (avgrq-sz = 8.00 avgqu-sz=8.00)
------------------------------------------------------------------------
READ: io=20655MB, aggrb=705010KB/s, minb=705010KB/s, maxb=705010KB/s, mint=30001msec, maxt=30001msec
Disk stats (read/write):
xvdb: ios=5274633/0, merge=22/0, ticks=243208/0, in_queue=242908, util=99.98%
========================================================================
modprobe null_blk irqmode=2 completion_nsec=1000000
========================================================================
------------------------------------------------------------------------
*no patch* (avgrq-sz = 34.00 avgqu-sz=38.00)
------------------------------------------------------------------------
READ: io=10372MB, aggrb=354008KB/s, minb=354008KB/s, maxb=354008KB/s, mint=30003msec, maxt=30003msec
Disk stats (read/write):
xvdb: ios=621760/0, *merge=1988170/0*, ticks=1136700/0, in_queue=1146020, util=99.76%
------------------------------------------------------------------------
*with patch* (avgrq-sz = 8.00 avgqu-sz=28.00)
------------------------------------------------------------------------
READ: io=2876.8MB, aggrb=98187KB/s, minb=98187KB/s, maxb=98187KB/s, mint=30002msec, maxt=30002msec
Disk stats (read/write):
xvdb: ios=734048/0, merge=0/0, ticks=843584/0, in_queue=843080, util=99.72%
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web