Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1706456 > unrolled thread

Re: [PATCH v2 0/4] Optimise 64-bit IOVA allocations

Started byGanapatrao Kulkarni <gklkml16@gmail.com>
First post2017-08-08 14:10 +0200
Last post2017-08-09 06:30 +0200
Articles 4 — 2 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH v2 0/4] Optimise 64-bit IOVA allocations Ganapatrao Kulkarni <gklkml16@gmail.com> - 2017-08-08 14:10 +0200
    Re: [PATCH v2 0/4] Optimise 64-bit IOVA allocations "Leizhen (ThunderTown)" <thunder.leizhen@huawei.com> - 2017-08-09 03:50 +0200
      Re: [PATCH v2 0/4] Optimise 64-bit IOVA allocations Ganapatrao Kulkarni <gklkml16@gmail.com> - 2017-08-09 05:30 +0200
        Re: [PATCH v2 0/4] Optimise 64-bit IOVA allocations "Leizhen (ThunderTown)" <thunder.leizhen@huawei.com> - 2017-08-09 06:30 +0200

#1706456 — Re: [PATCH v2 0/4] Optimise 64-bit IOVA allocations

FromGanapatrao Kulkarni <gklkml16@gmail.com>
Date2017-08-08 14:10 +0200
SubjectRe: [PATCH v2 0/4] Optimise 64-bit IOVA allocations
Message-ID<ucbuH-8cT-49@gated-at.bofh.it>
On Wed, Jul 26, 2017 at 4:47 PM, Leizhen (ThunderTown)
<thunder.leizhen@huawei.com> wrote:
>
>
> On 2017/7/26 19:08, Joerg Roedel wrote:
>> Hi Robin.
>>
>> On Fri, Jul 21, 2017 at 12:41:57PM +0100, Robin Murphy wrote:
>>> Hi all,
>>>
>>> In the wake of the ARM SMMU optimisation efforts, it seems that certain
>>> workloads (e.g. storage I/O with large scatterlists) probably remain quite
>>> heavily influenced by IOVA allocation performance. Separately, Ard also
>>> reported massive performance drops for a graphical desktop on AMD Seattle
>>> when enabling SMMUs via IORT, which we traced to dma_32bit_pfn in the DMA
>>> ops domain getting initialised differently for ACPI vs. DT, and exposing
>>> the overhead of the rbtree slow path. Whilst we could go around trying to
>>> close up all the little gaps that lead to hitting the slowest case, it
>>> seems a much better idea to simply make said slowest case a lot less slow.
>>
>> Do you have some numbers here? How big was the impact before these
>> patches and how is it with the patches?
> Here are some numbers:
>
> (before)$ iperf -s
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> [  4] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 35898
> [ ID] Interval       Transfer     Bandwidth
> [  4]  0.0-10.2 sec  7.88 MBytes  6.48 Mbits/sec
> [  5] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 35900
> [  5]  0.0-10.3 sec  7.88 MBytes  6.43 Mbits/sec
> [  4] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 35902
> [  4]  0.0-10.3 sec  7.88 MBytes  6.43 Mbits/sec
>
> (after)$ iperf -s
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> [  4] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 36330
> [ ID] Interval       Transfer     Bandwidth
> [  4]  0.0-10.0 sec  1.09 GBytes   933 Mbits/sec
> [  5] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 36332
> [  5]  0.0-10.0 sec  1.10 GBytes   939 Mbits/sec
> [  4] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 36334
> [  4]  0.0-10.0 sec  1.10 GBytes   938 Mbits/sec
>

Is this testing done on Host or on Guest/VM?

>>
>>
>>       Joerg
>>
>>
>> .
>>
>
> --
> Thanks!
> BestRegards
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

thanks
Ganapat

[toc] | [next] | [standalone]

#1707000

From"Leizhen (ThunderTown)" <thunder.leizhen@huawei.com>
Date2017-08-09 03:50 +0200
Message-ID<ucoie-8qy-15@gated-at.bofh.it>
In reply to#1706456

On 2017/8/8 20:03, Ganapatrao Kulkarni wrote:
> On Wed, Jul 26, 2017 at 4:47 PM, Leizhen (ThunderTown)
> <thunder.leizhen@huawei.com> wrote:
>>
>>
>> On 2017/7/26 19:08, Joerg Roedel wrote:
>>> Hi Robin.
>>>
>>> On Fri, Jul 21, 2017 at 12:41:57PM +0100, Robin Murphy wrote:
>>>> Hi all,
>>>>
>>>> In the wake of the ARM SMMU optimisation efforts, it seems that certain
>>>> workloads (e.g. storage I/O with large scatterlists) probably remain quite
>>>> heavily influenced by IOVA allocation performance. Separately, Ard also
>>>> reported massive performance drops for a graphical desktop on AMD Seattle
>>>> when enabling SMMUs via IORT, which we traced to dma_32bit_pfn in the DMA
>>>> ops domain getting initialised differently for ACPI vs. DT, and exposing
>>>> the overhead of the rbtree slow path. Whilst we could go around trying to
>>>> close up all the little gaps that lead to hitting the slowest case, it
>>>> seems a much better idea to simply make said slowest case a lot less slow.
>>>
>>> Do you have some numbers here? How big was the impact before these
>>> patches and how is it with the patches?
>> Here are some numbers:
>>
>> (before)$ iperf -s
>> ------------------------------------------------------------
>> Server listening on TCP port 5001
>> TCP window size: 85.3 KByte (default)
>> ------------------------------------------------------------
>> [  4] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 35898
>> [ ID] Interval       Transfer     Bandwidth
>> [  4]  0.0-10.2 sec  7.88 MBytes  6.48 Mbits/sec
>> [  5] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 35900
>> [  5]  0.0-10.3 sec  7.88 MBytes  6.43 Mbits/sec
>> [  4] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 35902
>> [  4]  0.0-10.3 sec  7.88 MBytes  6.43 Mbits/sec
>>
>> (after)$ iperf -s
>> ------------------------------------------------------------
>> Server listening on TCP port 5001
>> TCP window size: 85.3 KByte (default)
>> ------------------------------------------------------------
>> [  4] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 36330
>> [ ID] Interval       Transfer     Bandwidth
>> [  4]  0.0-10.0 sec  1.09 GBytes   933 Mbits/sec
>> [  5] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 36332
>> [  5]  0.0-10.0 sec  1.10 GBytes   939 Mbits/sec
>> [  4] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 36334
>> [  4]  0.0-10.0 sec  1.10 GBytes   938 Mbits/sec
>>
> 
> Is this testing done on Host or on Guest/VM?
Host

> 
>>>
>>>
>>>       Joerg
>>>
>>>
>>> .
>>>
>>
>> --
>> Thanks!
>> BestRegards
>>
>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> linux-arm-kernel@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 
> thanks
> Ganapat
> 
> .
> 

-- 
Thanks!
BestRegards

[toc] | [prev] | [next] | [standalone]

#1707034

FromGanapatrao Kulkarni <gklkml16@gmail.com>
Date2017-08-09 05:30 +0200
Message-ID<ucpQZ-1d8-7@gated-at.bofh.it>
In reply to#1707000
On Wed, Aug 9, 2017 at 7:12 AM, Leizhen (ThunderTown)
<thunder.leizhen@huawei.com> wrote:
>
>
> On 2017/8/8 20:03, Ganapatrao Kulkarni wrote:
>> On Wed, Jul 26, 2017 at 4:47 PM, Leizhen (ThunderTown)
>> <thunder.leizhen@huawei.com> wrote:
>>>
>>>
>>> On 2017/7/26 19:08, Joerg Roedel wrote:
>>>> Hi Robin.
>>>>
>>>> On Fri, Jul 21, 2017 at 12:41:57PM +0100, Robin Murphy wrote:
>>>>> Hi all,
>>>>>
>>>>> In the wake of the ARM SMMU optimisation efforts, it seems that certain
>>>>> workloads (e.g. storage I/O with large scatterlists) probably remain quite
>>>>> heavily influenced by IOVA allocation performance. Separately, Ard also
>>>>> reported massive performance drops for a graphical desktop on AMD Seattle
>>>>> when enabling SMMUs via IORT, which we traced to dma_32bit_pfn in the DMA
>>>>> ops domain getting initialised differently for ACPI vs. DT, and exposing
>>>>> the overhead of the rbtree slow path. Whilst we could go around trying to
>>>>> close up all the little gaps that lead to hitting the slowest case, it
>>>>> seems a much better idea to simply make said slowest case a lot less slow.
>>>>
>>>> Do you have some numbers here? How big was the impact before these
>>>> patches and how is it with the patches?
>>> Here are some numbers:
>>>
>>> (before)$ iperf -s
>>> ------------------------------------------------------------
>>> Server listening on TCP port 5001
>>> TCP window size: 85.3 KByte (default)
>>> ------------------------------------------------------------
>>> [  4] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 35898
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  4]  0.0-10.2 sec  7.88 MBytes  6.48 Mbits/sec
>>> [  5] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 35900
>>> [  5]  0.0-10.3 sec  7.88 MBytes  6.43 Mbits/sec
>>> [  4] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 35902
>>> [  4]  0.0-10.3 sec  7.88 MBytes  6.43 Mbits/sec
>>>
>>> (after)$ iperf -s
>>> ------------------------------------------------------------
>>> Server listening on TCP port 5001
>>> TCP window size: 85.3 KByte (default)
>>> ------------------------------------------------------------
>>> [  4] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 36330
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  4]  0.0-10.0 sec  1.09 GBytes   933 Mbits/sec
>>> [  5] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 36332
>>> [  5]  0.0-10.0 sec  1.10 GBytes   939 Mbits/sec
>>> [  4] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 36334
>>> [  4]  0.0-10.0 sec  1.10 GBytes   938 Mbits/sec
>>>
>>
>> Is this testing done on Host or on Guest/VM?
> Host

As per your log, iperf throughput is improved to 938 Mbits/sec
from  6.43 Mbits/sec.
IMO, this seems to be unrealistic, some thing wrong with the testing?

>
>>
>>>>
>>>>
>>>>       Joerg
>>>>
>>>>
>>>> .
>>>>
>>>
>>> --
>>> Thanks!
>>> BestRegards
>>>
>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>
>> thanks
>> Ganapat
>>
>> .
>>
>
> --
> Thanks!
> BestRegards
>

thanks
Ganapat

[toc] | [prev] | [next] | [standalone]

#1707051

From"Leizhen (ThunderTown)" <thunder.leizhen@huawei.com>
Date2017-08-09 06:30 +0200
Message-ID<ucqN3-1P4-7@gated-at.bofh.it>
In reply to#1707034

On 2017/8/9 11:24, Ganapatrao Kulkarni wrote:
> On Wed, Aug 9, 2017 at 7:12 AM, Leizhen (ThunderTown)
> <thunder.leizhen@huawei.com> wrote:
>>
>>
>> On 2017/8/8 20:03, Ganapatrao Kulkarni wrote:
>>> On Wed, Jul 26, 2017 at 4:47 PM, Leizhen (ThunderTown)
>>> <thunder.leizhen@huawei.com> wrote:
>>>>
>>>>
>>>> On 2017/7/26 19:08, Joerg Roedel wrote:
>>>>> Hi Robin.
>>>>>
>>>>> On Fri, Jul 21, 2017 at 12:41:57PM +0100, Robin Murphy wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> In the wake of the ARM SMMU optimisation efforts, it seems that certain
>>>>>> workloads (e.g. storage I/O with large scatterlists) probably remain quite
>>>>>> heavily influenced by IOVA allocation performance. Separately, Ard also
>>>>>> reported massive performance drops for a graphical desktop on AMD Seattle
>>>>>> when enabling SMMUs via IORT, which we traced to dma_32bit_pfn in the DMA
>>>>>> ops domain getting initialised differently for ACPI vs. DT, and exposing
>>>>>> the overhead of the rbtree slow path. Whilst we could go around trying to
>>>>>> close up all the little gaps that lead to hitting the slowest case, it
>>>>>> seems a much better idea to simply make said slowest case a lot less slow.
>>>>>
>>>>> Do you have some numbers here? How big was the impact before these
>>>>> patches and how is it with the patches?
>>>> Here are some numbers:
>>>>
>>>> (before)$ iperf -s
>>>> ------------------------------------------------------------
>>>> Server listening on TCP port 5001
>>>> TCP window size: 85.3 KByte (default)
>>>> ------------------------------------------------------------
>>>> [  4] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 35898
>>>> [ ID] Interval       Transfer     Bandwidth
>>>> [  4]  0.0-10.2 sec  7.88 MBytes  6.48 Mbits/sec
>>>> [  5] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 35900
>>>> [  5]  0.0-10.3 sec  7.88 MBytes  6.43 Mbits/sec
>>>> [  4] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 35902
>>>> [  4]  0.0-10.3 sec  7.88 MBytes  6.43 Mbits/sec
>>>>
>>>> (after)$ iperf -s
>>>> ------------------------------------------------------------
>>>> Server listening on TCP port 5001
>>>> TCP window size: 85.3 KByte (default)
>>>> ------------------------------------------------------------
>>>> [  4] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 36330
>>>> [ ID] Interval       Transfer     Bandwidth
>>>> [  4]  0.0-10.0 sec  1.09 GBytes   933 Mbits/sec
>>>> [  5] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 36332
>>>> [  5]  0.0-10.0 sec  1.10 GBytes   939 Mbits/sec
>>>> [  4] local 192.168.1.106 port 5001 connected with 192.168.1.198 port 36334
>>>> [  4]  0.0-10.0 sec  1.10 GBytes   938 Mbits/sec
>>>>
>>>
>>> Is this testing done on Host or on Guest/VM?
>> Host
> 
> As per your log, iperf throughput is improved to 938 Mbits/sec
> from  6.43 Mbits/sec.
> IMO, this seems to be unrealistic, some thing wrong with the testing?
For 64bits non-pci devices, the iova allocation is always searched from the last rb-tree node.
When many iovas allocated and keep a long time, the search process should check many rb nodes
then find a suitable free space. As my tracking, the average times exceeds 10K.
        [free-space][free][used][...][used]
                      ^     ^          ^
                      |     |          |-----rb_last
                      |     |--------- maybe more than 10K allocated iova nodes
                      |------- for 32bits devices, cached32_node remember the lastest freed node, which can help us reduce check times

This patch series add a new member "cached_node" to service for 64bits devices, like cached32_node service for 32bits devices.

> 
>>
>>>
>>>>>
>>>>>
>>>>>       Joerg
>>>>>
>>>>>
>>>>> .
>>>>>
>>>>
>>>> --
>>>> Thanks!
>>>> BestRegards
>>>>
>>>>
>>>> _______________________________________________
>>>> linux-arm-kernel mailing list
>>>> linux-arm-kernel@lists.infradead.org
>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>>
>>> thanks
>>> Ganapat
>>>
>>> .
>>>
>>
>> --
>> Thanks!
>> BestRegards
>>
> 
> thanks
> Ganapat
> 
> .
> 

-- 
Thanks!
BestRegards

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web