Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1451920 > unrolled thread

[Query] increased latency observed in cpu hotplug path

Back to article view | Back to linux.kernel

Contents

  [Query] increased latency observed in cpu hotplug path "Khan, Imran" <kimran@codeaurora.org> - 2016-07-28 15:20 +0200
    Re: [Query] increased latency observed in cpu hotplug path "Khan, Imran" <kimran@codeaurora.org> - 2016-08-01 11:30 +0200
      Re: [Query] increased latency observed in cpu hotplug path "Khan, Imran" <kimran@codeaurora.org> - 2016-08-05 09:20 +0200

#1451920 — [Query] increased latency observed in cpu hotplug path

Hi,

Recently we have observed some increased latency in CPU hotplug
event in CPU online path. For online latency we see that block
layer is executing notification handler for CPU_UP_PREPARE event
and this in turn waits for RCU grace period resulting (sometimes)
in an execution time of 15-20 ms for this notification handler.
This change was not there in 3.18 kernel but is present in 4.4
kernel and was introduced by following commit:

 
commit 5778322e67ed34dc9f391a4a5cbcbb856071ceba
Author: Akinobu Mita <akinobu.mita@gmail.com>
Date:   Sun Sep 27 02:09:23 2015 +0900
 
    blk-mq: avoid inserting requests before establishing new mapping

    Notifier callbacks for CPU_ONLINE action can be run on the other CPU
    than the CPU which was just onlined.  So it is possible for the
    process running on the just onlined CPU to insert request and run
    hw queue before establishing new mapping which is done by
    blk_mq_queue_reinit_notify().

    This can cause a problem when the CPU has just been onlined first time
    since the request queue was initialized.  At this time ctx->index_hw
    for the CPU, which is the index in hctx->ctxs[] for this ctx, is still
    zero before blk_mq_queue_reinit_notify() is called by notifier
    callbacks for CPU_ONLINE action.

    For example, there is a single hw queue (hctx) and two CPU queues
    (ctx0 for CPU0, and ctx1 for CPU1).  Now CPU1 is just onlined and
    a request is inserted into ctx1->rq_list and set bit0 in pending
    bitmap as ctx1->index_hw is still zero.

    And then while running hw queue, flush_busy_ctxs() finds bit0 is set
    in pending bitmap and tries to retrieve requests in
    hctx->ctxs[0]->rq_list.  But htx->ctxs[0] is a pointer to ctx0, so the
    request in ctx1->rq_list is ignored.

    Fix it by ensuring that new mapping is established before onlined cpu
    starts running.

    Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
    Reviewed-by: Ming Lei <tom.leiming@gmail.com>
    Cc: Jens Axboe <axboe@kernel.dk>
    Cc: Ming Lei <tom.leiming@gmail.com>
    Reviewed-by: Christoph Hellwig <hch@lst.de>

 
Upon reverting this commit I could see an improvement of 15-20 ms in
online latency. So I am looking for some help in analyzing the effects
of reverting this or should some other approach to reduce the online
latency must be taken.

Can you please provide some feedback in this regard?

-- 
Imran Khan
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a\nmember of the Code Aurora Forum, hosted by The Linux Foundation

[toc] | [next] | [standalone]

#1453058

On 7/30/2016 7:54 AM, Akinobu Mita wrote:
> 2016-07-28 22:18 GMT+09:00 Khan, Imran <kimran@codeaurora.org>:
>>
>> Hi,
>>
>> Recently we have observed some increased latency in CPU hotplug
>> event in CPU online path. For online latency we see that block
>> layer is executing notification handler for CPU_UP_PREPARE event
>> and this in turn waits for RCU grace period resulting (sometimes)
>> in an execution time of 15-20 ms for this notification handler.
>> This change was not there in 3.18 kernel but is present in 4.4
>> kernel and was introduced by following commit:
>>
>>
>> commit 5778322e67ed34dc9f391a4a5cbcbb856071ceba
>> Author: Akinobu Mita <akinobu.mita@gmail.com>
>> Date:   Sun Sep 27 02:09:23 2015 +0900
>>
>>     blk-mq: avoid inserting requests before establishing new mapping
> 
> ...
> 
>> Upon reverting this commit I could see an improvement of 15-20 ms in
>> online latency. So I am looking for some help in analyzing the effects
>> of reverting this or should some other approach to reduce the online
>> latency must be taken.
> 
> Can you observe the difference in online latency by removing
> get_online_cpus() and put_online_cpus() pair in blk_mq_init_allocated_queue()
> instead of full reverting the commit?
> 
Hi Akinobu,
I tried your suggestion but could not achieve any improvement. Actually the snippet that is causing the change in latency is the following one :

list_for_each_entry(q, &all_q_list, all_q_node) {
                blk_mq_freeze_queue_wait(q);

                /*
                 * timeout handler can't touch hw queue during the
                 * reinitialization
                 */
                del_timer_sync(&q->timeout);
 }

I understand that this is getting executed now for CPU_UP_PREPARE as well resulting in 
increased latency in the cpu online path. I am trying to reduce this latency while keeping the 
purpose of this commit intact. I would welcome further suggestions/feedback in this regard.

-- 
Imran Khan
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a\nmember of the Code Aurora Forum, hosted by The Linux Foundation

[toc] | [prev] | [next] | [standalone]

#1456908

On 8/1/2016 2:58 PM, Khan, Imran wrote:
> On 7/30/2016 7:54 AM, Akinobu Mita wrote:
>> 2016-07-28 22:18 GMT+09:00 Khan, Imran <kimran@codeaurora.org>:
>>>
>>> Hi,
>>>
>>> Recently we have observed some increased latency in CPU hotplug
>>> event in CPU online path. For online latency we see that block
>>> layer is executing notification handler for CPU_UP_PREPARE event
>>> and this in turn waits for RCU grace period resulting (sometimes)
>>> in an execution time of 15-20 ms for this notification handler.
>>> This change was not there in 3.18 kernel but is present in 4.4
>>> kernel and was introduced by following commit:
>>>
>>>
>>> commit 5778322e67ed34dc9f391a4a5cbcbb856071ceba
>>> Author: Akinobu Mita <akinobu.mita@gmail.com>
>>> Date:   Sun Sep 27 02:09:23 2015 +0900
>>>
>>>     blk-mq: avoid inserting requests before establishing new mapping
>>
>> ...
>>
>>> Upon reverting this commit I could see an improvement of 15-20 ms in
>>> online latency. So I am looking for some help in analyzing the effects
>>> of reverting this or should some other approach to reduce the online
>>> latency must be taken.
>>
>> Can you observe the difference in online latency by removing
>> get_online_cpus() and put_online_cpus() pair in blk_mq_init_allocated_queue()
>> instead of full reverting the commit?
>>
> Hi Akinobu,
> I tried your suggestion but could not achieve any improvement. Actually the snippet that is causing the change in latency is the following one :
> 
> list_for_each_entry(q, &all_q_list, all_q_node) {
>                 blk_mq_freeze_queue_wait(q);
> 
>                 /*
>                  * timeout handler can't touch hw queue during the
>                  * reinitialization
>                  */
>                 del_timer_sync(&q->timeout);
>  }
> 
> I understand that this is getting executed now for CPU_UP_PREPARE as well resulting in 
> increased latency in the cpu online path. I am trying to reduce this latency while keeping the 
> purpose of this commit intact. I would welcome further suggestions/feedback in this regard.
> 
Hi Akinobu,

I am not able to reduce the cpu online latency with this patch, could you please let me know what
functionality will be broken, if we avoid this patch in our kernel. Also if you have some other 
suggestions towards improving this patch please let me know.

-- 
Imran Khan
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a\nmember of the Code Aurora Forum, hosted by The Linux Foundation

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web