Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1310133 > unrolled thread

Re: [PATCH 07/13] IB: add a proper completion queue abstraction

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 07/13] IB: add a proper completion queue abstraction Parav Pandit <pandit.parav@gmail.com> - 2016-01-15 15:00 +0100
    Re: [PATCH 07/13] IB: add a proper completion queue abstraction Sagi Grimberg <sagig@dev.mellanox.co.il> - 2016-01-17 10:30 +0100
      Re: [PATCH 07/13] IB: add a proper completion queue abstraction Parav Pandit <pandit.parav@gmail.com> - 2016-01-17 12:10 +0100
        Re: [PATCH 07/13] IB: add a proper completion queue abstraction Parav Pandit <pandit.parav@gmail.com> - 2016-01-17 12:10 +0100
        Re: [PATCH 07/13] IB: add a proper completion queue abstraction Sagi Grimberg <sagig@dev.mellanox.co.il> - 2016-01-17 12:10 +0100

#1310133 — Re: [PATCH 07/13] IB: add a proper completion queue abstraction

Hi Christoph, Sagi,

On Tue, Dec 8, 2015 at 2:21 AM, Christoph Hellwig <hch@lst.de> wrote:

> +static void ib_cq_poll_work(struct work_struct *work)
> +{
> +       struct ib_cq *cq = container_of(work, struct ib_cq, work);
> +       int completed;
> +
> +       completed = __ib_process_cq(cq, IB_POLL_BUDGET_WORKQUEUE);
> +       if (completed >= IB_POLL_BUDGET_WORKQUEUE ||
> +           ib_req_notify_cq(cq, IB_POLL_FLAGS) > 0)
> +               queue_work(ib_comp_wq, &cq->work);
> +}

In above code, Let says completion is added in a time window where
ib_process_cq is completed (CQ is diarmed in hw at that point) and
ib_req_notify_cq is yet to be called.
Provider vendor driver say mlx4 or mlx5 as specific case always
returns ib_req_notify_cq = 0.
Will it result into a missed notification? (so queue_work is not done).

IB spec says: Any CQ entries that existed before the notify is enabled
will not result in a call to the handler.

I did try to follow this thread on similar notes.
http://www.spinics.net/lists/linux-nfs/msg54266.html

I believe above code possibly needs fix based above two comments? Or I
must be missing something basic here.

[toc] | [next] | [standalone]

#1311118

> Hi Christoph, Sagi,
>
> On Tue, Dec 8, 2015 at 2:21 AM, Christoph Hellwig <hch@lst.de> wrote:
>
>> +static void ib_cq_poll_work(struct work_struct *work)
>> +{
>> +       struct ib_cq *cq = container_of(work, struct ib_cq, work);
>> +       int completed;
>> +
>> +       completed = __ib_process_cq(cq, IB_POLL_BUDGET_WORKQUEUE);
>> +       if (completed >= IB_POLL_BUDGET_WORKQUEUE ||
>> +           ib_req_notify_cq(cq, IB_POLL_FLAGS) > 0)
>> +               queue_work(ib_comp_wq, &cq->work);
>> +}
>
> In above code, Let says completion is added in a time window where
> ib_process_cq is completed (CQ is diarmed in hw at that point) and
> ib_req_notify_cq is yet to be called.
> Provider vendor driver say mlx4 or mlx5 as specific case always
> returns ib_req_notify_cq = 0.
> Will it result into a missed notification? (so queue_work is not done).

If we have not drained the CQ (consumed budget or more) the second
condition (ib_req_notify_cq) will not be invoked. We are only rearming
the CQ when we drained it completely. So I don't see how we can end up
with missed notifications.

[toc] | [prev] | [next] | [standalone]

#1311133

Hi Sagi,

On Sun, Jan 17, 2016 at 2:54 PM, Sagi Grimberg <sagig@dev.mellanox.co.il> wrote:
>
>> Hi Christoph, Sagi,
>>
>> On Tue, Dec 8, 2015 at 2:21 AM, Christoph Hellwig <hch@lst.de> wrote:
>>
>>> +static void ib_cq_poll_work(struct work_struct *work)
>>> +{
>>> +       struct ib_cq *cq = container_of(work, struct ib_cq, work);
>>> +       int completed;
>>> +
>>> +       completed = __ib_process_cq(cq, IB_POLL_BUDGET_WORKQUEUE);
>>> +       if (completed >= IB_POLL_BUDGET_WORKQUEUE ||
>>> +           ib_req_notify_cq(cq, IB_POLL_FLAGS) > 0)
>>> +               queue_work(ib_comp_wq, &cq->work);
>>> +}
>>
>>
>> In above code, Let says completion is added in a time window where
>> ib_process_cq is completed (CQ is diarmed in hw at that point) and
>> ib_req_notify_cq is yet to be called.
>> Provider vendor driver say mlx4 or mlx5 as specific case always
>> returns ib_req_notify_cq = 0.
>> Will it result into a missed notification? (so queue_work is not done).
>
>
> If we have not drained the CQ (consumed budget or more) the second
> condition (ib_req_notify_cq) will not be invoked. We are only rearming
> the CQ when we drained it completely. So I don't see how we can end up
> with missed notifications.
We drain the CQ completely for whatever CQEs available at that time,
say for example,
33 CQEs drained at time t1. So now req_notify_cq will be invoked at time t2.
During time delta t2-t1, CQ in hardware remains unarmed.
If cqes are added during that time delta, Will event/interrupt raised
for it, for CQ in unarmed state?

At time time t2, CQ is armed containing pending CQEs. Will
event/interrupt raised for those pending CQEs on next arming?

[toc] | [prev] | [next] | [standalone]

#1311134

On Sun, Jan 17, 2016 at 4:36 PM, Sagi Grimberg <sagig@dev.mellanox.co.il> wrote:
>
>>> If we have not drained the CQ (consumed budget or more) the second
>>> condition (ib_req_notify_cq) will not be invoked. We are only rearming
>>> the CQ when we drained it completely. So I don't see how we can end up
>>> with missed notifications.
>>
>> We drain the CQ completely for whatever CQEs available at that time,
>> say for example,
>> 33 CQEs drained at time t1. So now req_notify_cq will be invoked at time
>> t2.
>> During time delta t2-t1, CQ in hardware remains unarmed.
>> If cqes are added during that time delta, Will event/interrupt raised
>> for it, for CQ in unarmed state?
>>
>> At time time t2, CQ is armed containing pending CQEs. Will
>> event/interrupt raised for those pending CQEs on next arming?
>
>
> If the device did not reported missed-events then it is the device
> responsibility to generate a new completion event after arming.
>
> Specifically, the mlx4/mlx5 HW is able to generate a completion event
> in your described scenario. A device that is not capable of doing so
> must report missed events to inform the core it has more completions
> to consume.
o.k. Thanks.

[toc] | [prev] | [next] | [standalone]

#1311135

>> If we have not drained the CQ (consumed budget or more) the second
>> condition (ib_req_notify_cq) will not be invoked. We are only rearming
>> the CQ when we drained it completely. So I don't see how we can end up
>> with missed notifications.
> We drain the CQ completely for whatever CQEs available at that time,
> say for example,
> 33 CQEs drained at time t1. So now req_notify_cq will be invoked at time t2.
> During time delta t2-t1, CQ in hardware remains unarmed.
> If cqes are added during that time delta, Will event/interrupt raised
> for it, for CQ in unarmed state?
>
> At time time t2, CQ is armed containing pending CQEs. Will
> event/interrupt raised for those pending CQEs on next arming?

If the device did not reported missed-events then it is the device
responsibility to generate a new completion event after arming.

Specifically, the mlx4/mlx5 HW is able to generate a completion event
in your described scenario. A device that is not capable of doing so
must report missed events to inform the core it has more completions
to consume.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web