Re: [PATCH 2/9] IB: add a proper completion queue abstraction

From	Tom Talpey <tom@talpey.com>
Newsgroups	linux.kernel
Subject	Re: [PATCH 2/9] IB: add a proper completion queue abstraction
Date	2015-11-24 14:10 +0100
Message-ID	<qylw6-5A-13@gated-at.bofh.it> (permalink)
References	(6 earlier) <qy8pc-8rU-3@gated-at.bofh.it> <qy9lg-zO-7@gated-at.bofh.it> <qy9lg-zO-5@gated-at.bofh.it> <qybQ6-22s-3@gated-at.bofh.it> <qyfTH-4UT-5@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

On 11/24/2015 2:03 AM, Jason Gunthorpe wrote:
> On Mon, Nov 23, 2015 at 06:35:28PM -0800, Caitlin Bestler wrote:
>> Are there actual HCAs that make this mistake?
>
> All IB HCAs have this behavior and require apps to see a send CQ
> completion before making any statements about the state of the send Q
> or buffers handed over to the HCA. Tom and I have seen this in real
> systems under proper stress conditions. [Which is why I am so certain
> about this, because when I first hit it years ago I dug into the spec
> and figured out it was not a HW bug I was looking at]

To be clear, I saw the reply-completion-before-request-completion on
Windows, not Linux, but the principle is identical. It's simply a
fact of life on a multiprocessor, unless you want to throw in locks
and synchronization rules that drivers have to follow to enforce
ordered completions across queues. Which trust me, you don't.

In Windows SMB Direct, we added reference counts around pretty much
every verb interaction associated with each upper layer operation,
and did not retire them until all refcounts went to zero. It is
excruciatingly correct yet performs incredibly well.

Tom.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Back to linux.kernel | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Re: [PATCH 2/9] IB: add a proper completion queue abstraction Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2015-11-23 21:40 +0100
  Re: [PATCH 2/9] IB: add a proper completion queue abstraction Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2015-11-23 22:30 +0100
    Re: [PATCH 2/9] IB: add a proper completion queue abstraction Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2015-11-23 23:20 +0100
      Re: [PATCH 2/9] IB: add a proper completion queue abstraction Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2015-11-24 00:10 +0100
        Re: [PATCH 2/9] IB: add a proper completion queue abstraction Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2015-11-24 01:10 +0100
          Re: [PATCH 2/9] IB: add a proper completion queue abstraction Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2015-11-24 01:50 +0100
          Re: [PATCH 2/9] IB: add a proper completion queue abstraction Tom Talpey <tom@talpey.com> - 2015-11-24 01:50 +0100
          Re: [PATCH 2/9] IB: add a proper completion queue abstraction Caitlin Bestler <cait@asomi.com> - 2015-11-24 03:50 +0100
            Re: [PATCH 2/9] IB: add a proper completion queue abstraction Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2015-11-24 08:10 +0100
              Re: [PATCH 2/9] IB: add a proper completion queue abstraction Tom Talpey <tom@talpey.com> - 2015-11-24 14:10 +0100

csiph-web