Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1614320

Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device

Show all headers | View raw

On 31/03/17 03:38 PM, Sinan Kaya wrote:
> On 3/31/2017 5:23 PM, Logan Gunthorpe wrote:
>> What exactly would you white/black list? It can't be the NIC or the
>> disk. If it's going to be a white/black list on the switch or root port
>> then you'd need essentially the same code to ensure they are all behind
>> the same switch or root port.
> 
> What is so special about being connected to the same switch?
> 
> Why don't we allow the feature by default and blacklist by the root ports
> that don't work with a quirk.

Well root ports have the same issue here. There may be more than one
root port or other buses (ie QPI) between the devices in question. So
you can't just say "this system has root port X therefore we can always
use p2pmem". In the end, if you want to do any kind of restrictions
you're going to have to walk the tree, as the code currently does, and
figure out what's between the devices being used and black or white list
accordingly. Then seeing there's just such a vast number of devices out
there you'd almost certainly have to use some kind of white list and not
a black list. Then the question becomes which devices will be white
listed? The first to be listed would be switches seeing they will always
work. This is pretty much what we have (though it doesn't currently
cover multiple levels of switches). The next step, if someone wanted to
test with specific hardware, might be to allow the case where all the
devices are behind the same root port which Intel Ivy Bridge or newer.
However, I don't think a comprehensive white list should be a
requirement for this work to go forward and I don't think anything
you've suggested will remove any of the "ugliness".

What we discussed at LSF was that only allowing cases with a switch was
the simplest way to be sure any given setup would actually work.

> I'm looking at this from portability perspective to be honest.

I'm looking at this from the fact that there's a vast number of
topologies and devices involved, and figuring out which will work is
very complicated and could require a lot of hardware testing. The LSF
folks were primarily concerned with not having users enable the feature
and see breakage or terrible performance.

> I'd rather see the feature enabled by default without any assumptions.
> Using it with a switch is just a use case that you happened to test.
> It can allow new architectures to use your code tomorrow.

That's why I was advocating for letting userspace decide such that if
you're setting up a system with this you say to use a specific p2pmem
device and then you are responsible to test and benchmark it and decide
to use it in going forward. However, this has received a lot of push back.

Logan

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory Logan Gunthorpe <logang@deltatee.com> - 2017-03-31 00:20 +0200
  [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device Logan Gunthorpe <logang@deltatee.com> - 2017-03-31 00:20 +0200
    Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device Sinan Kaya <okaya@codeaurora.org> - 2017-03-31 20:50 +0200
      Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device Logan Gunthorpe <logang@deltatee.com> - 2017-03-31 23:30 +0200
        Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device Sinan Kaya <okaya@codeaurora.org> - 2017-03-31 23:40 +0200
          Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device Logan Gunthorpe <logang@deltatee.com> - 2017-04-01 00:50 +0200
            Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device Sinan Kaya <okaya@codeaurora.org> - 2017-04-01 02:00 +0200
              Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device Logan Gunthorpe <logang@deltatee.com> - 2017-04-01 04:00 +0200
                Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device okaya@codeaurora.org - 2017-04-01 04:20 +0200
                Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device Logan Gunthorpe <logang@deltatee.com> - 2017-04-02 00:20 +0200
                Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device Sinan Kaya <okaya@codeaurora.org> - 2017-04-02 04:30 +0200
                Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device Logan Gunthorpe <logang@deltatee.com> - 2017-04-02 19:30 +0200
                Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device Sinan Kaya <okaya@codeaurora.org> - 2017-04-02 23:10 +0200
                Re: [RFC 1/8] Introduce Peer-to-Peer memory (p2pmem) device Logan Gunthorpe <logang@deltatee.com> - 2017-04-03 06:30 +0200
  [RFC 4/8] p2pmem: Add debugfs "stats" file Logan Gunthorpe <logang@deltatee.com> - 2017-03-31 00:20 +0200
    Re: [RFC 4/8] p2pmem: Add debugfs "stats" file Sagi Grimberg <sagi@grimberg.me> - 2017-04-04 12:50 +0200
      Re: [RFC 4/8] p2pmem: Add debugfs "stats" file Logan Gunthorpe <logang@deltatee.com> - 2017-04-04 19:30 +0200
      RE: [RFC 4/8] p2pmem: Add debugfs "stats" file "Steve Wise" <swise@opengridcomputing.com> - 2017-04-05 18:20 +0200
  [RFC 2/8] cxgb4: setup pcie memory window 4 and create p2pmem region Logan Gunthorpe <logang@deltatee.com> - 2017-03-31 00:20 +0200
    Re: [RFC 2/8] cxgb4: setup pcie memory window 4 and create p2pmem  region Sagi Grimberg <sagi@grimberg.me> - 2017-04-04 12:50 +0200
      Re: [RFC 2/8] cxgb4: setup pcie memory window 4 and create p2pmem  region Logan Gunthorpe <logang@deltatee.com> - 2017-04-04 18:00 +0200
      RE: [RFC 2/8] cxgb4: setup pcie memory window 4 and create p2pmem region "Steve Wise" <swise@opengridcomputing.com> - 2017-04-05 18:20 +0200
  [RFC 5/8] scatterlist: Modify SG copy functions to support io memory. Logan Gunthorpe <logang@deltatee.com> - 2017-03-31 00:20 +0200
    Re: [RFC 5/8] scatterlist: Modify SG copy functions to support io  memory. Christoph Hellwig <hch@infradead.org> - 2017-03-31 09:20 +0200
      Re: [RFC 5/8] scatterlist: Modify SG copy functions to support io  memory. Logan Gunthorpe <logang@deltatee.com> - 2017-03-31 17:50 +0200
        Re: [RFC 5/8] scatterlist: Modify SG copy functions to support io  memory. Logan Gunthorpe <logang@deltatee.com> - 2017-04-03 23:30 +0200
          Re: [RFC 5/8] scatterlist: Modify SG copy functions to support io memory. Dan Williams <dan.j.williams@intel.com> - 2017-04-03 23:50 +0200
            Re: [RFC 5/8] scatterlist: Modify SG copy functions to support io  memory. Logan Gunthorpe <logang@deltatee.com> - 2017-04-04 00:20 +0200
              Re: [RFC 5/8] scatterlist: Modify SG copy functions to support io memory. Dan Williams <dan.j.williams@intel.com> - 2017-04-04 00:50 +0200
                Re: [RFC 5/8] scatterlist: Modify SG copy functions to support io  memory. Logan Gunthorpe <logang@deltatee.com> - 2017-04-04 01:20 +0200
                Re: [RFC 5/8] scatterlist: Modify SG copy functions to support io memory. Dan Williams <dan.j.williams@intel.com> - 2017-04-04 02:10 +0200
                Re: [RFC 5/8] scatterlist: Modify SG copy functions to support io  memory. Logan Gunthorpe <logang@deltatee.com> - 2017-04-07 20:00 +0200
  [RFC 8/8] p2pmem: Added char device user interface Logan Gunthorpe <logang@deltatee.com> - 2017-03-31 00:20 +0200
  [RFC 7/8] p2pmem: Support device removal Logan Gunthorpe <logang@deltatee.com> - 2017-03-31 00:20 +0200

csiph-web