Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource

From	Tejun Heo <tj@kernel.org>
Newsgroups	linux.kernel
Subject	Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource
Date	2015-09-10 18:50 +0200
Message-ID	<q7dcR-79C-5@gated-at.bofh.it> (permalink)
References	<q6bwt-863-3@gated-at.bofh.it> <q6t0m-8ch-5@gated-at.bofh.it> <q6EI9-89L-1@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

Hello, Parav.

On Wed, Sep 09, 2015 at 09:27:40AM +0530, Parav Pandit wrote:
> This is one old white paper, but most of the reasoning still holds true on RDMA.
> http://h10032.www1.hp.com/ctg/Manual/c00257031.pdf

Just read it.  Much appreciated.

...
> These resources include are-  QP (queue pair) to transfer data, CQ
> (Completion queue) to indicate completion of data transfer operation,
> MR (memory region) to represent user application memory as source or
> destination for data transfer.
> Common resources are QP, SRQ (shared received queue), CQ, MR, AH
> (Address handle), FLOW, PD (protection domain), user context etc.

It's kinda bothering that all these are disparate resources.  I
suppose that each restriction comes from the underlying hardware and
there's no accepted higher level abstraction for these things?

> >> This patch-set allows limiting rdma resources to set of processes.
> >> It extend device cgroup controller for limiting rdma device limits.
> >
> > I don't think this belongs to devcg.  If these make sense as a set of
> > resources to be controlled via cgroup, the right way prolly would be a
> > separate controller.
> >
> 
> In past there has been similar comment to have dedicated cgroup
> controller for RDMA instead of merging with device cgroup.
> I am ok with both the approach, however I prefer to utilize device
> controller instead of spinning of new controller for new devices
> category.
> I anticipate more such need would arise and for new device category,
> it might not be worth to have new cgroup controller.
> RapidIO though very less popular and upcoming PCIe are on horizon to
> offer similar benefits as that of RDMA and in future having one
> controller for each of them again would not be right approach.
>
> I certainly seek your and others inputs in this email thread here whether
> (a) to continue to extend device cgroup (which support character,
> block devices white list) and now RDMA devices
> or
> (b) to spin of new controller, if so what are the compelling reasons
> that it can provide compare to extension.

I'm doubtful that these things are gonna be mainstream w/o building up
higher level abstractions on top and if we ever get there we won't be
talking about MR or CQ or whatever.  Also, whatever next-gen is
unlikely to have enough commonalities when the proposed resource knobs
are this low level, so let's please keep it separate, so that if/when
this goes out of fashion for one reason or another, the controller can
silently wither away too.

> Current scope of the patch is limited to RDMA resources as first
> patch, but for fact I am sure that there are more functionality in
> pipe to support via this cgroup by me and others.
> So keeping atleast these two aspects in mind, I need input on
> direction of dedicated controller or new one.
> 
> In future, I anticipate that we might have sub directory to device
> cgroup for individual device class to control.
> such as,
> <sys/fs/cgroup/devices/
>      /char
>      /block
>      /rdma
>      /pcie
>      /child_cgroup..1..N
> Each controllers cgroup access files would remain within their own
> scope. We are not there yet from base infrastructure but something to
> be done as it matures and users start using it.

I don't think that jives with the rest of cgroup and what generic
block or pcie attributes are directly exposed to applications and need
to be hierarchically controlled via cgroup?

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Thread

[PATCH 0/7] devcg: device cgroup extension for rdma resource Parav Pandit <pandit.parav@gmail.com> - 2015-09-07 22:50 +0200
  [PATCH 3/7] devcg: Added infrastructure for rdma device cgroup. Parav Pandit <pandit.parav@gmail.com> - 2015-09-07 22:50 +0200
    Re: [PATCH 3/7] devcg: Added infrastructure for rdma device cgroup. Parav Pandit <pandit.parav@gmail.com> - 2015-09-08 09:10 +0200
  [PATCH 4/7] devcg: Added rdma resource tracker object per task Parav Pandit <pandit.parav@gmail.com> - 2015-09-07 22:50 +0200
    Re: [PATCH 4/7] devcg: Added rdma resource tracker object per task Parav Pandit <pandit.parav@gmail.com> - 2015-09-08 09:10 +0200
      Re: [PATCH 4/7] devcg: Added rdma resource tracker object per task Parav Pandit <pandit.parav@gmail.com> - 2015-09-08 10:30 +0200
  [PATCH 7/7] devcg: Added Documentation of RDMA device cgroup. Parav Pandit <pandit.parav@gmail.com> - 2015-09-07 22:50 +0200
  Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Parav Pandit <pandit.parav@gmail.com> - 2015-09-07 23:00 +0200
  Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Tejun Heo <tj@kernel.org> - 2015-09-08 17:30 +0200
    Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Parav Pandit <pandit.parav@gmail.com> - 2015-09-09 06:00 +0200
      Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Tejun Heo <tj@kernel.org> - 2015-09-10 18:50 +0200
        Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Parav Pandit <pandit.parav@gmail.com> - 2015-09-10 19:50 +0200
          Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Tejun Heo <tj@kernel.org> - 2015-09-10 22:30 +0200
            Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Parav Pandit <pandit.parav@gmail.com> - 2015-09-11 05:50 +0200
              Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Tejun Heo <tj@kernel.org> - 2015-09-11 06:10 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Doug Ledford <dledford@redhat.com> - 2015-09-11 06:30 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Tejun Heo <tj@kernel.org> - 2015-09-11 17:00 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Parav Pandit <pandit.parav@gmail.com> - 2015-09-11 18:30 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Parav Pandit <pandit.parav@gmail.com> - 2015-09-11 18:40 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Tejun Heo <tj@kernel.org> - 2015-09-11 21:30 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Parav Pandit <pandit.parav@gmail.com> - 2015-09-14 12:20 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Tejun Heo <tj@kernel.org> - 2015-09-11 18:40 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Parav Pandit <pandit.parav@gmail.com> - 2015-09-11 18:50 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Tejun Heo <tj@kernel.org> - 2015-09-11 21:10 +0200
                RE: [PATCH 0/7] devcg: device cgroup extension for rdma resource "Hefty, Sean" <sean.hefty@intel.com> - 2015-09-11 21:30 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2015-09-11 21:50 +0200
                RE: [PATCH 0/7] devcg: device cgroup extension for rdma resource "Hefty, Sean" <sean.hefty@intel.com> - 2015-09-11 22:10 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Parav Pandit <pandit.parav@gmail.com> - 2015-09-14 13:10 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Parav Pandit <pandit.parav@gmail.com> - 2015-09-14 16:10 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Tejun Heo <tj@kernel.org> - 2015-09-14 17:30 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2015-09-14 19:30 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Parav Pandit <pandit.parav@gmail.com> - 2015-09-14 21:00 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2015-09-14 22:20 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Parav Pandit <pandit.parav@gmail.com> - 2015-09-15 05:10 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2015-09-15 05:50 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Parav Pandit <pandit.parav@gmail.com> - 2015-09-16 06:50 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Parav Pandit <pandit.parav@gmail.com> - 2015-09-14 12:20 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Parav Pandit <pandit.parav@gmail.com> - 2015-09-11 06:50 +0200
                Re: [PATCH 0/7] devcg: device cgroup extension for rdma resource Tejun Heo <tj@kernel.org> - 2015-09-11 17:10 +0200
        RE: [PATCH 0/7] devcg: device cgroup extension for rdma resource "Hefty, Sean" <sean.hefty@intel.com> - 2015-09-10 19:50 +0200

csiph-web