Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1531672 > unrolled thread

Re: [RFC 4/4] mm: Ignore cpuset enforcement when allocation flag has __GFP_THISNODE

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [RFC 4/4] mm: Ignore cpuset enforcement when allocation flag has  __GFP_THISNODE Dave Hansen <dave.hansen@intel.com> - 2016-11-28 22:20 +0100
    Re: [RFC 4/4] mm: Ignore cpuset enforcement when allocation flag has  __GFP_THISNODE Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-11-29 08:00 +0100
      Re: [RFC 4/4] mm: Ignore cpuset enforcement when allocation flag has  __GFP_THISNODE Dave Hansen <dave.hansen@intel.com> - 2016-11-29 18:00 +0100
        Re: [RFC 4/4] mm: Ignore cpuset enforcement when allocation flag has  __GFP_THISNODE Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-11-30 12:20 +0100
          Re: [RFC 4/4] mm: Ignore cpuset enforcement when allocation flag has  __GFP_THISNODE Dave Hansen <dave.hansen@intel.com> - 2016-11-30 20:50 +0100

#1531672 — Re: [RFC 4/4] mm: Ignore cpuset enforcement when allocation flag has __GFP_THISNODE

On 11/22/2016 06:19 AM, Anshuman Khandual wrote:
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -3715,7 +3715,7 @@ struct page *
>  		.migratetype = gfpflags_to_migratetype(gfp_mask),
>  	};
>  
> -	if (cpusets_enabled()) {
> +	if (cpusets_enabled() && !(alloc_mask & __GFP_THISNODE)) {
>  		alloc_mask |= __GFP_HARDWALL;
>  		alloc_flags |= ALLOC_CPUSET;
>  		if (!ac.nodemask)

This means now that any __GFP_THISNODE allocation can "escape" the
cpuset.  That seems like a pretty major change to how cpusets works.  Do
we know that *ALL* __GFP_THISNODE allocations are truly lacking in a
cpuset context that can be enforced?

[toc] | [next] | [standalone]

#1531947

On 11/29/2016 02:42 AM, Dave Hansen wrote:
> On 11/22/2016 06:19 AM, Anshuman Khandual wrote:
>> --- a/mm/page_alloc.c
>> +++ b/mm/page_alloc.c
>> @@ -3715,7 +3715,7 @@ struct page *
>>  		.migratetype = gfpflags_to_migratetype(gfp_mask),
>>  	};
>>  
>> -	if (cpusets_enabled()) {
>> +	if (cpusets_enabled() && !(alloc_mask & __GFP_THISNODE)) {
>>  		alloc_mask |= __GFP_HARDWALL;
>>  		alloc_flags |= ALLOC_CPUSET;
>>  		if (!ac.nodemask)
> 
> This means now that any __GFP_THISNODE allocation can "escape" the
> cpuset.  That seems like a pretty major change to how cpusets works.  Do
> we know that *ALL* __GFP_THISNODE allocations are truly lacking in a
> cpuset context that can be enforced?

Right, I know its a very blunt change. With the cpuset based isolation
of coherent device node for the user space tasks leads to a side effect
that a driver or even kernel cannot allocate memory from the coherent
device node in the task's own context (ioctl() calls or similar). For
non task context allocation (work queues, interrupts, anything async
etc) this problem can be fixed by modifying kernel thread's task->mems
_allowed to include all nodes of the system including the coherent
device nodes. Though I have not figured out the details yet. Whats
your thoughts on this ? What we are looking for is a explicit and
definite way of allocating from the coherent device node inside the
kernel.

[toc] | [prev] | [next] | [standalone]

#1532510

On 11/28/2016 10:51 PM, Anshuman Khandual wrote:
> On 11/29/2016 02:42 AM, Dave Hansen wrote:
>> > On 11/22/2016 06:19 AM, Anshuman Khandual wrote:
>>> >> --- a/mm/page_alloc.c
>>> >> +++ b/mm/page_alloc.c
>>> >> @@ -3715,7 +3715,7 @@ struct page *
>>> >>  		.migratetype = gfpflags_to_migratetype(gfp_mask),
>>> >>  	};
>>> >>  
>>> >> -	if (cpusets_enabled()) {
>>> >> +	if (cpusets_enabled() && !(alloc_mask & __GFP_THISNODE)) {
>>> >>  		alloc_mask |= __GFP_HARDWALL;
>>> >>  		alloc_flags |= ALLOC_CPUSET;
>>> >>  		if (!ac.nodemask)
>> > 
>> > This means now that any __GFP_THISNODE allocation can "escape" the
>> > cpuset.  That seems like a pretty major change to how cpusets works.  Do
>> > we know that *ALL* __GFP_THISNODE allocations are truly lacking in a
>> > cpuset context that can be enforced?
> Right, I know its a very blunt change. With the cpuset based isolation
> of coherent device node for the user space tasks leads to a side effect
> that a driver or even kernel cannot allocate memory from the coherent
...

Well, we have __GFP_HARDWALL:

	 * __GFP_HARDWALL enforces the cpuset memory allocation policy.

which you can clear in the places where you want to do an allocation but
want to ignore cpusets.  But, __cpuset_node_allowed() looks like it gets
a little funky if you do that since it would probably be falling back to
the root cpuset that also would not have the new node in mems_allowed.

What exactly are the kernel-internal places that need to allocate from
the coherent device node?  When would this be done out of the context of
an application *asking* for memory in the new node?

[toc] | [prev] | [next] | [standalone]

#1533231

On 11/29/2016 10:22 PM, Dave Hansen wrote:
> On 11/28/2016 10:51 PM, Anshuman Khandual wrote:
>> On 11/29/2016 02:42 AM, Dave Hansen wrote:
>>>> On 11/22/2016 06:19 AM, Anshuman Khandual wrote:
>>>>>> --- a/mm/page_alloc.c
>>>>>> +++ b/mm/page_alloc.c
>>>>>> @@ -3715,7 +3715,7 @@ struct page *
>>>>>>  		.migratetype = gfpflags_to_migratetype(gfp_mask),
>>>>>>  	};
>>>>>>  
>>>>>> -	if (cpusets_enabled()) {
>>>>>> +	if (cpusets_enabled() && !(alloc_mask & __GFP_THISNODE)) {
>>>>>>  		alloc_mask |= __GFP_HARDWALL;
>>>>>>  		alloc_flags |= ALLOC_CPUSET;
>>>>>>  		if (!ac.nodemask)
>>>>
>>>> This means now that any __GFP_THISNODE allocation can "escape" the
>>>> cpuset.  That seems like a pretty major change to how cpusets works.  Do
>>>> we know that *ALL* __GFP_THISNODE allocations are truly lacking in a
>>>> cpuset context that can be enforced?
>> Right, I know its a very blunt change. With the cpuset based isolation
>> of coherent device node for the user space tasks leads to a side effect
>> that a driver or even kernel cannot allocate memory from the coherent
> ...
> 
> Well, we have __GFP_HARDWALL:
> 
> 	 * __GFP_HARDWALL enforces the cpuset memory allocation policy.
> 
> which you can clear in the places where you want to do an allocation but
> want to ignore cpusets.  But, __cpuset_node_allowed() looks like it gets
> a little funky if you do that since it would probably be falling back to
> the root cpuset that also would not have the new node in mems_allowed.

Right but what is the rationale behind this ? This what is in the in-code
documentation for this function __cpuset_node_allowed().

 *	GFP_KERNEL   - any node in enclosing hardwalled cpuset ok
 
If the allocation has requested GFP_KERNEL, should not it look for the
entire system for memory ? Does cpuset still has to be enforced ?

> 
> What exactly are the kernel-internal places that need to allocate from
> the coherent device node?  When would this be done out of the context of
> an application *asking* for memory in the new node?

The primary user right now is a driver who wants to move around mapped
pages of an application from system RAM to CDM nodes and back. If the
application has requested for it though an ioctl(), during migration
the destination pages will be allocated on the CDM *in* the task context.

The driver could also have scheduled migration chunks in the work queue
which can execute later on. IIUC those execution and corresponding
allocation into CDM node will be *out* of context of the task.

Ideally looking for both the scenarios to work which dont right now.

[toc] | [prev] | [next] | [standalone]

#1533533

On 11/30/2016 03:17 AM, Anshuman Khandual wrote:
> Right but what is the rationale behind this ? This what is in the in-code
> documentation for this function __cpuset_node_allowed().
> 
>  *	GFP_KERNEL   - any node in enclosing hardwalled cpuset ok
>  
> If the allocation has requested GFP_KERNEL, should not it look for the
> entire system for memory ? Does cpuset still has to be enforced ?

Documentation/cgroup-v1/cpusets.txt explains it quite a bit.

>> What exactly are the kernel-internal places that need to allocate from
>> the coherent device node?  When would this be done out of the context of
>> an application *asking* for memory in the new node?
> 
> The primary user right now is a driver who wants to move around mapped
> pages of an application from system RAM to CDM nodes and back. If the
> application has requested for it though an ioctl(), during migration
> the destination pages will be allocated on the CDM *in* the task context.

Side note: uhh, so you're doing migrate_pages() through some kind of new
ioctl()?  Why?

I think you're actually pointing out a hole in how cpusets currently
works, especially about the workqueue.  I'm not quite sure if this is by
design for migrate_pages() (a task doing migrate_pages() can pages for a
task from a cpuset even though that task isn't able to allocate itself).

> The driver could also have scheduled migration chunks in the work queue
> which can execute later on. IIUC those execution and corresponding
> allocation into CDM node will be *out* of context of the task.

Yeah, the current->mems_allowed in __cpuset_node_allowed() does seem
rather wrong for something happening in another task's context.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web