Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1345753 > unrolled thread

Re: [RFC PATCH] mm: CONFIG_NR_ZONES_EXTENDED

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [RFC PATCH] mm: CONFIG_NR_ZONES_EXTENDED Vlastimil Babka <vbabka@suse.cz> - 2016-02-29 13:40 +0100
    Re: [RFC PATCH] mm: CONFIG_NR_ZONES_EXTENDED Dan Williams <dan.j.williams@intel.com> - 2016-02-29 19:00 +0100
      Re: [RFC PATCH] mm: CONFIG_NR_ZONES_EXTENDED Vlastimil Babka <vbabka@suse.cz> - 2016-03-01 01:10 +0100
        Re: [RFC PATCH] mm: CONFIG_NR_ZONES_EXTENDED Dan Williams <dan.j.williams@intel.com> - 2016-03-01 03:10 +0100
          Re: [RFC PATCH] mm: CONFIG_NR_ZONES_EXTENDED Vlastimil Babka <vbabka@suse.cz> - 2016-03-01 09:40 +0100
            Re: [RFC PATCH] mm: CONFIG_NR_ZONES_EXTENDED Dan Williams <dan.j.williams@intel.com> - 2016-03-02 00:50 +0100
              Re: [RFC PATCH] mm: CONFIG_NR_ZONES_EXTENDED Vlastimil Babka <vbabka@suse.cz> - 2016-03-02 09:20 +0100

#1345753 — Re: [RFC PATCH] mm: CONFIG_NR_ZONES_EXTENDED

On 02/02/2016 06:42 AM, Andrew Morton wrote:
> On Wed, 27 Jan 2016 22:19:14 -0800 Dan Williams <dan.j.williams@intel.com> wrote:
>
>> ZONE_DEVICE (merged in 4.3) and ZONE_CMA (proposed) are examples of new
>> mm zones that are bumping up against the current maximum limit of 4
>> zones, i.e. 2 bits in page->flags.  When adding a zone this equation
>> still needs to be satisified:
>>
>>      SECTIONS_WIDTH + ZONES_WIDTH + NODES_SHIFT + LAST_CPUPID_SHIFT
>> 	  <= BITS_PER_LONG - NR_PAGEFLAGS
>>
>> ZONE_DEVICE currently tries to satisfy this equation by requiring that
>> ZONE_DMA be disabled, but this is untenable given generic kernels want
>> to support ZONE_DEVICE and ZONE_DMA simultaneously.  ZONE_CMA would like
>> to increase the amount of memory covered per section, but that limits
>> the minimum granularity at which consecutive memory ranges can be added
>> via devm_memremap_pages().
>>
>> The trade-off of what is acceptable to sacrifice depends heavily on the
>> platform.  For example, ZONE_CMA is targeted for 32-bit platforms where
>> page->flags is constrained, but those platforms likely do not care about
>> the minimum granularity of memory hotplug.  A big iron machine with 1024
>> numa nodes can likely sacrifice ZONE_DMA where a general purpose
>> distribution kernel can not.
>>
>> CONFIG_NR_ZONES_EXTENDED is a configuration symbol that gets selected
>> when the number of configured zones exceeds 4.  It documents the
>> configuration symbols and definitions that get modified when ZONES_WIDTH
>> is greater than 2.
>>
>> For now, it steals a bit from NODES_SHIFT.  Later on it can be used to
>> document the definitions that get modified when a 32-bit configuration
>> wants more zone bits.
>
> So if you want ZONE_DMA, you're limited to 512 NUMA nodes?
>
> That seems reasonable.

Sorry for the late reply, but it seems that with !SPARSEMEM, or with 
SPARSEMEM_VMEMMAP, reducing NUMA nodes isn't even necessary, because 
SECTIONS_WIDTH is zero (see the diagrams in linux/page-flags-layout.h). 
In my brief tests with 4.4 based kernel with SPARSEMEM_VMEMMAP it seems 
that with 1024 NUMA nodes and 8192 CPU's, there's still 7 bits left 
(i.e. 6 with CONFIG_NR_ZONES_EXTENDED).

With the danger of becoming even more complex, could the limit also 
depend on CONFIG_SPARSEMEM/VMEMMAP to reflect that somehow?

Or does it even make sense to limit the Kconfig choice like this? Same 
reduction of bits could be achieved in multiple ways. Less CPU's means 
smaller LAST_CPUPID_SHIFT. NUMA_BALACING disabled means LAST_CPUPID_SHIFT=0.

What would be perhaps better is to (in case things don't fit) show what 
uses how many bits and what are the relevant config options to tune to 
make it fit?

[toc] | [next] | [standalone]

#1346049

On Mon, Feb 29, 2016 at 4:33 AM, Vlastimil Babka <vbabka@suse.cz> wrote:
> On 02/02/2016 06:42 AM, Andrew Morton wrote:
>>
>> On Wed, 27 Jan 2016 22:19:14 -0800 Dan Williams <dan.j.williams@intel.com>
>> wrote:
>>
>>> ZONE_DEVICE (merged in 4.3) and ZONE_CMA (proposed) are examples of new
>>> mm zones that are bumping up against the current maximum limit of 4
>>> zones, i.e. 2 bits in page->flags.  When adding a zone this equation
>>> still needs to be satisified:
>>>
>>>      SECTIONS_WIDTH + ZONES_WIDTH + NODES_SHIFT + LAST_CPUPID_SHIFT
>>>           <= BITS_PER_LONG - NR_PAGEFLAGS
>>>
>>> ZONE_DEVICE currently tries to satisfy this equation by requiring that
>>> ZONE_DMA be disabled, but this is untenable given generic kernels want
>>> to support ZONE_DEVICE and ZONE_DMA simultaneously.  ZONE_CMA would like
>>> to increase the amount of memory covered per section, but that limits
>>> the minimum granularity at which consecutive memory ranges can be added
>>> via devm_memremap_pages().
>>>
>>> The trade-off of what is acceptable to sacrifice depends heavily on the
>>> platform.  For example, ZONE_CMA is targeted for 32-bit platforms where
>>> page->flags is constrained, but those platforms likely do not care about
>>> the minimum granularity of memory hotplug.  A big iron machine with 1024
>>> numa nodes can likely sacrifice ZONE_DMA where a general purpose
>>> distribution kernel can not.
>>>
>>> CONFIG_NR_ZONES_EXTENDED is a configuration symbol that gets selected
>>> when the number of configured zones exceeds 4.  It documents the
>>> configuration symbols and definitions that get modified when ZONES_WIDTH
>>> is greater than 2.
>>>
>>> For now, it steals a bit from NODES_SHIFT.  Later on it can be used to
>>> document the definitions that get modified when a 32-bit configuration
>>> wants more zone bits.
>>
>>
>> So if you want ZONE_DMA, you're limited to 512 NUMA nodes?
>>
>> That seems reasonable.
>
>
> Sorry for the late reply, but it seems that with !SPARSEMEM, or with
> SPARSEMEM_VMEMMAP, reducing NUMA nodes isn't even necessary, because
> SECTIONS_WIDTH is zero (see the diagrams in linux/page-flags-layout.h). In
> my brief tests with 4.4 based kernel with SPARSEMEM_VMEMMAP it seems that
> with 1024 NUMA nodes and 8192 CPU's, there's still 7 bits left (i.e. 6 with
> CONFIG_NR_ZONES_EXTENDED).
>
> With the danger of becoming even more complex, could the limit also depend
> on CONFIG_SPARSEMEM/VMEMMAP to reflect that somehow?

In this case it's already part of the equation because:

config ZONE_DEVICE
       depends on MEMORY_HOTPLUG
       depends on MEMORY_HOTREMOVE

...and those in turn depend on SPARSEMEM.

[toc] | [prev] | [next] | [standalone]

#1346265

On 29.2.2016 18:55, Dan Williams wrote:
> On Mon, Feb 29, 2016 at 4:33 AM, Vlastimil Babka <vbabka@suse.cz> wrote:
>> On 02/02/2016 06:42 AM, Andrew Morton wrote:
>>> So if you want ZONE_DMA, you're limited to 512 NUMA nodes?
>>>
>>> That seems reasonable.
>>
>>
>> Sorry for the late reply, but it seems that with !SPARSEMEM, or with
>> SPARSEMEM_VMEMMAP, reducing NUMA nodes isn't even necessary, because
>> SECTIONS_WIDTH is zero (see the diagrams in linux/page-flags-layout.h). In
>> my brief tests with 4.4 based kernel with SPARSEMEM_VMEMMAP it seems that
>> with 1024 NUMA nodes and 8192 CPU's, there's still 7 bits left (i.e. 6 with
>> CONFIG_NR_ZONES_EXTENDED).
>>
>> With the danger of becoming even more complex, could the limit also depend
>> on CONFIG_SPARSEMEM/VMEMMAP to reflect that somehow?
> 
> In this case it's already part of the equation because:
> 
> config ZONE_DEVICE
>        depends on MEMORY_HOTPLUG
>        depends on MEMORY_HOTREMOVE
> 
> ...and those in turn depend on SPARSEMEM.

Fine, but then SPARSEMEM_VMEMMAP should be still an available subvariant of
SPARSEMEM with SECTION_WIDTH=0.

[toc] | [prev] | [next] | [standalone]

#1346296

On Mon, Feb 29, 2016 at 4:06 PM, Vlastimil Babka <vbabka@suse.cz> wrote:
> On 29.2.2016 18:55, Dan Williams wrote:
>> On Mon, Feb 29, 2016 at 4:33 AM, Vlastimil Babka <vbabka@suse.cz> wrote:
>>> On 02/02/2016 06:42 AM, Andrew Morton wrote:
>>>> So if you want ZONE_DMA, you're limited to 512 NUMA nodes?
>>>>
>>>> That seems reasonable.
>>>
>>>
>>> Sorry for the late reply, but it seems that with !SPARSEMEM, or with
>>> SPARSEMEM_VMEMMAP, reducing NUMA nodes isn't even necessary, because
>>> SECTIONS_WIDTH is zero (see the diagrams in linux/page-flags-layout.h). In
>>> my brief tests with 4.4 based kernel with SPARSEMEM_VMEMMAP it seems that
>>> with 1024 NUMA nodes and 8192 CPU's, there's still 7 bits left (i.e. 6 with
>>> CONFIG_NR_ZONES_EXTENDED).
>>>
>>> With the danger of becoming even more complex, could the limit also depend
>>> on CONFIG_SPARSEMEM/VMEMMAP to reflect that somehow?
>>
>> In this case it's already part of the equation because:
>>
>> config ZONE_DEVICE
>>        depends on MEMORY_HOTPLUG
>>        depends on MEMORY_HOTREMOVE
>>
>> ...and those in turn depend on SPARSEMEM.
>
> Fine, but then SPARSEMEM_VMEMMAP should be still an available subvariant of
> SPARSEMEM with SECTION_WIDTH=0.

It should be, but not for the ZONE_DEVICE case.  ZONE_DEVICE depends
on x86_64 which means ZONE_DEVICE also implies SPARSEMEM_VMEMMAP
since:

config ARCH_SPARSEMEM_ENABLE
       def_bool y
       depends on X86_64 || NUMA || X86_32 || X86_32_NON_STANDARD
       select SPARSEMEM_STATIC if X86_32
       select SPARSEMEM_VMEMMAP_ENABLE if X86_64

Now, if a future patch wants to reclaim page flags space for other
usages outside of ZONE_DEVICE it can do the work to handle the
SPARSEMEM_VMEMMAP=n case.  I don't see a reason to fold that
distinction into the current patch given the current constraints.

[toc] | [prev] | [next] | [standalone]

#1346447

On 03/01/2016 03:06 AM, Dan Williams wrote:
> On Mon, Feb 29, 2016 at 4:06 PM, Vlastimil Babka <vbabka@suse.cz> wrote:
>> On 29.2.2016 18:55, Dan Williams wrote:
>>> On Mon, Feb 29, 2016 at 4:33 AM, Vlastimil Babka <vbabka@suse.cz> wrote:
>>>> On 02/02/2016 06:42 AM, Andrew Morton wrote:
>>>
>>> In this case it's already part of the equation because:
>>>
>>> config ZONE_DEVICE
>>>        depends on MEMORY_HOTPLUG
>>>        depends on MEMORY_HOTREMOVE
>>>
>>> ...and those in turn depend on SPARSEMEM.
>>
>> Fine, but then SPARSEMEM_VMEMMAP should be still an available subvariant of
>> SPARSEMEM with SECTION_WIDTH=0.
>
> It should be, but not for the ZONE_DEVICE case.  ZONE_DEVICE depends
> on x86_64 which means ZONE_DEVICE also implies SPARSEMEM_VMEMMAP
> since:
>
> config ARCH_SPARSEMEM_ENABLE
>         def_bool y
>         depends on X86_64 || NUMA || X86_32 || X86_32_NON_STANDARD
>         select SPARSEMEM_STATIC if X86_32
>         select SPARSEMEM_VMEMMAP_ENABLE if X86_64
>
> Now, if a future patch wants to reclaim page flags space for other
> usages outside of ZONE_DEVICE it can do the work to handle the
> SPARSEMEM_VMEMMAP=n case.  I don't see a reason to fold that
> distinction into the current patch given the current constraints.

OK so that IUUC shows that x86_64 should be always fine without decreasing the 
range for NODES_SHIFT? That's basically my point - since there's a configuration 
where things don't fit (32bit?), the patch broadly decreases range for 
NODES_SHIFT for everyone, right?

> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>

[toc] | [prev] | [next] | [standalone]

#1347119

On Tue, Mar 1, 2016 at 12:31 AM, Vlastimil Babka <vbabka@suse.cz> wrote:
> On 03/01/2016 03:06 AM, Dan Williams wrote:
>>
>> On Mon, Feb 29, 2016 at 4:06 PM, Vlastimil Babka <vbabka@suse.cz> wrote:
>>>
>>> On 29.2.2016 18:55, Dan Williams wrote:
>>>>
>>>> On Mon, Feb 29, 2016 at 4:33 AM, Vlastimil Babka <vbabka@suse.cz> wrote:
>>>>>
>>>>> On 02/02/2016 06:42 AM, Andrew Morton wrote:
>>>>
>>>>
>>>> In this case it's already part of the equation because:
>>>>
>>>> config ZONE_DEVICE
>>>>        depends on MEMORY_HOTPLUG
>>>>        depends on MEMORY_HOTREMOVE
>>>>
>>>> ...and those in turn depend on SPARSEMEM.
>>>
>>>
>>> Fine, but then SPARSEMEM_VMEMMAP should be still an available subvariant
>>> of
>>> SPARSEMEM with SECTION_WIDTH=0.
>>
>>
>> It should be, but not for the ZONE_DEVICE case.  ZONE_DEVICE depends
>> on x86_64 which means ZONE_DEVICE also implies SPARSEMEM_VMEMMAP
>> since:
>>
>> config ARCH_SPARSEMEM_ENABLE
>>         def_bool y
>>         depends on X86_64 || NUMA || X86_32 || X86_32_NON_STANDARD
>>         select SPARSEMEM_STATIC if X86_32
>>         select SPARSEMEM_VMEMMAP_ENABLE if X86_64
>>
>> Now, if a future patch wants to reclaim page flags space for other
>> usages outside of ZONE_DEVICE it can do the work to handle the
>> SPARSEMEM_VMEMMAP=n case.  I don't see a reason to fold that
>> distinction into the current patch given the current constraints.
>
>
> OK so that IUUC shows that x86_64 should be always fine without decreasing
> the range for NODES_SHIFT? That's basically my point - since there's a
> configuration where things don't fit (32bit?), the patch broadly decreases
> range for NODES_SHIFT for everyone, right?

So I went hunting for the x86_64 config that sent me off in this
direction in the first place, but I can't reproduce it.  I'm indeed
able to fit ZONE_DEVICE + ZONE_DMA + NODES_SHIFT(10) without
overflowing page flags.  Maybe we reduced some usage page->flags usage
between 4.3 and 4.5 and I missed it?

In any event, you're right we can indeed fit ZONE_DEVICE into the
current MAXSMP definition.  I'll respin the patch.

Thanks for probing on this!

[toc] | [prev] | [next] | [standalone]

#1347842

On 03/02/2016 12:43 AM, Dan Williams wrote:
> On Tue, Mar 1, 2016 at 12:31 AM, Vlastimil Babka <vbabka@suse.cz> wrote:
>> On 03/01/2016 03:06 AM, Dan Williams wrote:
>>>
>>> On Mon, Feb 29, 2016 at 4:06 PM, Vlastimil Babka <vbabka@suse.cz> wrote:
>>>>
>>>> On 29.2.2016 18:55, Dan Williams wrote:
>>>>>
>>>>> On Mon, Feb 29, 2016 at 4:33 AM, Vlastimil Babka <vbabka@suse.cz> wrote:
>>>>>>
>>>>>> On 02/02/2016 06:42 AM, Andrew Morton wrote:
>>>>>
>>>>>
>>>>> In this case it's already part of the equation because:
>>>>>
>>>>> config ZONE_DEVICE
>>>>>         depends on MEMORY_HOTPLUG
>>>>>         depends on MEMORY_HOTREMOVE
>>>>>
>>>>> ...and those in turn depend on SPARSEMEM.
>>>>
>>>>
>>>> Fine, but then SPARSEMEM_VMEMMAP should be still an available subvariant
>>>> of
>>>> SPARSEMEM with SECTION_WIDTH=0.
>>>
>>>
>>> It should be, but not for the ZONE_DEVICE case.  ZONE_DEVICE depends
>>> on x86_64 which means ZONE_DEVICE also implies SPARSEMEM_VMEMMAP
>>> since:
>>>
>>> config ARCH_SPARSEMEM_ENABLE
>>>          def_bool y
>>>          depends on X86_64 || NUMA || X86_32 || X86_32_NON_STANDARD
>>>          select SPARSEMEM_STATIC if X86_32
>>>          select SPARSEMEM_VMEMMAP_ENABLE if X86_64
>>>
>>> Now, if a future patch wants to reclaim page flags space for other
>>> usages outside of ZONE_DEVICE it can do the work to handle the
>>> SPARSEMEM_VMEMMAP=n case.  I don't see a reason to fold that
>>> distinction into the current patch given the current constraints.
>>
>>
>> OK so that IUUC shows that x86_64 should be always fine without decreasing
>> the range for NODES_SHIFT? That's basically my point - since there's a
>> configuration where things don't fit (32bit?), the patch broadly decreases
>> range for NODES_SHIFT for everyone, right?
>
> So I went hunting for the x86_64 config that sent me off in this
> direction in the first place, but I can't reproduce it.  I'm indeed
> able to fit ZONE_DEVICE + ZONE_DMA + NODES_SHIFT(10) without
> overflowing page flags.  Maybe we reduced some usage page->flags usage
> between 4.3 and 4.5 and I missed it?

Oh, I think I see it now. SPARSEMEM_VMEMMAP_ENABLE only *allows to 
enable* CONFIG_SPARSEMEM_VMEMMAP, it doesn't force it:

config SPARSEMEM_VMEMMAP
         bool "Sparse Memory virtual memmap"
         depends on SPARSEMEM && SPARSEMEM_VMEMMAP_ENABLE
         default y

> In any event, you're right we can indeed fit ZONE_DEVICE into the
> current MAXSMP definition.  I'll respin the patch.

But I still believe that that your respin is better than this variant. 
We shouldn't broadly limit the range in one of the options, when there 
are multiple options affecting the usage of bits. There's a warning if 
the overal configuration is "too large", which could potentially be more 
detailed. But we never said configuring the kernel is trivial ;-)

Also in this case the "default y" for SPARSEMEM_VMEMMAP should prevent 
surprise when one enables ZONE_DEVICE through nvdimm and doesn't fiddle 
with the lowlevel details. As long as it takes multiple explicit choices 
differing from defaults to get to the warning, I'd say we are fine.

> Thanks for probing on this!
>

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web