Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64

From	Ard Biesheuvel <ard.biesheuvel@linaro.org>
Newsgroups	linux.kernel
Subject	Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64
Date	2016-04-11 12:40 +0200
Message-ID	<rmHqa-4eH-19@gated-at.bofh.it> (permalink)
References	(2 earlier) <rmAf0-75P-17@gated-at.bofh.it> <rmEBY-26A-7@gated-at.bofh.it> <rmEVl-2eN-15@gated-at.bofh.it> <rmF50-2CR-13@gated-at.bofh.it> <rmGX8-3Zo-9@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

On 11 April 2016 at 11:59, Chen Feng <puck.chen@hisilicon.com> wrote:
> Hi Ard,
>
> On 2016/4/11 16:00, Ard Biesheuvel wrote:
>> On 11 April 2016 at 09:55, Chen Feng <puck.chen@hisilicon.com> wrote:
>>> Hi Ard,
>>>
>>> On 2016/4/11 15:35, Ard Biesheuvel wrote:
>>>> On 11 April 2016 at 04:49, Chen Feng <puck.chen@hisilicon.com> wrote:
>>>>> Hi will,
>>>>> Thanks for review.
>>>>>
>>>>> On 2016/4/7 22:21, Will Deacon wrote:
>>>>>> On Tue, Apr 05, 2016 at 04:22:51PM +0800, Chen Feng wrote:
>>>>>>> We can reduce the memory allocated at mem-map
>>>>>>> by flatmem.
>>>>>>>
>>>>>>> currently, the default memory-model in arm64 is
>>>>>>> sparse memory. The mem-map array is not freed in
>>>>>>> this scene. If the physical address is too long,
>>>>>>> it will reserved too much memory for the mem-map
>>>>>>> array.
>>>>>>
>>>>>> Can you elaborate a bit more on this, please? We use the vmemmap, so any
>>>>>> spaces between memory banks only burns up virtual space. What exactly is
>>>>>> the problem you're seeing that makes you want to use flatmem (which is
>>>>>> probably unsuitable for the majority of arm64 machines).
>>>>>>
>>>>> The root cause we want to use flat-mem is the mam_map alloced in sparse-mem
>>>>> is not freed.
>>>>>
>>>>> take a look at here:
>>>>> arm64/mm/init.c
>>>>> void __init mem_init(void)
>>>>> {
>>>>> #ifndef CONFIG_SPARSEMEM_VMEMMAP
>>>>>         free_unused_memmap();
>>>>> #endif
>>>>> }
>>>>>
>>>>> Memory layout (3GB)
>>>>>
>>>>>  0             1.5G    2G             3.5G            4G
>>>>>  |              |      |               |              |
>>>>>  +--------------+------+---------------+--------------+
>>>>>  |    MEM       | hole |     MEM       |   IO (regs)  |
>>>>>  +--------------+------+---------------+--------------+
>>>>>
>>>>>
>>>>> Memory layout (4GB)
>>>>>
>>>>>  0                                    3.5G            4G    4.5G
>>>>>  |                                     |              |       |
>>>>>  +-------------------------------------+--------------+-------+
>>>>>  |                   MEM               |   IO (regs)  |  MEM  |
>>>>>  +-------------------------------------+--------------+-------+
>>>>>
>>>>> Currently, the sparse memory section is 1GB.
>>>>>
>>>>> 3GB ddr: the 1.5 ~2G and 3.5 ~ 4G are holes.
>>>>> 3GB ddr: the 3.5 ~ 4G and 4.5 ~ 5G are holes.
>>>>>
>>>>> This will alloc 1G/4K * (struct page) memory for mem_map array.
>>>>>
>>>>
>>>> No, this is incorrect. Sparsemem vmemmap only allocates struct pages
>>>> for memory regions that are actually populated.
>>>>
>>>> For instance, on the Foundation model with 4 GB of memory, you may see
>>>> something like this in the boot log
>>>>
>>>> [    0.000000]     vmemmap : 0xffffffbdc0000000 - 0xffffffbfc0000000
>>>> (     8 GB maximum)
>>>> [    0.000000]               0xffffffbdc0000000 - 0xffffffbde2000000
>>>> (   544 MB actual)
>>>>
>>>> but in reality, only the following regions have been allocated
>>>>
>>>> ---[ vmemmap start ]---
>>>> 0xffffffbdc0000000-0xffffffbdc2000000          32M       RW NX SHD AF
>>>>       BLK UXN MEM/NORMAL
>>>> 0xffffffbde0000000-0xffffffbde2000000          32M       RW NX SHD AF
>>>>       BLK UXN MEM/NORMAL
>>>> ---[ vmemmap end ]---
>>>>
>>>> so only 64 MB is used to back 4 GB of RAM with struct pages, which is
>>>> minimal. Moving to flatmem will not reduce the memory footprint at
>>>> all.
>>>
>>> Yes,but the populate is section, which is 1GB. Take a look at the above
>>> memory layout.
>>>
>>> The section 1G ~ 2G is a section. But 1.5G ~ 2G is a hole.
>>>
>>> The section 3G ~ 4G is a section. But 3.5G ~ 4G is a hole.
>>>>>  0             1.5G    2G             3.5G            4G
>>>>>  |              |      |               |              |
>>>>>  +--------------+------+---------------+--------------+
>>>>>  |    MEM       | hole |     MEM       |   IO (regs)  |
>>>>>  +--------------+------+---------------+--------------+
>>> The hole in 1.5G ~ 2G is also allocated mem-map array. And also with the 3.5G ~ 4G.
>>>
>>
>> No, it is not. It may be covered by a section, but that does not mean
>> sparsemem vmemmap will actually allocate backing for it. The
>> granularity used by sparsemem vmemmap on a 4k pages kernel is 128 MB,
>> due to the fact that the backing is performed at PMD granularity.
>>
>> Please, could you share the contents of the vmemmap section in
>> /sys/kernel/debug/kernel_page_tables of your system running with
>> sparsemem vmemmap enabled? You will need to set CONFIG_ARM64_PTDUMP=y
>>
>
> Please see the pg-tables below.
>
>
> With sparse and vmemmap enable.
>
> ---[ vmemmap start ]---
> 0xffffffbdc0200000-0xffffffbdc4800000          70M     RW NX SHD AF    UXN MEM/NORMAL
> ---[ vmemmap end ]---
>

OK, I see what you mean now. Sorry for taking so long to catch up.

> The board is 4GB, and the memap is 70MB
> 1G memory --- 14MB mem_map array.

No, this is incorrect. 1 GB corresponds with 16 MB worth of struct
pages assuming sizeof(struct page) == 64

So you are losing 6 MB to rounding here, which I agree is significant.
I wonder if it makes sense to use a lower value for SECTION_SIZE_BITS
on 4k pages kernels, but perhaps we're better off asking the opinion
of the other cc'ees.

Thanks,
Ard.

Thread

[PATCH 1/2] arm64: mem-model: add flatmem model for arm64 Chen Feng <puck.chen@hisilicon.com> - 2016-04-05 10:30 +0200
  Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64 Chen Feng <puck.chen@hisilicon.com> - 2016-04-07 09:50 +0200
  Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64 Will Deacon <will.deacon@arm.com> - 2016-04-07 16:30 +0200
    Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64 Chen Feng <puck.chen@hisilicon.com> - 2016-04-11 05:00 +0200
      Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64 Ard Biesheuvel <ard.biesheuvel@linaro.org> - 2016-04-11 09:40 +0200
        Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64 Chen Feng <puck.chen@hisilicon.com> - 2016-04-11 10:00 +0200
          Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64 Ard Biesheuvel <ard.biesheuvel@linaro.org> - 2016-04-11 10:10 +0200
            Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64 Chen Feng <puck.chen@hisilicon.com> - 2016-04-11 12:10 +0200
              Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64 Ard Biesheuvel <ard.biesheuvel@linaro.org> - 2016-04-11 12:40 +0200
                Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64 Will Deacon <will.deacon@arm.com> - 2016-04-11 12:50 +0200
                Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64 Chen Feng <puck.chen@hisilicon.com> - 2016-04-11 13:00 +0200
                Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64 Laura Abbott <labbott@redhat.com> - 2016-04-11 20:20 +0200
                Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64 Catalin Marinas <catalin.marinas@arm.com> - 2016-04-12 16:50 +0200
                Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64 Catalin Marinas <catalin.marinas@arm.com> - 2016-04-12 17:00 +0200
              Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64 Chen Feng <puck.chen@hisilicon.com> - 2016-04-11 12:50 +0200
                Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64 Ard Biesheuvel <ard.biesheuvel@linaro.org> - 2016-04-11 13:10 +0200
      Re: [PATCH 1/2] arm64: mem-model: add flatmem model for arm64 Jungseok Lee <jungseoklee85@gmail.com> - 2016-04-12 16:10 +0200

csiph-web