Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1206489 > unrolled thread

[RFC PATCH 1/7] x86, mm: ZONE_DEVICE for "device memory"

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  [RFC PATCH 1/7] x86, mm: ZONE_DEVICE for "device memory" Dan Williams <dan.j.williams@intel.com> - 2015-08-13 06:00 +0200
    Re: [RFC PATCH 1/7] x86, mm: ZONE_DEVICE for "device memory" Jerome Glisse <j.glisse@gmail.com> - 2015-08-14 23:40 +0200
      Re: [RFC PATCH 1/7] x86, mm: ZONE_DEVICE for "device memory" Dan Williams <dan.j.williams@intel.com> - 2015-08-15 00:00 +0200
        Re: [RFC PATCH 1/7] x86, mm: ZONE_DEVICE for "device memory" Jerome Glisse <j.glisse@gmail.com> - 2015-08-15 00:10 +0200
          Re: [RFC PATCH 1/7] x86, mm: ZONE_DEVICE for "device memory" Dan Williams <dan.j.williams@intel.com> - 2015-08-15 00:40 +0200
            Re: [RFC PATCH 1/7] x86, mm: ZONE_DEVICE for "device memory" Dan Williams <dan.j.williams@intel.com> - 2015-08-15 04:20 +0200
              Re: [RFC PATCH 1/7] x86, mm: ZONE_DEVICE for "device memory" Jerome Glisse <j.glisse@gmail.com> - 2015-08-17 23:50 +0200
                Re: [RFC PATCH 1/7] x86, mm: ZONE_DEVICE for "device memory" Dan Williams <dan.j.williams@intel.com> - 2015-08-18 02:50 +0200
                  Re: [RFC PATCH 1/7] x86, mm: ZONE_DEVICE for "device memory" Jerome Glisse <j.glisse@gmail.com> - 2015-08-18 19:00 +0200
                    Re: [RFC PATCH 1/7] x86, mm: ZONE_DEVICE for "device memory" Dan Williams <dan.j.williams@intel.com> - 2015-08-18 19:30 +0200
                      Re: [RFC PATCH 1/7] x86, mm: ZONE_DEVICE for "device memory" Jerome Glisse <j.glisse@gmail.com> - 2015-08-18 21:10 +0200
        Re: [RFC PATCH 1/7] x86, mm: ZONE_DEVICE for "device memory" Christoph Hellwig <hch@lst.de> - 2015-08-15 11:10 +0200
    Re: [RFC PATCH 1/7] x86, mm: ZONE_DEVICE for "device memory" Christoph Hellwig <hch@lst.de> - 2015-08-15 15:40 +0200

#1206489 — [RFC PATCH 1/7] x86, mm: ZONE_DEVICE for "device memory"

While pmem is usable as a block device or via DAX mappings to userspace
there are several usage scenarios that can not target pmem due to its
lack of struct page coverage. In preparation for "hot plugging" pmem
into the vmemmap add ZONE_DEVICE as a new zone to tag these pages
separately from the ones that are subject to standard page allocations.
Importantly "device memory" can be removed at will by userspace
unbinding the driver of the device.

Having a separate zone prevents allocation and otherwise marks these
pages that are distinct from typical uniform memory.  Device memory has
different lifetime and performance characteristics than RAM.  However,
since we have run out of ZONES_SHIFT bits this functionality currently
depends on sacrificing ZONE_DMA.

arch_add_memory() is reorganized a bit in preparation for a new
arch_add_dev_memory() api, for now there is no functional change to the
memory hotplug code.

Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: linux-mm@kvack.org
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
 arch/x86/Kconfig       |   13 +++++++++++++
 arch/x86/mm/init_64.c  |   32 +++++++++++++++++++++-----------
 include/linux/mmzone.h |   23 +++++++++++++++++++++++
 mm/memory_hotplug.c    |    5 ++++-
 mm/page_alloc.c        |    3 +++
 5 files changed, 64 insertions(+), 12 deletions(-)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index b3a1a5d77d92..64829b17980b 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -308,6 +308,19 @@ config ZONE_DMA
 
 	  If unsure, say Y.
 
+config ZONE_DEVICE
+	bool "Device memory (pmem, etc...) hotplug support" if EXPERT
+	default !ZONE_DMA
+	depends on !ZONE_DMA
+	help
+	  Device memory hotplug support allows for establishing pmem,
+	  or other device driver discovered memory regions, in the
+	  memmap. This allows pfn_to_page() lookups of otherwise
+	  "device-physical" addresses which is needed for using a DAX
+	  mapping in an O_DIRECT operation, among other things.
+
+	  If FS_DAX is enabled, then say Y.
+
 config SMP
 	bool "Symmetric multi-processing support"
 	---help---
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 3fba623e3ba5..94f0fa56f0ed 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -683,15 +683,8 @@ static void  update_end_of_memory_vars(u64 start, u64 size)
 	}
 }
 
-/*
- * Memory is added always to NORMAL zone. This means you will never get
- * additional DMA/DMA32 memory.
- */
-int arch_add_memory(int nid, u64 start, u64 size)
+static int __arch_add_memory(int nid, u64 start, u64 size, struct zone *zone)
 {
-	struct pglist_data *pgdat = NODE_DATA(nid);
-	struct zone *zone = pgdat->node_zones +
-		zone_for_memory(nid, start, size, ZONE_NORMAL);
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
 	int ret;
@@ -701,11 +694,28 @@ int arch_add_memory(int nid, u64 start, u64 size)
 	ret = __add_pages(nid, zone, start_pfn, nr_pages);
 	WARN_ON_ONCE(ret);
 
-	/* update max_pfn, max_low_pfn and high_memory */
-	update_end_of_memory_vars(start, size);
+	/*
+	 * Update max_pfn, max_low_pfn and high_memory, unless we added
+	 * "device memory" which should not effect max_pfn
+	 */
+	if (!is_dev_zone(zone))
+		update_end_of_memory_vars(start, size);
 
 	return ret;
 }
+
+/*
+ * Memory is added always to NORMAL zone. This means you will never get
+ * additional DMA/DMA32 memory.
+ */
+int arch_add_memory(int nid, u64 start, u64 size)
+{
+	struct pglist_data *pgdat = NODE_DATA(nid);
+	struct zone *zone = pgdat->node_zones +
+		zone_for_memory(nid, start, size, ZONE_NORMAL);
+
+	return __arch_add_memory(nid, start, size, zone);
+}
 EXPORT_SYMBOL_GPL(arch_add_memory);
 
 #define PAGE_INUSE 0xFD
@@ -1028,7 +1038,7 @@ int __ref arch_remove_memory(u64 start, u64 size)
 
 	return ret;
 }
-#endif
+#endif /* CONFIG_MEMORY_HOTREMOVE */
 #endif /* CONFIG_MEMORY_HOTPLUG */
 
 static struct kcore_list kcore_vsyscall;
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 754c25966a0a..9217fd93c25b 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -319,7 +319,11 @@ enum zone_type {
 	ZONE_HIGHMEM,
 #endif
 	ZONE_MOVABLE,
+#ifdef CONFIG_ZONE_DEVICE
+	ZONE_DEVICE,
+#endif
 	__MAX_NR_ZONES
+
 };
 
 #ifndef __GENERATING_BOUNDS_H
@@ -794,6 +798,25 @@ static inline bool pgdat_is_empty(pg_data_t *pgdat)
 	return !pgdat->node_start_pfn && !pgdat->node_spanned_pages;
 }
 
+static inline int zone_id(const struct zone *zone)
+{
+	struct pglist_data *pgdat = zone->zone_pgdat;
+
+	return zone - pgdat->node_zones;
+}
+
+#ifdef CONFIG_ZONE_DEVICE
+static inline bool is_dev_zone(const struct zone *zone)
+{
+	return zone_id(zone) == ZONE_DEVICE;
+}
+#else
+static inline bool is_dev_zone(const struct zone *zone)
+{
+	return false;
+}
+#endif
+
 #include <linux/memory_hotplug.h>
 
 extern struct mutex zonelists_mutex;
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 26fbba7d888f..6bc5b755ce98 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -770,7 +770,10 @@ int __remove_pages(struct zone *zone, unsigned long phys_start_pfn,
 
 	start = phys_start_pfn << PAGE_SHIFT;
 	size = nr_pages * PAGE_SIZE;
-	ret = release_mem_region_adjustable(&iomem_resource, start, size);
+
+	/* in the ZONE_DEVICE case device driver owns the memory region */
+	if (!is_dev_zone(zone))
+		ret = release_mem_region_adjustable(&iomem_resource, start, size);
 	if (ret) {
 		resource_size_t endres = start + size - 1;
 
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index ef19f22b2b7d..0f19b4e18233 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -207,6 +207,9 @@ static char * const zone_names[MAX_NR_ZONES] = {
 	 "HighMem",
 #endif
 	 "Movable",
+#ifdef CONFIG_ZONE_DEVICE
+	 "Device",
+#endif
 };
 
 int min_free_kbytes = 1024;

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1207913

On Wed, Aug 12, 2015 at 11:50:05PM -0400, Dan Williams wrote:
> While pmem is usable as a block device or via DAX mappings to userspace
> there are several usage scenarios that can not target pmem due to its
> lack of struct page coverage. In preparation for "hot plugging" pmem
> into the vmemmap add ZONE_DEVICE as a new zone to tag these pages
> separately from the ones that are subject to standard page allocations.
> Importantly "device memory" can be removed at will by userspace
> unbinding the driver of the device.
> 
> Having a separate zone prevents allocation and otherwise marks these
> pages that are distinct from typical uniform memory.  Device memory has
> different lifetime and performance characteristics than RAM.  However,
> since we have run out of ZONES_SHIFT bits this functionality currently
> depends on sacrificing ZONE_DMA.
> 
> arch_add_memory() is reorganized a bit in preparation for a new
> arch_add_dev_memory() api, for now there is no functional change to the
> memory hotplug code.
> 
> Cc: H. Peter Anvin <hpa@zytor.com>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Dave Hansen <dave.hansen@linux.intel.com>
> Cc: Rik van Riel <riel@redhat.com>
> Cc: Mel Gorman <mgorman@suse.de>
> Cc: linux-mm@kvack.org
> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
> ---
>  arch/x86/Kconfig       |   13 +++++++++++++
>  arch/x86/mm/init_64.c  |   32 +++++++++++++++++++++-----------
>  include/linux/mmzone.h |   23 +++++++++++++++++++++++
>  mm/memory_hotplug.c    |    5 ++++-
>  mm/page_alloc.c        |    3 +++
>  5 files changed, 64 insertions(+), 12 deletions(-)
> 
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index b3a1a5d77d92..64829b17980b 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -308,6 +308,19 @@ config ZONE_DMA
>  
>  	  If unsure, say Y.
>  
> +config ZONE_DEVICE
> +	bool "Device memory (pmem, etc...) hotplug support" if EXPERT
> +	default !ZONE_DMA
> +	depends on !ZONE_DMA
> +	help
> +	  Device memory hotplug support allows for establishing pmem,
> +	  or other device driver discovered memory regions, in the
> +	  memmap. This allows pfn_to_page() lookups of otherwise
> +	  "device-physical" addresses which is needed for using a DAX
> +	  mapping in an O_DIRECT operation, among other things.
> +
> +	  If FS_DAX is enabled, then say Y.
> +
>  config SMP
>  	bool "Symmetric multi-processing support"
>  	---help---
> diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
> index 3fba623e3ba5..94f0fa56f0ed 100644
> --- a/arch/x86/mm/init_64.c
> +++ b/arch/x86/mm/init_64.c
> @@ -683,15 +683,8 @@ static void  update_end_of_memory_vars(u64 start, u64 size)
>  	}
>  }
>  
> -/*
> - * Memory is added always to NORMAL zone. This means you will never get
> - * additional DMA/DMA32 memory.
> - */
> -int arch_add_memory(int nid, u64 start, u64 size)
> +static int __arch_add_memory(int nid, u64 start, u64 size, struct zone *zone)
>  {
> -	struct pglist_data *pgdat = NODE_DATA(nid);
> -	struct zone *zone = pgdat->node_zones +
> -		zone_for_memory(nid, start, size, ZONE_NORMAL);
>  	unsigned long start_pfn = start >> PAGE_SHIFT;
>  	unsigned long nr_pages = size >> PAGE_SHIFT;
>  	int ret;
> @@ -701,11 +694,28 @@ int arch_add_memory(int nid, u64 start, u64 size)
>  	ret = __add_pages(nid, zone, start_pfn, nr_pages);
>  	WARN_ON_ONCE(ret);
>  
> -	/* update max_pfn, max_low_pfn and high_memory */
> -	update_end_of_memory_vars(start, size);
> +	/*
> +	 * Update max_pfn, max_low_pfn and high_memory, unless we added
> +	 * "device memory" which should not effect max_pfn
> +	 */
> +	if (!is_dev_zone(zone))
> +		update_end_of_memory_vars(start, size);

What is the rational for not updating max_pfn, max_low_pfn, ... ?

Cheers,
Jérôme
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1207915

On Fri, Aug 14, 2015 at 2:37 PM, Jerome Glisse <j.glisse@gmail.com> wrote:
> On Wed, Aug 12, 2015 at 11:50:05PM -0400, Dan Williams wrote:
>> While pmem is usable as a block device or via DAX mappings to userspace
>> there are several usage scenarios that can not target pmem due to its
>> lack of struct page coverage. In preparation for "hot plugging" pmem
>> into the vmemmap add ZONE_DEVICE as a new zone to tag these pages
>> separately from the ones that are subject to standard page allocations.
>> Importantly "device memory" can be removed at will by userspace
>> unbinding the driver of the device.
>>
>> Having a separate zone prevents allocation and otherwise marks these
>> pages that are distinct from typical uniform memory.  Device memory has
>> different lifetime and performance characteristics than RAM.  However,
>> since we have run out of ZONES_SHIFT bits this functionality currently
>> depends on sacrificing ZONE_DMA.
>>
>> arch_add_memory() is reorganized a bit in preparation for a new
>> arch_add_dev_memory() api, for now there is no functional change to the
>> memory hotplug code.
>>
>> Cc: H. Peter Anvin <hpa@zytor.com>
>> Cc: Ingo Molnar <mingo@redhat.com>
>> Cc: Dave Hansen <dave.hansen@linux.intel.com>
>> Cc: Rik van Riel <riel@redhat.com>
>> Cc: Mel Gorman <mgorman@suse.de>
>> Cc: linux-mm@kvack.org
>> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
>> ---
>>  arch/x86/Kconfig       |   13 +++++++++++++
>>  arch/x86/mm/init_64.c  |   32 +++++++++++++++++++++-----------
>>  include/linux/mmzone.h |   23 +++++++++++++++++++++++
>>  mm/memory_hotplug.c    |    5 ++++-
>>  mm/page_alloc.c        |    3 +++
>>  5 files changed, 64 insertions(+), 12 deletions(-)
>>
>> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
>> index b3a1a5d77d92..64829b17980b 100644
>> --- a/arch/x86/Kconfig
>> +++ b/arch/x86/Kconfig
>> @@ -308,6 +308,19 @@ config ZONE_DMA
>>
>>         If unsure, say Y.
>>
>> +config ZONE_DEVICE
>> +     bool "Device memory (pmem, etc...) hotplug support" if EXPERT
>> +     default !ZONE_DMA
>> +     depends on !ZONE_DMA
>> +     help
>> +       Device memory hotplug support allows for establishing pmem,
>> +       or other device driver discovered memory regions, in the
>> +       memmap. This allows pfn_to_page() lookups of otherwise
>> +       "device-physical" addresses which is needed for using a DAX
>> +       mapping in an O_DIRECT operation, among other things.
>> +
>> +       If FS_DAX is enabled, then say Y.
>> +
>>  config SMP
>>       bool "Symmetric multi-processing support"
>>       ---help---
>> diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
>> index 3fba623e3ba5..94f0fa56f0ed 100644
>> --- a/arch/x86/mm/init_64.c
>> +++ b/arch/x86/mm/init_64.c
[..]
>> @@ -701,11 +694,28 @@ int arch_add_memory(int nid, u64 start, u64 size)
>>       ret = __add_pages(nid, zone, start_pfn, nr_pages);
>>       WARN_ON_ONCE(ret);
>>
>> -     /* update max_pfn, max_low_pfn and high_memory */
>> -     update_end_of_memory_vars(start, size);
>> +     /*
>> +      * Update max_pfn, max_low_pfn and high_memory, unless we added
>> +      * "device memory" which should not effect max_pfn
>> +      */
>> +     if (!is_dev_zone(zone))
>> +             update_end_of_memory_vars(start, size);
>
> What is the rational for not updating max_pfn, max_low_pfn, ... ?
>

The idea is that this memory is not meant to be available to the page
allocator and should not count as new memory capacity.  We're only
hotplugging it to get struct page coverage.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1207921

On Fri, Aug 14, 2015 at 02:52:15PM -0700, Dan Williams wrote:
> On Fri, Aug 14, 2015 at 2:37 PM, Jerome Glisse <j.glisse@gmail.com> wrote:
> > On Wed, Aug 12, 2015 at 11:50:05PM -0400, Dan Williams wrote:
> >> While pmem is usable as a block device or via DAX mappings to userspace
> >> there are several usage scenarios that can not target pmem due to its
> >> lack of struct page coverage. In preparation for "hot plugging" pmem
> >> into the vmemmap add ZONE_DEVICE as a new zone to tag these pages
> >> separately from the ones that are subject to standard page allocations.
> >> Importantly "device memory" can be removed at will by userspace
> >> unbinding the driver of the device.
> >>
> >> Having a separate zone prevents allocation and otherwise marks these
> >> pages that are distinct from typical uniform memory.  Device memory has
> >> different lifetime and performance characteristics than RAM.  However,
> >> since we have run out of ZONES_SHIFT bits this functionality currently
> >> depends on sacrificing ZONE_DMA.
> >>
> >> arch_add_memory() is reorganized a bit in preparation for a new
> >> arch_add_dev_memory() api, for now there is no functional change to the
> >> memory hotplug code.
> >>
> >> Cc: H. Peter Anvin <hpa@zytor.com>
> >> Cc: Ingo Molnar <mingo@redhat.com>
> >> Cc: Dave Hansen <dave.hansen@linux.intel.com>
> >> Cc: Rik van Riel <riel@redhat.com>
> >> Cc: Mel Gorman <mgorman@suse.de>
> >> Cc: linux-mm@kvack.org
> >> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
> >> ---
> >>  arch/x86/Kconfig       |   13 +++++++++++++
> >>  arch/x86/mm/init_64.c  |   32 +++++++++++++++++++++-----------
> >>  include/linux/mmzone.h |   23 +++++++++++++++++++++++
> >>  mm/memory_hotplug.c    |    5 ++++-
> >>  mm/page_alloc.c        |    3 +++
> >>  5 files changed, 64 insertions(+), 12 deletions(-)
> >>
> >> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> >> index b3a1a5d77d92..64829b17980b 100644
> >> --- a/arch/x86/Kconfig
> >> +++ b/arch/x86/Kconfig
> >> @@ -308,6 +308,19 @@ config ZONE_DMA
> >>
> >>         If unsure, say Y.
> >>
> >> +config ZONE_DEVICE
> >> +     bool "Device memory (pmem, etc...) hotplug support" if EXPERT
> >> +     default !ZONE_DMA
> >> +     depends on !ZONE_DMA
> >> +     help
> >> +       Device memory hotplug support allows for establishing pmem,
> >> +       or other device driver discovered memory regions, in the
> >> +       memmap. This allows pfn_to_page() lookups of otherwise
> >> +       "device-physical" addresses which is needed for using a DAX
> >> +       mapping in an O_DIRECT operation, among other things.
> >> +
> >> +       If FS_DAX is enabled, then say Y.
> >> +
> >>  config SMP
> >>       bool "Symmetric multi-processing support"
> >>       ---help---
> >> diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
> >> index 3fba623e3ba5..94f0fa56f0ed 100644
> >> --- a/arch/x86/mm/init_64.c
> >> +++ b/arch/x86/mm/init_64.c
> [..]
> >> @@ -701,11 +694,28 @@ int arch_add_memory(int nid, u64 start, u64 size)
> >>       ret = __add_pages(nid, zone, start_pfn, nr_pages);
> >>       WARN_ON_ONCE(ret);
> >>
> >> -     /* update max_pfn, max_low_pfn and high_memory */
> >> -     update_end_of_memory_vars(start, size);
> >> +     /*
> >> +      * Update max_pfn, max_low_pfn and high_memory, unless we added
> >> +      * "device memory" which should not effect max_pfn
> >> +      */
> >> +     if (!is_dev_zone(zone))
> >> +             update_end_of_memory_vars(start, size);
> >
> > What is the rational for not updating max_pfn, max_low_pfn, ... ?
> >
> 
> The idea is that this memory is not meant to be available to the page
> allocator and should not count as new memory capacity.  We're only
> hotplugging it to get struct page coverage.

But this sounds bogus to me to rely on max_pfn to stay smaller than
first_dev_pfn.  For instance you might plug a device that register
dev memory and then some regular memory might be hotplug, effectively
updating max_pfn to a value bigger than first_dev_pfn.

Also i do not think that the buddy allocator use max_pfn or max_low_pfn
to consider page/zone for allocation or not.

Cheers,
Jérôme
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1207926

On Fri, Aug 14, 2015 at 3:06 PM, Jerome Glisse <j.glisse@gmail.com> wrote:
> On Fri, Aug 14, 2015 at 02:52:15PM -0700, Dan Williams wrote:
>> On Fri, Aug 14, 2015 at 2:37 PM, Jerome Glisse <j.glisse@gmail.com> wrote:
>> > On Wed, Aug 12, 2015 at 11:50:05PM -0400, Dan Williams wrote:
[..]
>> > What is the rational for not updating max_pfn, max_low_pfn, ... ?
>> >
>>
>> The idea is that this memory is not meant to be available to the page
>> allocator and should not count as new memory capacity.  We're only
>> hotplugging it to get struct page coverage.
>
> But this sounds bogus to me to rely on max_pfn to stay smaller than
> first_dev_pfn.  For instance you might plug a device that register
> dev memory and then some regular memory might be hotplug, effectively
> updating max_pfn to a value bigger than first_dev_pfn.
>

True.

> Also i do not think that the buddy allocator use max_pfn or max_low_pfn
> to consider page/zone for allocation or not.

Yes, I took it out with no effects.  I'll investigate further whether
we should be touching those variables or not for this new usage.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1207978

On Fri, Aug 14, 2015 at 3:33 PM, Dan Williams <dan.j.williams@intel.com> wrote:
> On Fri, Aug 14, 2015 at 3:06 PM, Jerome Glisse <j.glisse@gmail.com> wrote:
>> On Fri, Aug 14, 2015 at 02:52:15PM -0700, Dan Williams wrote:
>>> On Fri, Aug 14, 2015 at 2:37 PM, Jerome Glisse <j.glisse@gmail.com> wrote:
>>> > On Wed, Aug 12, 2015 at 11:50:05PM -0400, Dan Williams wrote:
> [..]
>>> > What is the rational for not updating max_pfn, max_low_pfn, ... ?
>>> >
>>>
>>> The idea is that this memory is not meant to be available to the page
>>> allocator and should not count as new memory capacity.  We're only
>>> hotplugging it to get struct page coverage.
>>
>> But this sounds bogus to me to rely on max_pfn to stay smaller than
>> first_dev_pfn.  For instance you might plug a device that register
>> dev memory and then some regular memory might be hotplug, effectively
>> updating max_pfn to a value bigger than first_dev_pfn.
>>
>
> True.
>
>> Also i do not think that the buddy allocator use max_pfn or max_low_pfn
>> to consider page/zone for allocation or not.
>
> Yes, I took it out with no effects.  I'll investigate further whether
> we should be touching those variables or not for this new usage.

Although it does not offer perfect protection if device memory is at a
physically lower address than RAM, skipping the update of these
variables does seem to be what we want.  For example /dev/mem would
fail to allow write access to persistent memory if it fails a
valid_phys_addr_range() check.  Since /dev/mem does not know how to
write to PMEM in a reliably persistent way, it should not treat a
PMEM-pfn like RAM.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1208907

[Multipart message — attachments visible in raw view] — view raw

On Fri, Aug 14, 2015 at 07:11:27PM -0700, Dan Williams wrote:
> On Fri, Aug 14, 2015 at 3:33 PM, Dan Williams <dan.j.williams@intel.com> wrote:
> > On Fri, Aug 14, 2015 at 3:06 PM, Jerome Glisse <j.glisse@gmail.com> wrote:
> >> On Fri, Aug 14, 2015 at 02:52:15PM -0700, Dan Williams wrote:
> >>> On Fri, Aug 14, 2015 at 2:37 PM, Jerome Glisse <j.glisse@gmail.com> wrote:
> >>> > On Wed, Aug 12, 2015 at 11:50:05PM -0400, Dan Williams wrote:
> > [..]
> >>> > What is the rational for not updating max_pfn, max_low_pfn, ... ?
> >>> >
> >>>
> >>> The idea is that this memory is not meant to be available to the page
> >>> allocator and should not count as new memory capacity.  We're only
> >>> hotplugging it to get struct page coverage.
> >>
> >> But this sounds bogus to me to rely on max_pfn to stay smaller than
> >> first_dev_pfn.  For instance you might plug a device that register
> >> dev memory and then some regular memory might be hotplug, effectively
> >> updating max_pfn to a value bigger than first_dev_pfn.
> >>
> >
> > True.
> >
> >> Also i do not think that the buddy allocator use max_pfn or max_low_pfn
> >> to consider page/zone for allocation or not.
> >
> > Yes, I took it out with no effects.  I'll investigate further whether
> > we should be touching those variables or not for this new usage.
> 
> Although it does not offer perfect protection if device memory is at a
> physically lower address than RAM, skipping the update of these
> variables does seem to be what we want.  For example /dev/mem would
> fail to allow write access to persistent memory if it fails a
> valid_phys_addr_range() check.  Since /dev/mem does not know how to
> write to PMEM in a reliably persistent way, it should not treat a
> PMEM-pfn like RAM.

So i attach is a patch that should keep ZONE_DEVICE out of consideration
for the buddy allocator. You might also want to keep page reserved and not
free inside the zone, you could replace the generic_online_page() using
set_online_page_callback() while hotpluging device memory.

Regarding /dev/mem i would not worry about highmem, as /dev/mem is already
broken in respect to memory hole that might exist (at least that is my
understanding). Alternatively if you really care about /dev/mem you could
add an arch valid_phys_addr_range() that could check valid zone.

Cheers,
Jérôme

[toc] | [prev] | [next] | [standalone]

#1208986

On Mon, Aug 17, 2015 at 2:45 PM, Jerome Glisse <j.glisse@gmail.com> wrote:
> On Fri, Aug 14, 2015 at 07:11:27PM -0700, Dan Williams wrote:
>> Although it does not offer perfect protection if device memory is at a
>> physically lower address than RAM, skipping the update of these
>> variables does seem to be what we want.  For example /dev/mem would
>> fail to allow write access to persistent memory if it fails a
>> valid_phys_addr_range() check.  Since /dev/mem does not know how to
>> write to PMEM in a reliably persistent way, it should not treat a
>> PMEM-pfn like RAM.
>
> So i attach is a patch that should keep ZONE_DEVICE out of consideration
> for the buddy allocator. You might also want to keep page reserved and not
> free inside the zone, you could replace the generic_online_page() using
> set_online_page_callback() while hotpluging device memory.
>

Hmm, are we already protected by the fact that ZONE_DEVICE is not
represented in the GFP_ZONEMASK?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1209378

On Mon, Aug 17, 2015 at 05:46:43PM -0700, Dan Williams wrote:
> On Mon, Aug 17, 2015 at 2:45 PM, Jerome Glisse <j.glisse@gmail.com> wrote:
> > On Fri, Aug 14, 2015 at 07:11:27PM -0700, Dan Williams wrote:
> >> Although it does not offer perfect protection if device memory is at a
> >> physically lower address than RAM, skipping the update of these
> >> variables does seem to be what we want.  For example /dev/mem would
> >> fail to allow write access to persistent memory if it fails a
> >> valid_phys_addr_range() check.  Since /dev/mem does not know how to
> >> write to PMEM in a reliably persistent way, it should not treat a
> >> PMEM-pfn like RAM.
> >
> > So i attach is a patch that should keep ZONE_DEVICE out of consideration
> > for the buddy allocator. You might also want to keep page reserved and not
> > free inside the zone, you could replace the generic_online_page() using
> > set_online_page_callback() while hotpluging device memory.
> >
> 
> Hmm, are we already protected by the fact that ZONE_DEVICE is not
> represented in the GFP_ZONEMASK?

Yeah seems you right, high_zoneidx (which is derive using gfp_zone()) will
always limit which zones are considered. I thought that under memory presure
it would go over all of the zonelist entry and eventualy consider the device
zone. But it doesn't seems to be that way.

Keeping the device zone out of the zonelist might still be a good idea, if
only to avoid pointless iteration for the page allocator. Unless someone can
think of a reason why this would be bad.

Cheers,
Jérôme
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1209400

On Tue, Aug 18, 2015 at 9:55 AM, Jerome Glisse <j.glisse@gmail.com> wrote:
> On Mon, Aug 17, 2015 at 05:46:43PM -0700, Dan Williams wrote:
>> On Mon, Aug 17, 2015 at 2:45 PM, Jerome Glisse <j.glisse@gmail.com> wrote:
>> > On Fri, Aug 14, 2015 at 07:11:27PM -0700, Dan Williams wrote:
>> >> Although it does not offer perfect protection if device memory is at a
>> >> physically lower address than RAM, skipping the update of these
>> >> variables does seem to be what we want.  For example /dev/mem would
>> >> fail to allow write access to persistent memory if it fails a
>> >> valid_phys_addr_range() check.  Since /dev/mem does not know how to
>> >> write to PMEM in a reliably persistent way, it should not treat a
>> >> PMEM-pfn like RAM.
>> >
>> > So i attach is a patch that should keep ZONE_DEVICE out of consideration
>> > for the buddy allocator. You might also want to keep page reserved and not
>> > free inside the zone, you could replace the generic_online_page() using
>> > set_online_page_callback() while hotpluging device memory.
>> >
>>
>> Hmm, are we already protected by the fact that ZONE_DEVICE is not
>> represented in the GFP_ZONEMASK?
>
> Yeah seems you right, high_zoneidx (which is derive using gfp_zone()) will
> always limit which zones are considered. I thought that under memory presure
> it would go over all of the zonelist entry and eventualy consider the device
> zone. But it doesn't seems to be that way.
>
> Keeping the device zone out of the zonelist might still be a good idea, if
> only to avoid pointless iteration for the page allocator. Unless someone can
> think of a reason why this would be bad.
>

The other question I have is whether disabling ZONE_DMA is a realistic
tradeoff for enabling ZONE_DEVICE?  I.e. can ZONE_DMA default to off
going forward, lose some ISA device support, or do we need to figure
out how to enable > 4 zones.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1209434

On Tue, Aug 18, 2015 at 10:23:38AM -0700, Dan Williams wrote:
> On Tue, Aug 18, 2015 at 9:55 AM, Jerome Glisse <j.glisse@gmail.com> wrote:
> > On Mon, Aug 17, 2015 at 05:46:43PM -0700, Dan Williams wrote:
> >> On Mon, Aug 17, 2015 at 2:45 PM, Jerome Glisse <j.glisse@gmail.com> wrote:
> >> > On Fri, Aug 14, 2015 at 07:11:27PM -0700, Dan Williams wrote:
> >> >> Although it does not offer perfect protection if device memory is at a
> >> >> physically lower address than RAM, skipping the update of these
> >> >> variables does seem to be what we want.  For example /dev/mem would
> >> >> fail to allow write access to persistent memory if it fails a
> >> >> valid_phys_addr_range() check.  Since /dev/mem does not know how to
> >> >> write to PMEM in a reliably persistent way, it should not treat a
> >> >> PMEM-pfn like RAM.
> >> >
> >> > So i attach is a patch that should keep ZONE_DEVICE out of consideration
> >> > for the buddy allocator. You might also want to keep page reserved and not
> >> > free inside the zone, you could replace the generic_online_page() using
> >> > set_online_page_callback() while hotpluging device memory.
> >> >
> >>
> >> Hmm, are we already protected by the fact that ZONE_DEVICE is not
> >> represented in the GFP_ZONEMASK?
> >
> > Yeah seems you right, high_zoneidx (which is derive using gfp_zone()) will
> > always limit which zones are considered. I thought that under memory presure
> > it would go over all of the zonelist entry and eventualy consider the device
> > zone. But it doesn't seems to be that way.
> >
> > Keeping the device zone out of the zonelist might still be a good idea, if
> > only to avoid pointless iteration for the page allocator. Unless someone can
> > think of a reason why this would be bad.
> >
> 
> The other question I have is whether disabling ZONE_DMA is a realistic
> tradeoff for enabling ZONE_DEVICE?  I.e. can ZONE_DMA default to off
> going forward, lose some ISA device support, or do we need to figure
> out how to enable > 4 zones.

That require some auditing a quick look and it seems to matter for s390
arch and there is still few driver that use it. I think we can forget
about ISA bus, i would be surprise if you could still run a recent kernel
on a computer that has ISA bus.

Thought maybe you don't need a new ZONE_DEV and all you need is valid
struct page for this device memory, and you don't want this page to be
useable by the general memory allocator. There is surely other ways to
achieve that like marking all as reserved when you hotplug them.

Cheers,
Jérôme
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1208043

On Fri, Aug 14, 2015 at 02:52:15PM -0700, Dan Williams wrote:
> The idea is that this memory is not meant to be available to the page
> allocator and should not count as new memory capacity.  We're only
> hotplugging it to get struct page coverage.

This might need a bigger audit of the max_pfn usages.  I remember
architectures using it as a decisions for using IOMMUs or similar.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1208083

On Wed, Aug 12, 2015 at 11:50:05PM -0400, Dan Williams wrote:
> arch_add_memory() is reorganized a bit in preparation for a new
> arch_add_dev_memory() api, for now there is no functional change to the
> memory hotplug code.

Instead of the new arch_add_dev_memory call I'd just add a bool device
argument to arch_add_memory and zone_for_memory (and later the altmap
pointer aswell).

arch_add_memory is a candidate to be factored into common code,
except for s390 everything could be done with two small arch callouts.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web