Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1628767 > unrolled thread
| Started by | Jérôme Glisse <jglisse@redhat.com> |
|---|---|
| First post | 2017-04-22 05:40 +0200 |
| Last post | 2017-04-22 07:00 +0200 |
| Articles | 10 — 4 participants |
Back to article view | Back to linux.kernel
[HMM 00/15] HMM (Heterogeneous Memory Management) v20 Jérôme Glisse <jglisse@redhat.com> - 2017-04-22 05:40 +0200
[HMM 03/15] mm/unaddressable-memory: new type of ZONE_DEVICE for unaddressable memory Jérôme Glisse <jglisse@redhat.com> - 2017-04-22 05:50 +0200
Re: [HMM 03/15] mm/unaddressable-memory: new type of ZONE_DEVICE for unaddressable memory Dan Williams <dan.j.williams@intel.com> - 2017-04-22 07:40 +0200
Re: [HMM 03/15] mm/unaddressable-memory: new type of ZONE_DEVICE for unaddressable memory Jerome Glisse <jglisse@redhat.com> - 2017-04-22 20:20 +0200
Re: [HMM 03/15] mm/unaddressable-memory: new type of ZONE_DEVICE for unaddressable memory Dan Williams <dan.j.williams@intel.com> - 2017-04-23 15:20 +0200
Re: [HMM 03/15] mm/unaddressable-memory: new type of ZONE_DEVICE for unaddressable memory John Hubbard <jhubbard@nvidia.com> - 2017-04-24 02:50 +0200
[HMM 01/15] mm, memory_hotplug: introduce add_pages Jérôme Glisse <jglisse@redhat.com> - 2017-04-22 05:50 +0200
[HMM 07/15] mm/hmm: heterogeneous memory management (HMM for short) v2 Jérôme Glisse <jglisse@redhat.com> - 2017-04-22 06:10 +0200
[HMM 08/15] mm/hmm/mirror: mirror process address space on device with HMM helpers v2 Jérôme Glisse <jglisse@redhat.com> - 2017-04-22 06:10 +0200
[HMM 14/15] mm/hmm/devmem: dummy HMM device for ZONE_DEVICE memory v3 Jérôme Glisse <jglisse@redhat.com> - 2017-04-22 07:00 +0200
| From | Jérôme Glisse <jglisse@redhat.com> |
|---|---|
| Date | 2017-04-22 05:40 +0200 |
| Subject | [HMM 00/15] HMM (Heterogeneous Memory Management) v20 |
| Message-ID | <tyU3T-61v-3@gated-at.bofh.it> |
Patchset is on top of mmotm mmotm-2017-04-18 and Michal patchset
([PATCH -v3 0/13] mm: make movable onlining suck less). Branch:
https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-v20
I have included all suggestion made since v19, it is all build
fix and change in respect to memory hotplug with Michal rework.
Changes since v19:
- Included various build fix and compilation warning fix
- Limit HMM to x86-64 (easy to enable other arch as separate patch)
- Rebase on top of Michal memory hotplug rework
Heterogeneous Memory Management (HMM) (description and justification)
Today device driver expose dedicated memory allocation API through their
device file, often relying on a combination of IOCTL and mmap calls. The
device can only access and use memory allocated through this API. This
effectively split the program address space into object allocated for the
device and useable by the device and other regular memory (malloc, mmap
of a file, share memory, …) only accessible by CPU (or in a very limited
way by a device by pinning memory).
Allowing different isolated component of a program to use a device thus
require duplication of the input data structure using device memory
allocator. This is reasonable for simple data structure (array, grid,
image, …) but this get extremely complex with advance data structure
(list, tree, graph, …) that rely on a web of memory pointers. This is
becoming a serious limitation on the kind of work load that can be
offloaded to device like GPU.
New industry standard like C++, OpenCL or CUDA are pushing to remove this
barrier. This require a shared address space between GPU device and CPU so
that GPU can access any memory of a process (while still obeying memory
protection like read only). This kind of feature is also appearing in
various other operating systems.
HMM is a set of helpers to facilitate several aspects of address space
sharing and device memory management. Unlike existing sharing mechanism
that rely on pining pages use by a device, HMM relies on mmu_notifier to
propagate CPU page table update to device page table.
Duplicating CPU page table is only one aspect necessary for efficiently
using device like GPU. GPU local memory have bandwidth in the TeraBytes/
second range but they are connected to main memory through a system bus
like PCIE that is limited to 32GigaBytes/second (PCIE 4.0 16x). Thus it
is necessary to allow migration of process memory from main system memory
to device memory. Issue is that on platform that only have PCIE the device
memory is not accessible by the CPU with the same properties as main
memory (cache coherency, atomic operations, …).
To allow migration from main memory to device memory HMM provides a set
of helper to hotplug device memory as a new type of ZONE_DEVICE memory
which is un-addressable by CPU but still has struct page representing it.
This allow most of the core kernel logic that deals with a process memory
to stay oblivious of the peculiarity of device memory.
When page backing an address of a process is migrated to device memory
the CPU page table entry is set to a new specific swap entry. CPU access
to such address triggers a migration back to system memory, just like if
the page was swap on disk. HMM also blocks any one from pinning a
ZONE_DEVICE page so that it can always be migrated back to system memory
if CPU access it. Conversely HMM does not migrate to device memory any
page that is pin in system memory.
To allow efficient migration between device memory and main memory a new
migrate_vma() helpers is added with this patchset. It allows to leverage
device DMA engine to perform the copy operation.
This feature will be use by upstream driver like nouveau mlx5 and probably
other in the future (amdgpu is next suspect in line). We are actively
working on nouveau and mlx5 support. To test this patchset we also worked
with NVidia close source driver team, they have more resources than us to
test this kind of infrastructure and also a bigger and better userspace
eco-system with various real industry workload they can be use to test and
profile HMM.
The expected workload is a program builds a data set on the CPU (from disk,
from network, from sensors, …). Program uses GPU API (OpenCL, CUDA, ...)
to give hint on memory placement for the input data and also for the output
buffer. Program call GPU API to schedule a GPU job, this happens using
device driver specific ioctl. All this is hidden from programmer point of
view in case of C++ compiler that transparently offload some part of a
program to GPU. Program can keep doing other stuff on the CPU while the
GPU is crunching numbers.
It is expected that CPU will not access the same data set as the GPU while
GPU is working on it, but this is not mandatory. In fact we expect some
small memory object to be actively access by both GPU and CPU concurrently
as synchronization channel and/or for monitoring purposes. Such object will
stay in system memory and should not be bottlenecked by system bus
bandwidth (rare write and read access from both CPU and GPU).
As we are relying on device driver API, HMM does not introduce any new
syscall nor does it modify any existing ones. It does not change any POSIX
semantics or behaviors. For instance the child after a fork of a process
that is using HMM will not be impacted in anyway, nor is there any data
hazard between child COW or parent COW of memory that was migrated to
device prior to fork.
HMM assume a numbers of hardware features. Device must allow device page
table to be updated at any time (ie device job must be preemptable). Device
page table must provides memory protection such as read only. Device must
track write access (dirty bit). Device must have a minimum granularity that
match PAGE_SIZE (ie 4k).
Reviewer (just hint):
Patch 1 add new add_pages() helper to avoid modifying each arch memory
hot plug function
Patch 2 move the page reference decrement from put_page() to
put_zone_device_page() Dan Williams is the best person to review
this change
Patch 3 add a new memory type for ZONE_DEVICE and also add all the logic
in various core mm to support this new type. Dan Williams and
any core mm contributor are best people to review each half of
this patchset
Patch 4 add a new migrate mode. Any one familiar with page migration is
welcome to review.
Patch 5 introduce a new migration helper (migrate_vma()) that allow to
migrate a range of virtual address of a process using device DMA
engine to perform the copy. It is not limited to do copy from and
to device but can also do copy between any kind of source and
destination memory. Again anyone familiar with migration code
should be able to verify the logic.
Patch 6 optimize the new migrate_vma() by unmapping pages while we are
collecting them. This can be review by any mm folks.
Patch 7 introduce core infrastructure and definition of HMM, pretty
small patch and easy to review
Patch 8 introduce the mirror functionality of HMM, it relies on
mmu_notifier and thus someone familiar with that part would be
in better position to review
Patch 9 is an helper to snapshot CPU page table while synchronizing with
concurrent page table update. Understanding mmu_notifier makes
review easier.
Patch 10 is mostly a wrapper around handle_mm_fault()
Patch 11 add unaddressable memory migration to helper introduced in patch
6, this can be review by anyone familiar with migration code
Patch 12 add a feature that allow device to allocate non-present page on
the GPU when migrating a range of address to device memory. This
is an helper for device driver to avoid having to first allocate
system memory before migration to device memory
Patch 13 add helper to hotplug un-addressable device memory as new type
of ZONE_DEVICE memory (new type introducted in patch 3 of this
serie). This is boiler plate code around memory hotplug and it
also pick a free range of physical address for the device memory.
Note that the physical address do not point to anything (at least
as far as the kernel knows).
Patch 14 introduce a new hmm_device class as an helper for device driver
that want to expose multiple device memory under a common fake
device driver. This is usefull for multi-gpu configuration.
Anyone familiar with device driver infrastructure can review
this. Boiler plate code really.
Patch 15 is the documentation for everything
Previous patchset posting :
v1 http://lwn.net/Articles/597289/
v2 https://lkml.org/lkml/2014/6/12/559
v3 https://lkml.org/lkml/2014/6/13/633
v4 https://lkml.org/lkml/2014/8/29/423
v5 https://lkml.org/lkml/2014/11/3/759
v6 http://lwn.net/Articles/619737/
v7 http://lwn.net/Articles/627316/
v8 https://lwn.net/Articles/645515/
v9 https://lwn.net/Articles/651553/
v10 https://lwn.net/Articles/654430/
v11 http://www.gossamer-threads.com/lists/linux/kernel/2286424
v12 http://www.kernelhub.org/?msg=972982&p=2
v13 https://lwn.net/Articles/706856/
v14 https://lkml.org/lkml/2016/12/8/344
v15 http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1304107.html
v16 http://www.spinics.net/lists/linux-mm/msg119814.html
v17 https://lkml.org/lkml/2017/1/27/847
v18 https://lkml.org/lkml/2017/3/16/596
v19 https://lkml.org/lkml/2017/4/5/831
Jérôme Glisse (14):
mm/put_page: move ZONE_DEVICE page reference decrement v2
mm/unaddressable-memory: new type of ZONE_DEVICE for unaddressable
memory
mm/migrate: new migrate mode MIGRATE_SYNC_NO_COPY
mm/migrate: new memory migration helper for use with device memory v4
mm/migrate: migrate_vma() unmap page from vma while collecting pages
mm/hmm: heterogeneous memory management (HMM for short) v2
mm/hmm/mirror: mirror process address space on device with HMM helpers
v2
mm/hmm/mirror: helper to snapshot CPU page table v2
mm/hmm/mirror: device page fault handler
mm/migrate: support un-addressable ZONE_DEVICE page in migration
mm/migrate: allow migrate_vma() to alloc new page on empty entry v2
mm/hmm/devmem: device memory hotplug using ZONE_DEVICE v3
mm/hmm/devmem: dummy HMM device for ZONE_DEVICE memory v3
hmm: heterogeneous memory management documentation
Michal Hocko (1):
mm, memory_hotplug: introduce add_pages
Documentation/vm/hmm.txt | 362 ++++++++++++
MAINTAINERS | 7 +
arch/x86/Kconfig | 4 +
arch/x86/mm/init_64.c | 22 +-
fs/aio.c | 8 +
fs/f2fs/data.c | 5 +-
fs/hugetlbfs/inode.c | 5 +-
fs/proc/task_mmu.c | 7 +
fs/ubifs/file.c | 5 +-
include/linux/hmm.h | 468 ++++++++++++++++
include/linux/ioport.h | 1 +
include/linux/memory_hotplug.h | 11 +
include/linux/memremap.h | 82 +++
include/linux/migrate.h | 115 ++++
include/linux/migrate_mode.h | 5 +
include/linux/mm.h | 14 +-
include/linux/mm_types.h | 5 +
include/linux/swap.h | 24 +-
include/linux/swapops.h | 68 +++
kernel/fork.c | 2 +
kernel/memremap.c | 47 ++
mm/Kconfig | 47 ++
mm/Makefile | 1 +
mm/balloon_compaction.c | 8 +
mm/hmm.c | 1203 ++++++++++++++++++++++++++++++++++++++++
mm/memory.c | 61 ++
mm/memory_hotplug.c | 10 +-
mm/migrate.c | 789 +++++++++++++++++++++++++-
mm/mprotect.c | 14 +
mm/page_vma_mapped.c | 10 +
mm/rmap.c | 25 +
mm/zsmalloc.c | 8 +
32 files changed, 3412 insertions(+), 31 deletions(-)
create mode 100644 Documentation/vm/hmm.txt
create mode 100644 include/linux/hmm.h
create mode 100644 mm/hmm.c
--
2.9.3
[toc] | [next] | [standalone]
| From | Jérôme Glisse <jglisse@redhat.com> |
|---|---|
| Date | 2017-04-22 05:50 +0200 |
| Subject | [HMM 03/15] mm/unaddressable-memory: new type of ZONE_DEVICE for unaddressable memory |
| Message-ID | <tyUdA-64D-1@gated-at.bofh.it> |
| In reply to | #1628767 |
HMM (heterogeneous memory management) need struct page to support migration
from system main memory to device memory. Reasons for HMM and migration to
device memory is explained with HMM core patch.
This patch deals with device memory that is un-addressable memory (ie CPU
can not access it). Hence we do not want those struct page to be manage
like regular memory. That is why we extend ZONE_DEVICE to support different
types of memory.
A persistent memory type is define for existing user of ZONE_DEVICE and a
new device un-addressable type is added for the un-addressable memory type.
There is a clear separation between what is expected from each memory type
and existing user of ZONE_DEVICE are un-affected by new requirement and new
use of the un-addressable type. All specific code path are protect with
test against the memory type.
Because memory is un-addressable we use a new special swap type for when
a page is migrated to device memory (this reduces the number of maximum
swap file).
The main two additions beside memory type to ZONE_DEVICE is two callbacks.
First one, page_free() is call whenever page refcount reach 1 (which means
the page is free as ZONE_DEVICE page never reach a refcount of 0). This
allow device driver to manage its memory and associated struct page.
The second callback page_fault() happens when there is a CPU access to
an address that is back by a device page (which are un-addressable by the
CPU). This callback is responsible to migrate the page back to system
main memory. Device driver can not block migration back to system memory,
HMM make sure that such page can not be pin into device memory.
If device is in some error condition and can not migrate memory back then
a CPU page fault to device memory should end with SIGBUS.
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Ross Zwisler <ross.zwisler@linux.intel.com>
---
fs/proc/task_mmu.c | 7 +++++
include/linux/ioport.h | 1 +
include/linux/memremap.h | 82 ++++++++++++++++++++++++++++++++++++++++++++++++
include/linux/swap.h | 24 ++++++++++++--
include/linux/swapops.h | 68 +++++++++++++++++++++++++++++++++++++++
kernel/memremap.c | 43 ++++++++++++++++++++++++-
mm/Kconfig | 13 ++++++++
mm/memory.c | 61 +++++++++++++++++++++++++++++++++++
mm/memory_hotplug.c | 10 ++++--
mm/mprotect.c | 14 +++++++++
10 files changed, 317 insertions(+), 6 deletions(-)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index f0c8b33..a12ba94 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -542,6 +542,8 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr,
}
} else if (is_migration_entry(swpent))
page = migration_entry_to_page(swpent);
+ else if (is_device_entry(swpent))
+ page = device_entry_to_page(swpent);
} else if (unlikely(IS_ENABLED(CONFIG_SHMEM) && mss->check_shmem_swap
&& pte_none(*pte))) {
page = find_get_entry(vma->vm_file->f_mapping,
@@ -704,6 +706,8 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask,
if (is_migration_entry(swpent))
page = migration_entry_to_page(swpent);
+ else if (is_device_entry(swpent))
+ page = device_entry_to_page(swpent);
}
if (page) {
int mapcount = page_mapcount(page);
@@ -1196,6 +1200,9 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm,
flags |= PM_SWAP;
if (is_migration_entry(entry))
page = migration_entry_to_page(entry);
+
+ if (is_device_entry(entry))
+ page = device_entry_to_page(entry);
}
if (page && !PageAnon(page))
diff --git a/include/linux/ioport.h b/include/linux/ioport.h
index 6230064..ec619dc 100644
--- a/include/linux/ioport.h
+++ b/include/linux/ioport.h
@@ -130,6 +130,7 @@ enum {
IORES_DESC_ACPI_NV_STORAGE = 3,
IORES_DESC_PERSISTENT_MEMORY = 4,
IORES_DESC_PERSISTENT_MEMORY_LEGACY = 5,
+ IORES_DESC_DEVICE_MEMORY_UNADDRESSABLE = 6,
};
/* helpers to define resources */
diff --git a/include/linux/memremap.h b/include/linux/memremap.h
index 9341619..365fb4e 100644
--- a/include/linux/memremap.h
+++ b/include/linux/memremap.h
@@ -35,24 +35,101 @@ static inline struct vmem_altmap *to_vmem_altmap(unsigned long memmap_start)
}
#endif
+/*
+ * Specialize ZONE_DEVICE memory into multiple types each having differents
+ * usage.
+ *
+ * MEMORY_DEVICE_PERSISTENT:
+ * Persistent device memory (pmem): struct page might be allocated in different
+ * memory and architecture might want to perform special actions. It is similar
+ * to regular memory, in that the CPU can access it transparently. However,
+ * it is likely to have different bandwidth and latency than regular memory.
+ * See Documentation/nvdimm/nvdimm.txt for more information.
+ *
+ * MEMORY_DEVICE_UNADDRESSABLE:
+ * Device memory that is not directly addressable by the CPU: CPU can neither
+ * read nor write _UNADDRESSABLE memory. In this case, we do still have struct
+ * pages backing the device memory. Doing so simplifies the implementation, but
+ * it is important to remember that there are certain points at which the struct
+ * page must be treated as an opaque object, rather than a "normal" struct page.
+ * A more complete discussion of unaddressable memory may be found in
+ * include/linux/hmm.h and Documentation/vm/hmm.txt.
+ */
+enum memory_type {
+ MEMORY_DEVICE_PERSISTENT = 0,
+ MEMORY_DEVICE_UNADDRESSABLE,
+};
+
+/*
+ * For MEMORY_DEVICE_UNADDRESSABLE we use ZONE_DEVICE and extend it with two
+ * callbacks:
+ * page_fault()
+ * page_free()
+ *
+ * Additional notes about MEMORY_DEVICE_UNADDRESSABLE may be found in
+ * include/linux/hmm.h and Documentation/vm/hmm.txt. There is also a brief
+ * explanation in include/linux/memory_hotplug.h.
+ *
+ * The page_fault() callback must migrate page back, from device memory to
+ * system memory, so that the CPU can access it. This might fail for various
+ * reasons (device issues, device have been unplugged, ...). When such error
+ * conditions happen, the page_fault() callback must return VM_FAULT_SIGBUS and
+ * set the CPU page table entry to "poisoned".
+ *
+ * Note that because memory cgroup charges are transferred to the device memory,
+ * this should never fail due to memory restrictions. However, allocation
+ * of a regular system page might still fail because we are out of memory. If
+ * that happens, the page_fault() callback must return VM_FAULT_OOM.
+ *
+ * The page_fault() callback can also try to migrate back multiple pages in one
+ * chunk, as an optimization. It must, however, prioritize the faulting address
+ * over all the others.
+ *
+ *
+ * The page_free() callback is called once the page refcount reaches 1
+ * (ZONE_DEVICE pages never reach 0 refcount unless there is a refcount bug.
+ * This allows the device driver to implement its own memory management.)
+ */
+typedef int (*dev_page_fault_t)(struct vm_area_struct *vma,
+ unsigned long addr,
+ struct page *page,
+ unsigned int flags,
+ pmd_t *pmdp);
+typedef void (*dev_page_free_t)(struct page *page, void *data);
+
/**
* struct dev_pagemap - metadata for ZONE_DEVICE mappings
+ * @page_fault: callback when CPU fault on an unaddressable device page
+ * @page_free: free page callback when page refcount reaches 1
* @altmap: pre-allocated/reserved memory for vmemmap allocations
* @res: physical address range covered by @ref
* @ref: reference count that pins the devm_memremap_pages() mapping
* @dev: host device of the mapping for debug
+ * @data: private data pointer for page_free()
+ * @type: memory type: see MEMORY_* in memory_hotplug.h
*/
struct dev_pagemap {
+ dev_page_fault_t page_fault;
+ dev_page_free_t page_free;
struct vmem_altmap *altmap;
const struct resource *res;
struct percpu_ref *ref;
struct device *dev;
+ void *data;
+ enum memory_type type;
};
#ifdef CONFIG_ZONE_DEVICE
void *devm_memremap_pages(struct device *dev, struct resource *res,
struct percpu_ref *ref, struct vmem_altmap *altmap);
struct dev_pagemap *find_dev_pagemap(resource_size_t phys);
+
+static inline bool is_device_unaddressable_page(const struct page *page)
+{
+ /* See MEMORY_DEVICE_UNADDRESSABLE in include/linux/memory_hotplug.h */
+ return ((page_zonenum(page) == ZONE_DEVICE) &&
+ (page->pgmap->type == MEMORY_DEVICE_UNADDRESSABLE));
+}
#else
static inline void *devm_memremap_pages(struct device *dev,
struct resource *res, struct percpu_ref *ref,
@@ -71,6 +148,11 @@ static inline struct dev_pagemap *find_dev_pagemap(resource_size_t phys)
{
return NULL;
}
+
+static inline bool is_device_unaddressable_page(const struct page *page)
+{
+ return false;
+}
#endif
/**
diff --git a/include/linux/swap.h b/include/linux/swap.h
index ba58824..52a137d 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -51,6 +51,23 @@ static inline int current_is_kswapd(void)
*/
/*
+ * Unaddressable device memory support. See include/linux/hmm.h and
+ * Documentation/vm/hmm.txt. Short description is we need struct pages for
+ * device memory that is unaddressable (inaccessible) by CPU, so that we can
+ * migrate part of a process memory to device memory.
+ *
+ * When a page is migrated from CPU to device, we set the CPU page table entry
+ * to a special SWP_DEVICE_* entry.
+ */
+#ifdef CONFIG_DEVICE_UNADDRESSABLE
+#define SWP_DEVICE_NUM 2
+#define SWP_DEVICE_WRITE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM)
+#define SWP_DEVICE_READ (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+1)
+#else
+#define SWP_DEVICE_NUM 0
+#endif
+
+/*
* NUMA node memory migration support
*/
#ifdef CONFIG_MIGRATION
@@ -72,7 +89,8 @@ static inline int current_is_kswapd(void)
#endif
#define MAX_SWAPFILES \
- ((1 << MAX_SWAPFILES_SHIFT) - SWP_MIGRATION_NUM - SWP_HWPOISON_NUM)
+ ((1 << MAX_SWAPFILES_SHIFT) - SWP_DEVICE_NUM - \
+ SWP_MIGRATION_NUM - SWP_HWPOISON_NUM)
/*
* Magic header for a swap area. The first part of the union is
@@ -432,8 +450,8 @@ static inline void show_swap_cache_info(void)
{
}
-#define free_swap_and_cache(swp) is_migration_entry(swp)
-#define swapcache_prepare(swp) is_migration_entry(swp)
+#define free_swap_and_cache(e) (is_migration_entry(e) || is_device_entry(e))
+#define swapcache_prepare(e) (is_migration_entry(e) || is_device_entry(e))
static inline int add_swap_count_continuation(swp_entry_t swp, gfp_t gfp_mask)
{
diff --git a/include/linux/swapops.h b/include/linux/swapops.h
index 5c3a5f3..960eb6b 100644
--- a/include/linux/swapops.h
+++ b/include/linux/swapops.h
@@ -100,6 +100,74 @@ static inline void *swp_to_radix_entry(swp_entry_t entry)
return (void *)(value | RADIX_TREE_EXCEPTIONAL_ENTRY);
}
+#if IS_ENABLED(CONFIG_DEVICE_UNADDRESSABLE)
+static inline swp_entry_t make_device_entry(struct page *page, bool write)
+{
+ return swp_entry(write ? SWP_DEVICE_WRITE : SWP_DEVICE_READ,
+ page_to_pfn(page));
+}
+
+static inline bool is_device_entry(swp_entry_t entry)
+{
+ int type = swp_type(entry);
+ return type == SWP_DEVICE_READ || type == SWP_DEVICE_WRITE;
+}
+
+static inline void make_device_entry_read(swp_entry_t *entry)
+{
+ *entry = swp_entry(SWP_DEVICE_READ, swp_offset(*entry));
+}
+
+static inline bool is_write_device_entry(swp_entry_t entry)
+{
+ return unlikely(swp_type(entry) == SWP_DEVICE_WRITE);
+}
+
+static inline struct page *device_entry_to_page(swp_entry_t entry)
+{
+ return pfn_to_page(swp_offset(entry));
+}
+
+int device_entry_fault(struct vm_area_struct *vma,
+ unsigned long addr,
+ swp_entry_t entry,
+ unsigned int flags,
+ pmd_t *pmdp);
+#else /* CONFIG_DEVICE_UNADDRESSABLE */
+static inline swp_entry_t make_device_entry(struct page *page, bool write)
+{
+ return swp_entry(0, 0);
+}
+
+static inline void make_device_entry_read(swp_entry_t *entry)
+{
+}
+
+static inline bool is_device_entry(swp_entry_t entry)
+{
+ return false;
+}
+
+static inline bool is_write_device_entry(swp_entry_t entry)
+{
+ return false;
+}
+
+static inline struct page *device_entry_to_page(swp_entry_t entry)
+{
+ return NULL;
+}
+
+static inline int device_entry_fault(struct vm_area_struct *vma,
+ unsigned long addr,
+ swp_entry_t entry,
+ unsigned int flags,
+ pmd_t *pmdp)
+{
+ return VM_FAULT_SIGBUS;
+}
+#endif /* CONFIG_DEVICE_UNADDRESSABLE */
+
#ifdef CONFIG_MIGRATION
static inline swp_entry_t make_migration_entry(struct page *page, int write)
{
diff --git a/kernel/memremap.c b/kernel/memremap.c
index 97ef676..dce3b29 100644
--- a/kernel/memremap.c
+++ b/kernel/memremap.c
@@ -18,6 +18,8 @@
#include <linux/io.h>
#include <linux/mm.h>
#include <linux/memory_hotplug.h>
+#include <linux/swap.h>
+#include <linux/swapops.h>
#ifndef ioremap_cache
/* temporary while we convert existing ioremap_cache users to memremap */
@@ -194,12 +196,47 @@ void put_zone_device_page(struct page *page)
* ZONE_DEVICE page refcount should never reach 0 and never be freed
* to kernel memory allocator.
*/
- page_ref_dec(page);
+ int count = page_ref_dec_return(page);
+
+ /*
+ * If refcount is 1 then page is freed and refcount is stable as nobody
+ * holds a reference on the page.
+ */
+ if (page->pgmap->page_free && count == 1)
+ page->pgmap->page_free(page, page->pgmap->data);
put_dev_pagemap(page->pgmap);
}
EXPORT_SYMBOL(put_zone_device_page);
+#if IS_ENABLED(CONFIG_DEVICE_UNADDRESSABLE)
+int device_entry_fault(struct vm_area_struct *vma,
+ unsigned long addr,
+ swp_entry_t entry,
+ unsigned int flags,
+ pmd_t *pmdp)
+{
+ struct page *page = device_entry_to_page(entry);
+
+ /*
+ * The page_fault() callback must migrate page back to system memory
+ * so that CPU can access it. This might fail for various reasons
+ * (device issue, device was unsafely unplugged, ...). When such
+ * error conditions happen, the callback must return VM_FAULT_SIGBUS.
+ *
+ * Note that because memory cgroup charges are accounted to the device
+ * memory, this should never fail because of memory restrictions (but
+ * allocation of regular system page might still fail because we are
+ * out of memory).
+ *
+ * There is a more in-depth description of what that callback can and
+ * cannot do, in include/linux/memremap.h
+ */
+ return page->pgmap->page_fault(vma, addr, page, flags, pmdp);
+}
+EXPORT_SYMBOL(device_entry_fault);
+#endif /* CONFIG_DEVICE_UNADDRESSABLE */
+
static void pgmap_radix_release(struct resource *res)
{
resource_size_t key, align_start, align_size, align_end;
@@ -332,6 +369,10 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,
}
pgmap->ref = ref;
pgmap->res = &page_map->res;
+ pgmap->type = MEMORY_DEVICE_PERSISTENT;
+ pgmap->page_fault = NULL;
+ pgmap->page_free = NULL;
+ pgmap->data = NULL;
mutex_lock(&pgmap_lock);
error = 0;
diff --git a/mm/Kconfig b/mm/Kconfig
index c89f472..ad4bd50 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -700,6 +700,19 @@ config ZONE_DEVICE
If FS_DAX is enabled, then say Y.
+config DEVICE_UNADDRESSABLE
+ bool "Unaddressable device memory (GPU memory, ...)"
+ depends on X86_64
+ depends on ZONE_DEVICE
+ depends on MEMORY_HOTPLUG
+ depends on MEMORY_HOTREMOVE
+ depends on SPARSEMEM_VMEMMAP
+
+ help
+ Allows creation of struct pages to represent unaddressable device
+ memory; i.e., memory that is only accessible from the device (or
+ group of devices).
+
config FRAME_VECTOR
bool
diff --git a/mm/memory.c b/mm/memory.c
index 6ff5d72..4116ab2 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -49,6 +49,7 @@
#include <linux/swap.h>
#include <linux/highmem.h>
#include <linux/pagemap.h>
+#include <linux/memremap.h>
#include <linux/ksm.h>
#include <linux/rmap.h>
#include <linux/export.h>
@@ -927,6 +928,35 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
pte = pte_swp_mksoft_dirty(pte);
set_pte_at(src_mm, addr, src_pte, pte);
}
+ } else if (is_device_entry(entry)) {
+ page = device_entry_to_page(entry);
+
+ /*
+ * Update rss count even for unaddressable pages, as
+ * they should treated just like normal pages in this
+ * respect.
+ *
+ * We will likely want to have some new rss counters
+ * for unaddressable pages, at some point. But for now
+ * keep things as they are.
+ */
+ get_page(page);
+ rss[mm_counter(page)]++;
+ page_dup_rmap(page, false);
+
+ /*
+ * We do not preserve soft-dirty information, because so
+ * far, checkpoint/restore is the only feature that
+ * requires that. And checkpoint/restore does not work
+ * when a device driver is involved (you cannot easily
+ * save and restore device driver state).
+ */
+ if (is_write_device_entry(entry) &&
+ is_cow_mapping(vm_flags)) {
+ make_device_entry_read(&entry);
+ pte = swp_entry_to_pte(entry);
+ set_pte_at(src_mm, addr, src_pte, pte);
+ }
}
goto out_set_pte;
}
@@ -1243,6 +1273,29 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
}
continue;
}
+
+ entry = pte_to_swp_entry(ptent);
+ if (non_swap_entry(entry) && is_device_entry(entry)) {
+ struct page *page = device_entry_to_page(entry);
+
+ if (unlikely(details && details->check_mapping)) {
+ /*
+ * unmap_shared_mapping_pages() wants to
+ * invalidate cache without truncating:
+ * unmap shared but keep private pages.
+ */
+ if (details->check_mapping !=
+ page_rmapping(page))
+ continue;
+ }
+
+ pte_clear_not_present_full(mm, addr, pte, tlb->fullmm);
+ rss[mm_counter(page)]--;
+ page_remove_rmap(page, false);
+ put_page(page);
+ continue;
+ }
+
/* If details->check_mapping, we leave swap entries. */
if (unlikely(details))
continue;
@@ -2690,6 +2743,14 @@ int do_swap_page(struct vm_fault *vmf)
if (is_migration_entry(entry)) {
migration_entry_wait(vma->vm_mm, vmf->pmd,
vmf->address);
+ } else if (is_device_entry(entry)) {
+ /*
+ * For un-addressable device memory we call the pgmap
+ * fault handler callback. The callback must migrate
+ * the page back to some CPU accessible page.
+ */
+ ret = device_entry_fault(vma, vmf->address, entry,
+ vmf->flags, vmf->pmd);
} else if (is_hwpoison_entry(entry)) {
ret = VM_FAULT_HWPOISON;
} else {
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index de30822..d988fbe 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -156,7 +156,7 @@ void mem_hotplug_done(void)
/* add this memory to iomem resource */
static struct resource *register_memory_resource(u64 start, u64 size)
{
- struct resource *res;
+ struct resource *res, *conflict;
res = kzalloc(sizeof(struct resource), GFP_KERNEL);
if (!res)
return ERR_PTR(-ENOMEM);
@@ -165,7 +165,13 @@ static struct resource *register_memory_resource(u64 start, u64 size)
res->start = start;
res->end = start + size - 1;
res->flags = IORESOURCE_SYSTEM_RAM | IORESOURCE_BUSY;
- if (request_resource(&iomem_resource, res) < 0) {
+ conflict = request_resource_conflict(&iomem_resource, res);
+ if (conflict) {
+ if (conflict->desc == IORES_DESC_DEVICE_MEMORY_UNADDRESSABLE) {
+ pr_debug("Device unaddressable memory block "
+ "memory hotplug at %#010llx !\n",
+ (unsigned long long)start);
+ }
pr_debug("System RAM resource %pR cannot be added\n", res);
kfree(res);
return ERR_PTR(-EEXIST);
diff --git a/mm/mprotect.c b/mm/mprotect.c
index 8edd0d5..dadb020 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -126,6 +126,20 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
pages++;
}
+
+ if (is_write_device_entry(entry)) {
+ pte_t newpte;
+
+ /*
+ * We do not preserve soft-dirtiness. See
+ * copy_one_pte() for explanation.
+ */
+ make_device_entry_read(&entry);
+ newpte = swp_entry_to_pte(entry);
+ set_pte_at(mm, addr, pte, newpte);
+
+ pages++;
+ }
}
} while (pte++, addr += PAGE_SIZE, addr != end);
arch_leave_lazy_mmu_mode();
--
2.9.3
[toc] | [prev] | [next] | [standalone]
| From | Dan Williams <dan.j.williams@intel.com> |
|---|---|
| Date | 2017-04-22 07:40 +0200 |
| Subject | Re: [HMM 03/15] mm/unaddressable-memory: new type of ZONE_DEVICE for unaddressable memory |
| Message-ID | <tyVW2-7fF-5@gated-at.bofh.it> |
| In reply to | #1628771 |
On Fri, Apr 21, 2017 at 8:30 PM, Jérôme Glisse <jglisse@redhat.com> wrote:
> HMM (heterogeneous memory management) need struct page to support migration
> from system main memory to device memory. Reasons for HMM and migration to
> device memory is explained with HMM core patch.
>
> This patch deals with device memory that is un-addressable memory (ie CPU
> can not access it). Hence we do not want those struct page to be manage
> like regular memory. That is why we extend ZONE_DEVICE to support different
> types of memory.
>
> A persistent memory type is define for existing user of ZONE_DEVICE and a
> new device un-addressable type is added for the un-addressable memory type.
> There is a clear separation between what is expected from each memory type
> and existing user of ZONE_DEVICE are un-affected by new requirement and new
> use of the un-addressable type. All specific code path are protect with
> test against the memory type.
>
> Because memory is un-addressable we use a new special swap type for when
> a page is migrated to device memory (this reduces the number of maximum
> swap file).
>
> The main two additions beside memory type to ZONE_DEVICE is two callbacks.
> First one, page_free() is call whenever page refcount reach 1 (which means
> the page is free as ZONE_DEVICE page never reach a refcount of 0). This
> allow device driver to manage its memory and associated struct page.
>
> The second callback page_fault() happens when there is a CPU access to
> an address that is back by a device page (which are un-addressable by the
> CPU). This callback is responsible to migrate the page back to system
> main memory. Device driver can not block migration back to system memory,
> HMM make sure that such page can not be pin into device memory.
>
> If device is in some error condition and can not migrate memory back then
> a CPU page fault to device memory should end with SIGBUS.
>
> Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
> Cc: Dan Williams <dan.j.williams@intel.com>
> Cc: Ross Zwisler <ross.zwisler@linux.intel.com>
> ---
> fs/proc/task_mmu.c | 7 +++++
> include/linux/ioport.h | 1 +
> include/linux/memremap.h | 82 ++++++++++++++++++++++++++++++++++++++++++++++++
> include/linux/swap.h | 24 ++++++++++++--
> include/linux/swapops.h | 68 +++++++++++++++++++++++++++++++++++++++
> kernel/memremap.c | 43 ++++++++++++++++++++++++-
> mm/Kconfig | 13 ++++++++
> mm/memory.c | 61 +++++++++++++++++++++++++++++++++++
> mm/memory_hotplug.c | 10 ++++--
> mm/mprotect.c | 14 +++++++++
> 10 files changed, 317 insertions(+), 6 deletions(-)
>
> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index f0c8b33..a12ba94 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
> @@ -542,6 +542,8 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr,
> }
> } else if (is_migration_entry(swpent))
> page = migration_entry_to_page(swpent);
> + else if (is_device_entry(swpent))
> + page = device_entry_to_page(swpent);
> } else if (unlikely(IS_ENABLED(CONFIG_SHMEM) && mss->check_shmem_swap
> && pte_none(*pte))) {
> page = find_get_entry(vma->vm_file->f_mapping,
> @@ -704,6 +706,8 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask,
>
> if (is_migration_entry(swpent))
> page = migration_entry_to_page(swpent);
> + else if (is_device_entry(swpent))
> + page = device_entry_to_page(swpent);
> }
> if (page) {
> int mapcount = page_mapcount(page);
> @@ -1196,6 +1200,9 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm,
> flags |= PM_SWAP;
> if (is_migration_entry(entry))
> page = migration_entry_to_page(entry);
> +
> + if (is_device_entry(entry))
> + page = device_entry_to_page(entry);
> }
>
> if (page && !PageAnon(page))
> diff --git a/include/linux/ioport.h b/include/linux/ioport.h
> index 6230064..ec619dc 100644
> --- a/include/linux/ioport.h
> +++ b/include/linux/ioport.h
> @@ -130,6 +130,7 @@ enum {
> IORES_DESC_ACPI_NV_STORAGE = 3,
> IORES_DESC_PERSISTENT_MEMORY = 4,
> IORES_DESC_PERSISTENT_MEMORY_LEGACY = 5,
> + IORES_DESC_DEVICE_MEMORY_UNADDRESSABLE = 6,
> };
>
> /Fput* helpers to define resources */
> diff --git a/include/linux/memremap.h b/include/linux/memremap.h
> index 9341619..365fb4e 100644
> --- a/include/linux/memremap.h
> +++ b/include/linux/memremap.h
> @@ -35,24 +35,101 @@ static inline struct vmem_altmap *to_vmem_altmap(unsigned long memmap_start)
> }
> #endif
>
> +/*
> + * Specialize ZONE_DEVICE memory into multiple types each having differents
> + * usage.
> + *
> + * MEMORY_DEVICE_PERSISTENT:
> + * Persistent device memory (pmem): struct page might be allocated in different
> + * memory and architecture might want to perform special actions. It is similar
> + * to regular memory, in that the CPU can access it transparently. However,
> + * it is likely to have different bandwidth and latency than regular memory.
> + * See Documentation/nvdimm/nvdimm.txt for more information.
> + *
> + * MEMORY_DEVICE_UNADDRESSABLE:
> + * Device memory that is not directly addressable by the CPU: CPU can neither
> + * read nor write _UNADDRESSABLE memory. In this case, we do still have struct
> + * pages backing the device memory. Doing so simplifies the implementation, but
> + * it is important to remember that there are certain points at which the struct
> + * page must be treated as an opaque object, rather than a "normal" struct page.
> + * A more complete discussion of unaddressable memory may be found in
> + * include/linux/hmm.h and Documentation/vm/hmm.txt.
> + */
> +enum memory_type {
> + MEMORY_DEVICE_PERSISTENT = 0,
> + MEMORY_DEVICE_UNADDRESSABLE,
> +};
Ok, this is a bikeshed, but I think it is important. I think these
should be called MEMORY_DEVICE_PUBLIC and MEMORY_DEVICE_PRIVATE. The
reason is that persistence has nothing to do with the code paths that
deal with the pmem use case of ZONE_DEVICE. The only property the mm
cares about is that the address range behaves the same as host memory
for dma and cpu accesses. The "unaddressable" designation always
confuses me because a memory range isn't memory if it's
"unaddressable". It is addressable, it's just "private" to the device.
> +
> +/*
> + * For MEMORY_DEVICE_UNADDRESSABLE we use ZONE_DEVICE and extend it with two
> + * callbacks:
> + * page_fault()
> + * page_free()
> + *
> + * Additional notes about MEMORY_DEVICE_UNADDRESSABLE may be found in
> + * include/linux/hmm.h and Documentation/vm/hmm.txt. There is also a brief
> + * explanation in include/linux/memory_hotplug.h.
> + *
> + * The page_fault() callback must migrate page back, from device memory to
> + * system memory, so that the CPU can access it. This might fail for various
> + * reasons (device issues, device have been unplugged, ...). When such error
> + * conditions happen, the page_fault() callback must return VM_FAULT_SIGBUS and
> + * set the CPU page table entry to "poisoned".
> + *
> + * Note that because memory cgroup charges are transferred to the device memory,
> + * this should never fail due to memory restrictions. However, allocation
> + * of a regular system page might still fail because we are out of memory. If
> + * that happens, the page_fault() callback must return VM_FAULT_OOM.
> + *
> + * The page_fault() callback can also try to migrate back multiple pages in one
> + * chunk, as an optimization. It must, however, prioritize the faulting address
> + * over all the others.
> + *
> + *
> + * The page_free() callback is called once the page refcount reaches 1
> + * (ZONE_DEVICE pages never reach 0 refcount unless there is a refcount bug.
> + * This allows the device driver to implement its own memory management.)
> + */
> +typedef int (*dev_page_fault_t)(struct vm_area_struct *vma,
> + unsigned long addr,
> + struct page *page,
> + unsigned int flags,
> + pmd_t *pmdp);
> +typedef void (*dev_page_free_t)(struct page *page, void *data);
> +
> /**
> * struct dev_pagemap - metadata for ZONE_DEVICE mappings
> + * @page_fault: callback when CPU fault on an unaddressable device page
> + * @page_free: free page callback when page refcount reaches 1
> * @altmap: pre-allocated/reserved memory for vmemmap allocations
> * @res: physical address range covered by @ref
> * @ref: reference count that pins the devm_memremap_pages() mapping
> * @dev: host device of the mapping for debug
> + * @data: private data pointer for page_free()
> + * @type: memory type: see MEMORY_* in memory_hotplug.h
> */
> struct dev_pagemap {
> + dev_page_fault_t page_fault;
> + dev_page_free_t page_free;
> struct vmem_altmap *altmap;
> const struct resource *res;
> struct percpu_ref *ref;
> struct device *dev;
> + void *data;
> + enum memory_type type;
> };
I think the only attribute that belongs here is @type, the rest are
specific to the device_private case.
struct dev_private_pagemap {
dev_page_fault_t page_fault;
dev_page_free_t page_free;
void *data;
struct dev_pagemap pgmap;
};
As Logan pointed out we can kill the internal struct page_map in
kernel/memremap.c and let devm_memremap_pages() take a "struct
dev_pagemap *" pointer. That way when future dev_pagemap users come
along they can wrap the core structure with whatever specific data
they need for their use case.
>
> #ifdef CONFIG_ZONE_DEVICE
> void *devm_memremap_pages(struct device *dev, struct resource *res,
> struct percpu_ref *ref, struct vmem_altmap *altmap);
> struct dev_pagemap *find_dev_pagemap(resource_size_t phys);
> +
> +static inline bool is_device_unaddressable_page(const struct page *page)
> +{
> + /* See MEMORY_DEVICE_UNADDRESSABLE in include/linux/memory_hotplug.h */
> + return ((page_zonenum(page) == ZONE_DEVICE) &&
> + (page->pgmap->type == MEMORY_DEVICE_UNADDRESSABLE));
> +}
> #else
> static inline void *devm_memremap_pages(struct device *dev,
> struct resource *res, struct percpu_ref *ref,
> @@ -71,6 +148,11 @@ static inline struct dev_pagemap *find_dev_pagemap(resource_size_t phys)
> {
> return NULL;
> }
> +
> +static inline bool is_device_unaddressable_page(const struct page *page)
> +{
> + return false;
> +}
> #endif
>
> /**
> diff --git a/include/linux/swap.h b/include/linux/swap.h
> index ba58824..52a137d 100644
> --- a/include/linux/swap.h
> +++ b/include/linux/swap.h
> @@ -51,6 +51,23 @@ static inline int current_is_kswapd(void)
> */
>
> /*
> + * Unaddressable device memory support. See include/linux/hmm.h and
> + * Documentation/vm/hmm.txt. Short description is we need struct pages for
> + * device memory that is unaddressable (inaccessible) by CPU, so that we can
> + * migrate part of a process memory to device memory.
> + *
> + * When a page is migrated from CPU to device, we set the CPU page table entry
> + * to a special SWP_DEVICE_* entry.
> + */
> +#ifdef CONFIG_DEVICE_UNADDRESSABLE
> +#define SWP_DEVICE_NUM 2
> +#define SWP_DEVICE_WRITE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM)
> +#define SWP_DEVICE_READ (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+1)
> +#else
> +#define SWP_DEVICE_NUM 0
> +#endif
> +
> +/*
> * NUMA node memory migration support
> */
> #ifdef CONFIG_MIGRATION
> @@ -72,7 +89,8 @@ static inline int current_is_kswapd(void)
> #endif
>
> #define MAX_SWAPFILES \
> - ((1 << MAX_SWAPFILES_SHIFT) - SWP_MIGRATION_NUM - SWP_HWPOISON_NUM)
> + ((1 << MAX_SWAPFILES_SHIFT) - SWP_DEVICE_NUM - \
> + SWP_MIGRATION_NUM - SWP_HWPOISON_NUM)
>
> /*
> * Magic header for a swap area. The first part of the union is
> @@ -432,8 +450,8 @@ static inline void show_swap_cache_info(void)
> {
> }
>
> -#define free_swap_and_cache(swp) is_migration_entry(swp)
> -#define swapcache_prepare(swp) is_migration_entry(swp)
> +#define free_swap_and_cache(e) (is_migration_entry(e) || is_device_entry(e))
> +#define swapcache_prepare(e) (is_migration_entry(e) || is_device_entry(e))
>
> static inline int add_swap_count_continuation(swp_entry_t swp, gfp_t gfp_mask)
> {
> diff --git a/include/linux/swapops.h b/include/linux/swapops.h
> index 5c3a5f3..960eb6b 100644
> --- a/include/linux/swapops.h
> +++ b/include/linux/swapops.h
> @@ -100,6 +100,74 @@ static inline void *swp_to_radix_entry(swp_entry_t entry)
> return (void *)(value | RADIX_TREE_EXCEPTIONAL_ENTRY);
> }
>
> +#if IS_ENABLED(CONFIG_DEVICE_UNADDRESSABLE)
> +static inline swp_entry_t make_device_entry(struct page *page, bool write)
> +{
> + return swp_entry(write ? SWP_DEVICE_WRITE : SWP_DEVICE_READ,
> + page_to_pfn(page));
> +}
> +
> +static inline bool is_device_entry(swp_entry_t entry)
> +{
> + int type = swp_type(entry);
> + return type == SWP_DEVICE_READ || type == SWP_DEVICE_WRITE;
> +}
> +
> +static inline void make_device_entry_read(swp_entry_t *entry)
> +{
> + *entry = swp_entry(SWP_DEVICE_READ, swp_offset(*entry));
> +}
> +
> +static inline bool is_write_device_entry(swp_entry_t entry)
> +{
> + return unlikely(swp_type(entry) == SWP_DEVICE_WRITE);
> +}
> +
> +static inline struct page *device_entry_to_page(swp_entry_t entry)
> +{
> + return pfn_to_page(swp_offset(entry));
> +}
> +
> +int device_entry_fault(struct vm_area_struct *vma,
> + unsigned long addr,
> + swp_entry_t entry,
> + unsigned int flags,
> + pmd_t *pmdp);
The use of the "device" term is ambiguous. Since these changes are
specific to the MEMORY_DEVICE_PRIVATE can we reflect "private" in the
name? Change "device" to "devpriv" or "device_private"?
> +#else /* CONFIG_DEVICE_UNADDRESSABLE */
> +static inline swp_entry_t make_device_entry(struct page *page, bool write)
> +{
> + return swp_entry(0, 0);
> +}
> +
> +static inline void make_device_entry_read(swp_entry_t *entry)
> +{
> +}
> +
> +static inline bool is_device_entry(swp_entry_t entry)
> +{
> + return false;
> +}
> +
> +static inline bool is_write_device_entry(swp_entry_t entry)
> +{
> + return false;
> +}
> +
> +static inline struct page *device_entry_to_page(swp_entry_t entry)
> +{
> + return NULL;
> +}
> +
> +static inline int device_entry_fault(struct vm_area_struct *vma,
> + unsigned long addr,
> + swp_entry_t entry,
> + unsigned int flags,
> + pmd_t *pmdp)
> +{
> + return VM_FAULT_SIGBUS;
> +}
> +#endif /* CONFIG_DEVICE_UNADDRESSABLE */
> +
> #ifdef CONFIG_MIGRATION
> static inline swp_entry_t make_migration_entry(struct page *page, int write)
> {
> diff --git a/kernel/memremap.c b/kernel/memremap.c
> index 97ef676..dce3b29 100644
> --- a/kernel/memremap.c
> +++ b/kernel/memremap.c
> @@ -18,6 +18,8 @@
> #include <linux/io.h>
> #include <linux/mm.h>
> #include <linux/memory_hotplug.h>
> +#include <linux/swap.h>
> +#include <linux/swapops.h>
>
> #ifndef ioremap_cache
> /* temporary while we convert existing ioremap_cache users to memremap */
> @@ -194,12 +196,47 @@ void put_zone_device_page(struct page *page)
> * ZONE_DEVICE page refcount should never reach 0 and never be freed
> * to kernel memory allocator.
> */
> - page_ref_dec(page);
> + int count = page_ref_dec_return(page);
> +
> + /*
> + * If refcount is 1 then page is freed and refcount is stable as nobody
> + * holds a reference on the page.
> + */
> + if (page->pgmap->page_free && count == 1)
> + page->pgmap->page_free(page, page->pgmap->data);
Reading this it isn't clear whether we are dealing with the public or
private case. I think this should be:
if (count == 1)
zone_device_page_free(page);
...where zone_device_page_free() is a helper that is explicit about
the expectations of which ZONE_DEVICE types need to take action on a
'free' event.
static void zone_device_page_free(struct page *page)
{
struct dev_private_pagemap *priv_pgmap;
if (page->pgmap->type == MEMORY_DEVICE_PUBLIC)
return;
priv_pgmap = container_of(page->pgmap, typeof(*priv_pgmap), pgmap);
priv_pgmap->page_free(page);
}
>
> put_dev_pagemap(page->pgmap);
> }
> EXPORT_SYMBOL(put_zone_device_page);
>
> +#if IS_ENABLED(CONFIG_DEVICE_UNADDRESSABLE)
> +int device_entry_fault(struct vm_area_struct *vma,
> + unsigned long addr,
> + swp_entry_t entry,
> + unsigned int flags,
> + pmd_t *pmdp)
> +{
> + struct page *page = device_entry_to_page(entry);
> +
> + /*
> + * The page_fault() callback must migrate page back to system memory
> + * so that CPU can access it. This might fail for various reasons
> + * (device issue, device was unsafely unplugged, ...). When such
> + * error conditions happen, the callback must return VM_FAULT_SIGBUS.
> + *
> + * Note that because memory cgroup charges are accounted to the device
> + * memory, this should never fail because of memory restrictions (but
> + * allocation of regular system page might still fail because we are
> + * out of memory).
> + *
> + * There is a more in-depth description of what that callback can and
> + * cannot do, in include/linux/memremap.h
> + */
> + return page->pgmap->page_fault(vma, addr, page, flags, pmdp);
> +}
> +EXPORT_SYMBOL(device_entry_fault);
> +#endif /* CONFIG_DEVICE_UNADDRESSABLE */
> +
> static void pgmap_radix_release(struct resource *res)
> {
> resource_size_t key, align_start, align_size, align_end;
> @@ -332,6 +369,10 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,
> }
> pgmap->ref = ref;
> pgmap->res = &page_map->res;
> + pgmap->type = MEMORY_DEVICE_PERSISTENT;
> + pgmap->page_fault = NULL;
> + pgmap->page_free = NULL;
> + pgmap->data = NULL;
This goes away if we convert to passing in the dev_pagemap, and I
think that is a useful cleanup.
> mutex_lock(&pgmap_lock);
> error = 0;
> diff --git a/mm/Kconfig b/mm/Kconfig
> index c89f472..ad4bd50 100644
> --- a/mm/Kconfig
> +++ b/mm/Kconfig
> @@ -700,6 +700,19 @@ config ZONE_DEVICE
>
> If FS_DAX is enabled, then say Y.
>
> +config DEVICE_UNADDRESSABLE
> + bool "Unaddressable device memory (GPU memory, ...)"
> + depends on X86_64
> + depends on ZONE_DEVICE
> + depends on MEMORY_HOTPLUG
> + depends on MEMORY_HOTREMOVE
> + depends on SPARSEMEM_VMEMMAP
> +
> + help
> + Allows creation of struct pages to represent unaddressable device
> + memory; i.e., memory that is only accessible from the device (or
> + group of devices).
> +
> config FRAME_VECTOR
> bool
>
> diff --git a/mm/memory.c b/mm/memory.c
> index 6ff5d72..4116ab2 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -49,6 +49,7 @@
> #include <linux/swap.h>
> #include <linux/highmem.h>
> #include <linux/pagemap.h>
> +#include <linux/memremap.h>
> #include <linux/ksm.h>
> #include <linux/rmap.h>
> #include <linux/export.h>
> @@ -927,6 +928,35 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
> pte = pte_swp_mksoft_dirty(pte);
> set_pte_at(src_mm, addr, src_pte, pte);
> }
> + } else if (is_device_entry(entry)) {
> + page = device_entry_to_page(entry);
> +
> + /*
> + * Update rss count even for unaddressable pages, as
> + * they should treated just like normal pages in this
> + * respect.
> + *
> + * We will likely want to have some new rss counters
> + * for unaddressable pages, at some point. But for now
> + * keep things as they are.
> + */
> + get_page(page);
> + rss[mm_counter(page)]++;
> + page_dup_rmap(page, false);
> +
> + /*
> + * We do not preserve soft-dirty information, because so
> + * far, checkpoint/restore is the only feature that
> + * requires that. And checkpoint/restore does not work
> + * when a device driver is involved (you cannot easily
> + * save and restore device driver state).
> + */
> + if (is_write_device_entry(entry) &&
> + is_cow_mapping(vm_flags)) {
> + make_device_entry_read(&entry);
> + pte = swp_entry_to_pte(entry);
> + set_pte_at(src_mm, addr, src_pte, pte);
> + }
> }
> goto out_set_pte;
> }
> @@ -1243,6 +1273,29 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
> }
> continue;
> }
> +
> + entry = pte_to_swp_entry(ptent);
> + if (non_swap_entry(entry) && is_device_entry(entry)) {
> + struct page *page = device_entry_to_page(entry);
> +
> + if (unlikely(details && details->check_mapping)) {
> + /*
> + * unmap_shared_mapping_pages() wants to
> + * invalidate cache without truncating:
> + * unmap shared but keep private pages.
> + */
> + if (details->check_mapping !=
> + page_rmapping(page))
> + continue;
> + }
> +
> + pte_clear_not_present_full(mm, addr, pte, tlb->fullmm);
> + rss[mm_counter(page)]--;
> + page_remove_rmap(page, false);
> + put_page(page);
> + continue;
> + }
> +
> /* If details->check_mapping, we leave swap entries. */
> if (unlikely(details))
> continue;
> @@ -2690,6 +2743,14 @@ int do_swap_page(struct vm_fault *vmf)
> if (is_migration_entry(entry)) {
> migration_entry_wait(vma->vm_mm, vmf->pmd,
> vmf->address);
> + } else if (is_device_entry(entry)) {
> + /*
> + * For un-addressable device memory we call the pgmap
> + * fault handler callback. The callback must migrate
> + * the page back to some CPU accessible page.
> + */
> + ret = device_entry_fault(vma, vmf->address, entry,
> + vmf->flags, vmf->pmd);
> } else if (is_hwpoison_entry(entry)) {
> ret = VM_FAULT_HWPOISON;
> } else {
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index de30822..d988fbe 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -156,7 +156,7 @@ void mem_hotplug_done(void)
> /* add this memory to iomem resource */
> static struct resource *register_memory_resource(u64 start, u64 size)
> {
> - struct resource *res;
> + struct resource *res, *conflict;
> res = kzalloc(sizeof(struct resource), GFP_KERNEL);
> if (!res)
> return ERR_PTR(-ENOMEM);
> @@ -165,7 +165,13 @@ static struct resource *register_memory_resource(u64 start, u64 size)
> res->start = start;
> res->end = start + size - 1;
> res->flags = IORESOURCE_SYSTEM_RAM | IORESOURCE_BUSY;
> - if (request_resource(&iomem_resource, res) < 0) {
> + conflict = request_resource_conflict(&iomem_resource, res);
> + if (conflict) {
> + if (conflict->desc == IORES_DESC_DEVICE_MEMORY_UNADDRESSABLE) {
> + pr_debug("Device unaddressable memory block "
> + "memory hotplug at %#010llx !\n",
> + (unsigned long long)start);
> + }
> pr_debug("System RAM resource %pR cannot be added\n", res);
> kfree(res);
> return ERR_PTR(-EEXIST);
> diff --git a/mm/mprotect.c b/mm/mprotect.c
> index 8edd0d5..dadb020 100644
> --- a/mm/mprotect.c
> +++ b/mm/mprotect.c
> @@ -126,6 +126,20 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
>
> pages++;
> }
> +
> + if (is_write_device_entry(entry)) {
> + pte_t newpte;
> +
> + /*
> + * We do not preserve soft-dirtiness. See
> + * copy_one_pte() for explanation.
> + */
> + make_device_entry_read(&entry);
> + newpte = swp_entry_to_pte(entry);
> + set_pte_at(mm, addr, pte, newpte);
> +
> + pages++;
> + }
> }
> } while (pte++, addr += PAGE_SIZE, addr != end);
> arch_leave_lazy_mmu_mode();
I think this is a good example of how the usage of the generic
"device" term makes the new "private" code paths harder to spot /
discern from the current "public" requirements.
[toc] | [prev] | [next] | [standalone]
| From | Jerome Glisse <jglisse@redhat.com> |
|---|---|
| Date | 2017-04-22 20:20 +0200 |
| Subject | Re: [HMM 03/15] mm/unaddressable-memory: new type of ZONE_DEVICE for unaddressable memory |
| Message-ID | <tz7Nv-6cn-5@gated-at.bofh.it> |
| In reply to | #1628779 |
On Fri, Apr 21, 2017 at 10:30:01PM -0700, Dan Williams wrote:
> On Fri, Apr 21, 2017 at 8:30 PM, Jérôme Glisse <jglisse@redhat.com> wrote:
[...]
> > +/*
> > + * Specialize ZONE_DEVICE memory into multiple types each having differents
> > + * usage.
> > + *
> > + * MEMORY_DEVICE_PERSISTENT:
> > + * Persistent device memory (pmem): struct page might be allocated in different
> > + * memory and architecture might want to perform special actions. It is similar
> > + * to regular memory, in that the CPU can access it transparently. However,
> > + * it is likely to have different bandwidth and latency than regular memory.
> > + * See Documentation/nvdimm/nvdimm.txt for more information.
> > + *
> > + * MEMORY_DEVICE_UNADDRESSABLE:
> > + * Device memory that is not directly addressable by the CPU: CPU can neither
> > + * read nor write _UNADDRESSABLE memory. In this case, we do still have struct
> > + * pages backing the device memory. Doing so simplifies the implementation, but
> > + * it is important to remember that there are certain points at which the struct
> > + * page must be treated as an opaque object, rather than a "normal" struct page.
> > + * A more complete discussion of unaddressable memory may be found in
> > + * include/linux/hmm.h and Documentation/vm/hmm.txt.
> > + */
> > +enum memory_type {
> > + MEMORY_DEVICE_PERSISTENT = 0,
> > + MEMORY_DEVICE_UNADDRESSABLE,
> > +};
>
> Ok, this is a bikeshed, but I think it is important. I think these
> should be called MEMORY_DEVICE_PUBLIC and MEMORY_DEVICE_PRIVATE. The
> reason is that persistence has nothing to do with the code paths that
> deal with the pmem use case of ZONE_DEVICE. The only property the mm
> cares about is that the address range behaves the same as host memory
> for dma and cpu accesses. The "unaddressable" designation always
> confuses me because a memory range isn't memory if it's
> "unaddressable". It is addressable, it's just "private" to the device.
I can change the name but the memory is truely unaddressable, the CPU
can not access it whatsoever (well it can access a small window but
even that is not guaranteed).
> > +/*
> > + * For MEMORY_DEVICE_UNADDRESSABLE we use ZONE_DEVICE and extend it with two
> > + * callbacks:
> > + * page_fault()
> > + * page_free()
> > + *
> > + * Additional notes about MEMORY_DEVICE_UNADDRESSABLE may be found in
> > + * include/linux/hmm.h and Documentation/vm/hmm.txt. There is also a brief
> > + * explanation in include/linux/memory_hotplug.h.
> > + *
> > + * The page_fault() callback must migrate page back, from device memory to
> > + * system memory, so that the CPU can access it. This might fail for various
> > + * reasons (device issues, device have been unplugged, ...). When such error
> > + * conditions happen, the page_fault() callback must return VM_FAULT_SIGBUS and
> > + * set the CPU page table entry to "poisoned".
> > + *
> > + * Note that because memory cgroup charges are transferred to the device memory,
> > + * this should never fail due to memory restrictions. However, allocation
> > + * of a regular system page might still fail because we are out of memory. If
> > + * that happens, the page_fault() callback must return VM_FAULT_OOM.
> > + *
> > + * The page_fault() callback can also try to migrate back multiple pages in one
> > + * chunk, as an optimization. It must, however, prioritize the faulting address
> > + * over all the others.
> > + *
> > + *
> > + * The page_free() callback is called once the page refcount reaches 1
> > + * (ZONE_DEVICE pages never reach 0 refcount unless there is a refcount bug.
> > + * This allows the device driver to implement its own memory management.)
> > + */
> > +typedef int (*dev_page_fault_t)(struct vm_area_struct *vma,
> > + unsigned long addr,
> > + struct page *page,
> > + unsigned int flags,
> > + pmd_t *pmdp);
> > +typedef void (*dev_page_free_t)(struct page *page, void *data);
> > +
> > /**
> > * struct dev_pagemap - metadata for ZONE_DEVICE mappings
> > + * @page_fault: callback when CPU fault on an unaddressable device page
> > + * @page_free: free page callback when page refcount reaches 1
> > * @altmap: pre-allocated/reserved memory for vmemmap allocations
> > * @res: physical address range covered by @ref
> > * @ref: reference count that pins the devm_memremap_pages() mapping
> > * @dev: host device of the mapping for debug
> > + * @data: private data pointer for page_free()
> > + * @type: memory type: see MEMORY_* in memory_hotplug.h
> > */
> > struct dev_pagemap {
> > + dev_page_fault_t page_fault;
> > + dev_page_free_t page_free;
> > struct vmem_altmap *altmap;
> > const struct resource *res;
> > struct percpu_ref *ref;
> > struct device *dev;
> > + void *data;
> > + enum memory_type type;
> > };
>
> I think the only attribute that belongs here is @type, the rest are
> specific to the device_private case.
>
> struct dev_private_pagemap {
> dev_page_fault_t page_fault;
> dev_page_free_t page_free;
> void *data;
> struct dev_pagemap pgmap;
> };
>
> As Logan pointed out we can kill the internal struct page_map in
> kernel/memremap.c and let devm_memremap_pages() take a "struct
> dev_pagemap *" pointer. That way when future dev_pagemap users come
> along they can wrap the core structure with whatever specific data
> they need for their use case.
I don't want to make that change as part of HMM, this can be done
as a patchset on top. Also it is easier to add callback to dev_pagemap
especially as we want p2p and HMM side by side not as something
exclusive.
[...]
> > +#if IS_ENABLED(CONFIG_DEVICE_UNADDRESSABLE)
> > +static inline swp_entry_t make_device_entry(struct page *page, bool write)
> > +{
> > + return swp_entry(write ? SWP_DEVICE_WRITE : SWP_DEVICE_READ,
> > + page_to_pfn(page));
> > +}
> > +
> > +static inline bool is_device_entry(swp_entry_t entry)
> > +{
> > + int type = swp_type(entry);
> > + return type == SWP_DEVICE_READ || type == SWP_DEVICE_WRITE;
> > +}
> > +
> > +static inline void make_device_entry_read(swp_entry_t *entry)
> > +{
> > + *entry = swp_entry(SWP_DEVICE_READ, swp_offset(*entry));
> > +}
> > +
> > +static inline bool is_write_device_entry(swp_entry_t entry)
> > +{
> > + return unlikely(swp_type(entry) == SWP_DEVICE_WRITE);
> > +}
> > +
> > +static inline struct page *device_entry_to_page(swp_entry_t entry)
> > +{
> > + return pfn_to_page(swp_offset(entry));
> > +}
> > +
> > +int device_entry_fault(struct vm_area_struct *vma,
> > + unsigned long addr,
> > + swp_entry_t entry,
> > + unsigned int flags,
> > + pmd_t *pmdp);
>
>
> The use of the "device" term is ambiguous. Since these changes are
> specific to the MEMORY_DEVICE_PRIVATE can we reflect "private" in the
> name? Change "device" to "devpriv" or "device_private"?
Yes we can.
[...]
> > @@ -194,12 +196,47 @@ void put_zone_device_page(struct page *page)
> > * ZONE_DEVICE page refcount should never reach 0 and never be freed
> > * to kernel memory allocator.
> > */
> > - page_ref_dec(page);
> > + int count = page_ref_dec_return(page);
> > +
> > + /*
> > + * If refcount is 1 then page is freed and refcount is stable as nobody
> > + * holds a reference on the page.
> > + */
> > + if (page->pgmap->page_free && count == 1)
> > + page->pgmap->page_free(page, page->pgmap->data);
>
>
> Reading this it isn't clear whether we are dealing with the public or
> private case. I think this should be:
>
> if (count == 1)
> zone_device_page_free(page);
>
> ...where zone_device_page_free() is a helper that is explicit about
> the expectations of which ZONE_DEVICE types need to take action on a
> 'free' event.
>
> static void zone_device_page_free(struct page *page)
> {
> struct dev_private_pagemap *priv_pgmap;
>
> if (page->pgmap->type == MEMORY_DEVICE_PUBLIC)
> return;
>
> priv_pgmap = container_of(page->pgmap, typeof(*priv_pgmap), pgmap);
> priv_pgmap->page_free(page);
> }
Ok.
> > @@ -332,6 +369,10 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,
> > }
> > pgmap->ref = ref;
> > pgmap->res = &page_map->res;
> > + pgmap->type = MEMORY_DEVICE_PERSISTENT;
> > + pgmap->page_fault = NULL;
> > + pgmap->page_free = NULL;
> > + pgmap->data = NULL;
>
>
> This goes away if we convert to passing in the dev_pagemap, and I
> think that is a useful cleanup.
Yes but i rather do it as a patchset on top as i am sure p2p folks
will want to give input and i don't want to delay until we can agree
on exact fields we want/need.
[...]
> > diff --git a/mm/mprotect.c b/mm/mprotect.c
> > index 8edd0d5..dadb020 100644
> > --- a/mm/mprotect.c
> > +++ b/mm/mprotect.c
> > @@ -126,6 +126,20 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
> >
> > pages++;
> > }
> > +
> > + if (is_write_device_entry(entry)) {
> > + pte_t newpte;
> > +
> > + /*
> > + * We do not preserve soft-dirtiness. See
> > + * copy_one_pte() for explanation.
> > + */
> > + make_device_entry_read(&entry);
> > + newpte = swp_entry_to_pte(entry);
> > + set_pte_at(mm, addr, pte, newpte);
> > +
> > + pages++;
> > + }
> > }
> > } while (pte++, addr += PAGE_SIZE, addr != end);
> > arch_leave_lazy_mmu_mode();
>
> I think this is a good example of how the usage of the generic
> "device" term makes the new "private" code paths harder to spot /
> discern from the current "public" requirements.
I will change to devpriv
Thank for reviewing.
Cheers,
Jérôme
[toc] | [prev] | [next] | [standalone]
| From | Dan Williams <dan.j.williams@intel.com> |
|---|---|
| Date | 2017-04-23 15:20 +0200 |
| Subject | Re: [HMM 03/15] mm/unaddressable-memory: new type of ZONE_DEVICE for unaddressable memory |
| Message-ID | <tzpAJ-yN-1@gated-at.bofh.it> |
| In reply to | #1628871 |
On Sat, Apr 22, 2017 at 11:11 AM, Jerome Glisse <jglisse@redhat.com> wrote:
> On Fri, Apr 21, 2017 at 10:30:01PM -0700, Dan Williams wrote:
>> On Fri, Apr 21, 2017 at 8:30 PM, Jérôme Glisse <jglisse@redhat.com> wrote:
>
> [...]
>
>> > +/*
>> > + * Specialize ZONE_DEVICE memory into multiple types each having differents
>> > + * usage.
>> > + *
>> > + * MEMORY_DEVICE_PERSISTENT:
>> > + * Persistent device memory (pmem): struct page might be allocated in different
>> > + * memory and architecture might want to perform special actions. It is similar
>> > + * to regular memory, in that the CPU can access it transparently. However,
>> > + * it is likely to have different bandwidth and latency than regular memory.
>> > + * See Documentation/nvdimm/nvdimm.txt for more information.
>> > + *
>> > + * MEMORY_DEVICE_UNADDRESSABLE:
>> > + * Device memory that is not directly addressable by the CPU: CPU can neither
>> > + * read nor write _UNADDRESSABLE memory. In this case, we do still have struct
>> > + * pages backing the device memory. Doing so simplifies the implementation, but
>> > + * it is important to remember that there are certain points at which the struct
>> > + * page must be treated as an opaque object, rather than a "normal" struct page.
>> > + * A more complete discussion of unaddressable memory may be found in
>> > + * include/linux/hmm.h and Documentation/vm/hmm.txt.
>> > + */
>> > +enum memory_type {
>> > + MEMORY_DEVICE_PERSISTENT = 0,
>> > + MEMORY_DEVICE_UNADDRESSABLE,
>> > +};
>>
>> Ok, this is a bikeshed, but I think it is important. I think these
>> should be called MEMORY_DEVICE_PUBLIC and MEMORY_DEVICE_PRIVATE. The
>> reason is that persistence has nothing to do with the code paths that
>> deal with the pmem use case of ZONE_DEVICE. The only property the mm
>> cares about is that the address range behaves the same as host memory
>> for dma and cpu accesses. The "unaddressable" designation always
>> confuses me because a memory range isn't memory if it's
>> "unaddressable". It is addressable, it's just "private" to the device.
>
> I can change the name but the memory is truely unaddressable, the CPU
> can not access it whatsoever (well it can access a small window but
> even that is not guaranteed).
>
Understood, but that's still "addressable only by certain agents or
through a proxy" which seems closer to "private" to me.
>
>> > +/*
>> > + * For MEMORY_DEVICE_UNADDRESSABLE we use ZONE_DEVICE and extend it with two
>> > + * callbacks:
>> > + * page_fault()
>> > + * page_free()
>> > + *
>> > + * Additional notes about MEMORY_DEVICE_UNADDRESSABLE may be found in
>> > + * include/linux/hmm.h and Documentation/vm/hmm.txt. There is also a brief
>> > + * explanation in include/linux/memory_hotplug.h.
>> > + *
>> > + * The page_fault() callback must migrate page back, from device memory to
>> > + * system memory, so that the CPU can access it. This might fail for various
>> > + * reasons (device issues, device have been unplugged, ...). When such error
>> > + * conditions happen, the page_fault() callback must return VM_FAULT_SIGBUS and
>> > + * set the CPU page table entry to "poisoned".
>> > + *
>> > + * Note that because memory cgroup charges are transferred to the device memory,
>> > + * this should never fail due to memory restrictions. However, allocation
>> > + * of a regular system page might still fail because we are out of memory. If
>> > + * that happens, the page_fault() callback must return VM_FAULT_OOM.
>> > + *
>> > + * The page_fault() callback can also try to migrate back multiple pages in one
>> > + * chunk, as an optimization. It must, however, prioritize the faulting address
>> > + * over all the others.
>> > + *
>> > + *
>> > + * The page_free() callback is called once the page refcount reaches 1
>> > + * (ZONE_DEVICE pages never reach 0 refcount unless there is a refcount bug.
>> > + * This allows the device driver to implement its own memory management.)
>> > + */
>> > +typedef int (*dev_page_fault_t)(struct vm_area_struct *vma,
>> > + unsigned long addr,
>> > + struct page *page,
>> > + unsigned int flags,
>> > + pmd_t *pmdp);
>> > +typedef void (*dev_page_free_t)(struct page *page, void *data);
>> > +
>> > /**
>> > * struct dev_pagemap - metadata for ZONE_DEVICE mappings
>> > + * @page_fault: callback when CPU fault on an unaddressable device page
>> > + * @page_free: free page callback when page refcount reaches 1
>> > * @altmap: pre-allocated/reserved memory for vmemmap allocations
>> > * @res: physical address range covered by @ref
>> > * @ref: reference count that pins the devm_memremap_pages() mapping
>> > * @dev: host device of the mapping for debug
>> > + * @data: private data pointer for page_free()
>> > + * @type: memory type: see MEMORY_* in memory_hotplug.h
>> > */
>> > struct dev_pagemap {
>> > + dev_page_fault_t page_fault;
>> > + dev_page_free_t page_free;
>> > struct vmem_altmap *altmap;
>> > const struct resource *res;
>> > struct percpu_ref *ref;
>> > struct device *dev;
>> > + void *data;
>> > + enum memory_type type;
>> > };
>>
>> I think the only attribute that belongs here is @type, the rest are
>> specific to the device_private case.
>>
>> struct dev_private_pagemap {
>> dev_page_fault_t page_fault;
>> dev_page_free_t page_free;
>> void *data;
>> struct dev_pagemap pgmap;
>> };
>>
>> As Logan pointed out we can kill the internal struct page_map in
>> kernel/memremap.c and let devm_memremap_pages() take a "struct
>> dev_pagemap *" pointer. That way when future dev_pagemap users come
>> along they can wrap the core structure with whatever specific data
>> they need for their use case.
>
> I don't want to make that change as part of HMM, this can be done
> as a patchset on top. Also it is easier to add callback to dev_pagemap
> especially as we want p2p and HMM side by side not as something
> exclusive.
Fair enough, I'm ok to do the conversion as a post HMM merge cleanup.
Does the p2p case actually need ->fault() and ->free()? I'd rather we
start with dev_private_pagemap and if p2p needs a subset of the same
fields we can think about a rename or shuffling things around.
>
> [...]
>
>> > +#if IS_ENABLED(CONFIG_DEVICE_UNADDRESSABLE)
>> > +static inline swp_entry_t make_device_entry(struct page *page, bool write)
>> > +{
>> > + return swp_entry(write ? SWP_DEVICE_WRITE : SWP_DEVICE_READ,
>> > + page_to_pfn(page));
>> > +}
>> > +
>> > +static inline bool is_device_entry(swp_entry_t entry)
>> > +{
>> > + int type = swp_type(entry);
>> > + return type == SWP_DEVICE_READ || type == SWP_DEVICE_WRITE;
>> > +}
>> > +
>> > +static inline void make_device_entry_read(swp_entry_t *entry)
>> > +{
>> > + *entry = swp_entry(SWP_DEVICE_READ, swp_offset(*entry));
>> > +}
>> > +
>> > +static inline bool is_write_device_entry(swp_entry_t entry)
>> > +{
>> > + return unlikely(swp_type(entry) == SWP_DEVICE_WRITE);
>> > +}
>> > +
>> > +static inline struct page *device_entry_to_page(swp_entry_t entry)
>> > +{
>> > + return pfn_to_page(swp_offset(entry));
>> > +}
>> > +
>> > +int device_entry_fault(struct vm_area_struct *vma,
>> > + unsigned long addr,
>> > + swp_entry_t entry,
>> > + unsigned int flags,
>> > + pmd_t *pmdp);
>>
>>
>> The use of the "device" term is ambiguous. Since these changes are
>> specific to the MEMORY_DEVICE_PRIVATE can we reflect "private" in the
>> name? Change "device" to "devpriv" or "device_private"?
>
> Yes we can.
>
> [...]
>
>> > @@ -194,12 +196,47 @@ void put_zone_device_page(struct page *page)
>> > * ZONE_DEVICE page refcount should never reach 0 and never be freed
>> > * to kernel memory allocator.
>> > */
>> > - page_ref_dec(page);
>> > + int count = page_ref_dec_return(page);
>> > +
>> > + /*
>> > + * If refcount is 1 then page is freed and refcount is stable as nobody
>> > + * holds a reference on the page.
>> > + */
>> > + if (page->pgmap->page_free && count == 1)
>> > + page->pgmap->page_free(page, page->pgmap->data);
>>
>>
>> Reading this it isn't clear whether we are dealing with the public or
>> private case. I think this should be:
>>
>> if (count == 1)
>> zone_device_page_free(page);
>>
>> ...where zone_device_page_free() is a helper that is explicit about
>> the expectations of which ZONE_DEVICE types need to take action on a
>> 'free' event.
>>
>> static void zone_device_page_free(struct page *page)
>> {
>> struct dev_private_pagemap *priv_pgmap;
>>
>> if (page->pgmap->type == MEMORY_DEVICE_PUBLIC)
>> return;
>>
>> priv_pgmap = container_of(page->pgmap, typeof(*priv_pgmap), pgmap);
>> priv_pgmap->page_free(page);
>> }
>
> Ok.
>
>> > @@ -332,6 +369,10 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,
>> > }
>> > pgmap->ref = ref;
>> > pgmap->res = &page_map->res;
>> > + pgmap->type = MEMORY_DEVICE_PERSISTENT;
>> > + pgmap->page_fault = NULL;
>> > + pgmap->page_free = NULL;
>> > + pgmap->data = NULL;
>>
>>
>> This goes away if we convert to passing in the dev_pagemap, and I
>> think that is a useful cleanup.
>
> Yes but i rather do it as a patchset on top as i am sure p2p folks
> will want to give input and i don't want to delay until we can agree
> on exact fields we want/need.
Ok.
>
> [...]
>
>> > diff --git a/mm/mprotect.c b/mm/mprotect.c
>> > index 8edd0d5..dadb020 100644
>> > --- a/mm/mprotect.c
>> > +++ b/mm/mprotect.c
>> > @@ -126,6 +126,20 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
>> >
>> > pages++;
>> > }
>> > +
>> > + if (is_write_device_entry(entry)) {
>> > + pte_t newpte;
>> > +
>> > + /*
>> > + * We do not preserve soft-dirtiness. See
>> > + * copy_one_pte() for explanation.
>> > + */
>> > + make_device_entry_read(&entry);
>> > + newpte = swp_entry_to_pte(entry);
>> > + set_pte_at(mm, addr, pte, newpte);
>> > +
>> > + pages++;
>> > + }
>> > }
>> > } while (pte++, addr += PAGE_SIZE, addr != end);
>> > arch_leave_lazy_mmu_mode();
>>
>> I think this is a good example of how the usage of the generic
>> "device" term makes the new "private" code paths harder to spot /
>> discern from the current "public" requirements.
>
> I will change to devpriv
>
> Thank for reviewing.
Thanks for working through this late breaking feedback.
[toc] | [prev] | [next] | [standalone]
| From | John Hubbard <jhubbard@nvidia.com> |
|---|---|
| Date | 2017-04-24 02:50 +0200 |
| Subject | Re: [HMM 03/15] mm/unaddressable-memory: new type of ZONE_DEVICE for unaddressable memory |
| Message-ID | <tzAmt-7Jg-5@gated-at.bofh.it> |
| In reply to | #1629022 |
On 4/23/17 6:13 AM, Dan Williams wrote:
> On Sat, Apr 22, 2017 at 11:11 AM, Jerome Glisse <jglisse@redhat.com> wrote:
>> On Fri, Apr 21, 2017 at 10:30:01PM -0700, Dan Williams wrote:
>>> On Fri, Apr 21, 2017 at 8:30 PM, Jérôme Glisse <jglisse@redhat.com> wrote:
>>
>> [...]
>>
>>>> +/*
>>>> + * Specialize ZONE_DEVICE memory into multiple types each having differents
>>>> + * usage.
>>>> + *
>>>> + * MEMORY_DEVICE_PERSISTENT:
>>>> + * Persistent device memory (pmem): struct page might be allocated in different
>>>> + * memory and architecture might want to perform special actions. It is similar
>>>> + * to regular memory, in that the CPU can access it transparently. However,
>>>> + * it is likely to have different bandwidth and latency than regular memory.
>>>> + * See Documentation/nvdimm/nvdimm.txt for more information.
>>>> + *
>>>> + * MEMORY_DEVICE_UNADDRESSABLE:
>>>> + * Device memory that is not directly addressable by the CPU: CPU can neither
>>>> + * read nor write _UNADDRESSABLE memory. In this case, we do still have struct
>>>> + * pages backing the device memory. Doing so simplifies the implementation, but
>>>> + * it is important to remember that there are certain points at which the struct
>>>> + * page must be treated as an opaque object, rather than a "normal" struct page.
>>>> + * A more complete discussion of unaddressable memory may be found in
>>>> + * include/linux/hmm.h and Documentation/vm/hmm.txt.
>>>> + */
>>>> +enum memory_type {
>>>> + MEMORY_DEVICE_PERSISTENT = 0,
>>>> + MEMORY_DEVICE_UNADDRESSABLE,
>>>> +};
>>>
>>> Ok, this is a bikeshed, but I think it is important. I think these
>>> should be called MEMORY_DEVICE_PUBLIC and MEMORY_DEVICE_PRIVATE. The
>>> reason is that persistence has nothing to do with the code paths that
>>> deal with the pmem use case of ZONE_DEVICE. The only property the mm
>>> cares about is that the address range behaves the same as host memory
>>> for dma and cpu accesses. The "unaddressable" designation always
>>> confuses me because a memory range isn't memory if it's
>>> "unaddressable". It is addressable, it's just "private" to the device.
>>
>> I can change the name but the memory is truely unaddressable, the CPU
>> can not access it whatsoever (well it can access a small window but
>> even that is not guaranteed).
>>
>
> Understood, but that's still "addressable only by certain agents or
> through a proxy" which seems closer to "private" to me.
>
Actually, MEMORY_DEVICE_PRIVATE / _PUBLIC seems like a good choice to
me, because the memory may not remain CPU-unaddressable in the future.
By that, I mean that I know of at least one company (ours) that is
working on products that will support hardware-based memory coherence
(and access counters to go along with that). If someone were to enable
HMM on such a system, then the device memory would be, in fact, directly
addressable by a CPU--thus exactly contradicting the "unaddressable" name.
Yes, it is true that we would have to modify HMM anyway, in order to
work in that situation, partly because HMM today relies on CPU and
device page faults in order to work. And it's also true that we might
want to take a different approach than HMM, to support that kind of
device: for example, making it a NUMA node has been debated here, recently.
But even so, I think the potential for the "unaddressable" memory
actually becoming "addressable" someday, is a good argument for using a
different name.
thanks,
--
John Hubbard
NVIDIA
[toc] | [prev] | [next] | [standalone]
| From | Jérôme Glisse <jglisse@redhat.com> |
|---|---|
| Date | 2017-04-22 05:50 +0200 |
| Subject | [HMM 01/15] mm, memory_hotplug: introduce add_pages |
| Message-ID | <tyUdA-64D-5@gated-at.bofh.it> |
| In reply to | #1628767 |
From: Michal Hocko <mhocko@suse.com>
There are new users of memory hotplug emerging. Some of them require
different subset of arch_add_memory. There are some which only require
allocation of struct pages without mapping those pages to the kernel
address space. We currently have __add_pages for that purpose. But this
is rather lowlevel and not very suitable for the code outside of the
memory hotplug. E.g. x86_64 wants to update max_pfn which should be
done by the caller. Introduce add_pages() which should care about those
details if they are needed. Each architecture should define its
implementation and select CONFIG_ARCH_HAS_ADD_PAGES. All others use
the currently existing __add_pages.
Signed-off-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
---
arch/x86/Kconfig | 4 ++++
arch/x86/mm/init_64.c | 22 +++++++++++++++-------
include/linux/memory_hotplug.h | 11 +++++++++++
3 files changed, 30 insertions(+), 7 deletions(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index c43f476..e515dc2 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -2263,6 +2263,10 @@ source "kernel/livepatch/Kconfig"
endmenu
+config ARCH_HAS_ADD_PAGES
+ def_bool y
+ depends on X86_64 && ARCH_ENABLE_MEMORY_HOTPLUG
+
config ARCH_ENABLE_MEMORY_HOTPLUG
def_bool y
depends on X86_64 || (X86_32 && HIGHMEM)
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index ffeba90..a573ebc 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -671,7 +671,7 @@ void __init paging_init(void)
* After memory hotplug the variables max_pfn, max_low_pfn and high_memory need
* updating.
*/
-static void update_end_of_memory_vars(u64 start, u64 size)
+static void update_end_of_memory_vars(u64 start, u64 size)
{
unsigned long end_pfn = PFN_UP(start + size);
@@ -682,22 +682,30 @@ static void update_end_of_memory_vars(u64 start, u64 size)
}
}
-int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
+int add_pages(int nid, unsigned long start_pfn,
+ unsigned long nr_pages, bool want_memblock)
{
- unsigned long start_pfn = start >> PAGE_SHIFT;
- unsigned long nr_pages = size >> PAGE_SHIFT;
int ret;
- init_memory_mapping(start, start + size);
-
ret = __add_pages(nid, start_pfn, nr_pages, want_memblock);
WARN_ON_ONCE(ret);
/* update max_pfn, max_low_pfn and high_memory */
- update_end_of_memory_vars(start, size);
+ update_end_of_memory_vars(start_pfn << PAGE_SHIFT,
+ nr_pages << PAGE_SHIFT);
return ret;
}
+
+int arch_add_memory(int nid, u64 start, u64 size, bool want_memblock)
+{
+ unsigned long start_pfn = start >> PAGE_SHIFT;
+ unsigned long nr_pages = size >> PAGE_SHIFT;
+
+ init_memory_mapping(start, start + size);
+
+ return add_pages(nid, start_pfn, nr_pages, want_memblock);
+}
EXPORT_SYMBOL_GPL(arch_add_memory);
#define PAGE_INUSE 0xFD
diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index aec8865..5ec6d64 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -126,6 +126,17 @@ extern int __remove_pages(struct zone *zone, unsigned long start_pfn,
extern int __add_pages(int nid, unsigned long start_pfn,
unsigned long nr_pages, bool want_memblock);
+#ifndef CONFIG_ARCH_HAS_ADD_PAGES
+static inline int add_pages(int nid, unsigned long start_pfn,
+ unsigned long nr_pages, bool want_memblock)
+{
+ return __add_pages(nid, start_pfn, nr_pages, want_memblock);
+}
+#else /* ARCH_HAS_ADD_PAGES */
+int add_pages(int nid, unsigned long start_pfn,
+ unsigned long nr_pages, bool want_memblock);
+#endif /* ARCH_HAS_ADD_PAGES */
+
#ifdef CONFIG_NUMA
extern int memory_add_physaddr_to_nid(u64 start);
#else
--
2.9.3
[toc] | [prev] | [next] | [standalone]
| From | Jérôme Glisse <jglisse@redhat.com> |
|---|---|
| Date | 2017-04-22 06:10 +0200 |
| Subject | [HMM 07/15] mm/hmm: heterogeneous memory management (HMM for short) v2 |
| Message-ID | <tyUwV-6q7-1@gated-at.bofh.it> |
| In reply to | #1628767 |
HMM provides 3 separate types of functionality:
- Mirroring: synchronize CPU page table and device page table
- Device memory: allocating struct page for device memory
- Migration: migrating regular memory to device memory
This patch introduces some common helpers and definitions to all of
those 3 functionality.
Changed since v1:
- Kconfig logic (depend on x86-64 and use ARCH_HAS pattern)
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
Signed-off-by: Evgeny Baskakov <ebaskakov@nvidia.com>
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
Signed-off-by: Mark Hairgrove <mhairgrove@nvidia.com>
Signed-off-by: Sherry Cheung <SCheung@nvidia.com>
Signed-off-by: Subhash Gutti <sgutti@nvidia.com>
---
MAINTAINERS | 7 +++
include/linux/hmm.h | 146 +++++++++++++++++++++++++++++++++++++++++++++++
include/linux/mm_types.h | 5 ++
kernel/fork.c | 2 +
mm/Kconfig | 13 +++++
mm/Makefile | 1 +
mm/hmm.c | 71 +++++++++++++++++++++++
7 files changed, 245 insertions(+)
create mode 100644 include/linux/hmm.h
create mode 100644 mm/hmm.c
diff --git a/MAINTAINERS b/MAINTAINERS
index 09341a9..6df21a9 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6052,6 +6052,13 @@ S: Supported
F: drivers/scsi/hisi_sas/
F: Documentation/devicetree/bindings/scsi/hisilicon-sas.txt
+HMM - Heterogeneous Memory Management
+M: Jérôme Glisse <jglisse@redhat.com>
+L: linux-mm@kvack.org
+S: Maintained
+F: mm/hmm*
+F: include/linux/hmm*
+
HOST AP DRIVER
M: Jouni Malinen <j@w1.fi>
L: linux-wireless@vger.kernel.org
diff --git a/include/linux/hmm.h b/include/linux/hmm.h
new file mode 100644
index 0000000..93b363d
--- /dev/null
+++ b/include/linux/hmm.h
@@ -0,0 +1,146 @@
+/*
+ * Copyright 2013 Red Hat Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * Authors: Jérôme Glisse <jglisse@redhat.com>
+ */
+/*
+ * Heterogeneous Memory Management (HMM)
+ *
+ * See Documentation/vm/hmm.txt for reasons and overview of what HMM is and it
+ * is for. Here we focus on the HMM API description, with some explanation of
+ * the underlying implementation.
+ *
+ * Short description: HMM provides a set of helpers to share a virtual address
+ * space between CPU and a device, so that the device can access any valid
+ * address of the process (while still obeying memory protection). HMM also
+ * provides helpers to migrate process memory to device memory, and back. Each
+ * set of functionality (address space mirroring, and migration to and from
+ * device memory) can be used independently of the other.
+ *
+ *
+ * HMM address space mirroring API:
+ *
+ * Use HMM address space mirroring if you want to mirror range of the CPU page
+ * table of a process into a device page table. Here, "mirror" means "keep
+ * synchronized". Prerequisites: the device must provide the ability to write-
+ * protect its page tables (at PAGE_SIZE granularity), and must be able to
+ * recover from the resulting potential page faults.
+ *
+ * HMM guarantees that at any point in time, a given virtual address points to
+ * either the same memory in both CPU and device page tables (that is: CPU and
+ * device page tables each point to the same pages), or that one page table (CPU
+ * or device) points to no entry, while the other still points to the old page
+ * for the address. The latter case happens when the CPU page table update
+ * happens first, and then the update is mirrored over to the device page table.
+ * This does not cause any issue, because the CPU page table cannot start
+ * pointing to a new page until the device page table is invalidated.
+ *
+ * HMM uses mmu_notifiers to monitor the CPU page tables, and forwards any
+ * updates to each device driver that has registered a mirror. It also provides
+ * some API calls to help with taking a snapshot of the CPU page table, and to
+ * synchronize with any updates that might happen concurrently.
+ *
+ *
+ * HMM migration to and from device memory:
+ *
+ * HMM provides a set of helpers to hotplug device memory as ZONE_DEVICE, with
+ * a new MEMORY_DEVICE_UNADDRESSABLE type. This provides a struct page for
+ * each page of the device memory, and allows the device driver to manage its
+ * memory using those struct pages. Having struct pages for device memory makes
+ * migration easier. Because that memory is not addressable by the CPU it must
+ * never be pinned to the device; in other words, any CPU page fault can always
+ * cause the device memory to be migrated (copied/moved) back to regular memory.
+ *
+ * A new migrate helper (migrate_vma()) has been added (see mm/migrate.c) that
+ * allows use of a device DMA engine to perform the copy operation between
+ * regular system memory and device memory.
+ */
+#ifndef LINUX_HMM_H
+#define LINUX_HMM_H
+
+#include <linux/kconfig.h>
+
+#if IS_ENABLED(CONFIG_HMM)
+
+
+/*
+ * hmm_pfn_t - HMM uses its own pfn type to keep several flags per page
+ *
+ * Flags:
+ * HMM_PFN_VALID: pfn is valid
+ * HMM_PFN_WRITE: CPU page table has write permission set
+ */
+typedef unsigned long hmm_pfn_t;
+
+#define HMM_PFN_VALID (1 << 0)
+#define HMM_PFN_WRITE (1 << 1)
+#define HMM_PFN_SHIFT 2
+
+/*
+ * hmm_pfn_t_to_page() - return struct page pointed to by a valid hmm_pfn_t
+ * @pfn: hmm_pfn_t to convert to struct page
+ * Returns: struct page pointer if pfn is a valid hmm_pfn_t, NULL otherwise
+ *
+ * If the hmm_pfn_t is valid (ie valid flag set) then return the struct page
+ * matching the pfn value stored in the hmm_pfn_t. Otherwise return NULL.
+ */
+static inline struct page *hmm_pfn_t_to_page(hmm_pfn_t pfn)
+{
+ if (!(pfn & HMM_PFN_VALID))
+ return NULL;
+ return pfn_to_page(pfn >> HMM_PFN_SHIFT);
+}
+
+/*
+ * hmm_pfn_t_to_pfn() - return pfn value store in a hmm_pfn_t
+ * @pfn: hmm_pfn_t to extract pfn from
+ * Returns: pfn value if hmm_pfn_t is valid, -1UL otherwise
+ */
+static inline unsigned long hmm_pfn_t_to_pfn(hmm_pfn_t pfn)
+{
+ if (!(pfn & HMM_PFN_VALID))
+ return -1UL;
+ return (pfn >> HMM_PFN_SHIFT);
+}
+
+/*
+ * hmm_pfn_t_from_page() - create a valid hmm_pfn_t value from struct page
+ * @page: struct page pointer for which to create the hmm_pfn_t
+ * Returns: valid hmm_pfn_t for the page
+ */
+static inline hmm_pfn_t hmm_pfn_t_from_page(struct page *page)
+{
+ return (page_to_pfn(page) << HMM_PFN_SHIFT) | HMM_PFN_VALID;
+}
+
+/*
+ * hmm_pfn_t_from_pfn() - create a valid hmm_pfn_t value from pfn
+ * @pfn: pfn value for which to create the hmm_pfn_t
+ * Returns: valid hmm_pfn_t for the pfn
+ */
+static inline hmm_pfn_t hmm_pfn_t_from_pfn(unsigned long pfn)
+{
+ return (pfn << HMM_PFN_SHIFT) | HMM_PFN_VALID;
+}
+
+
+/* Below are for HMM internal use only! Not to be used by device driver! */
+void hmm_mm_destroy(struct mm_struct *mm);
+
+#else /* IS_ENABLED(CONFIG_HMM) */
+
+/* Below are for HMM internal use only! Not to be used by device driver! */
+static inline void hmm_mm_destroy(struct mm_struct *mm) {}
+
+#endif /* IS_ENABLED(CONFIG_HMM) */
+#endif /* LINUX_HMM_H */
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 45cdb27..5ddeacc7 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -23,6 +23,7 @@
struct address_space;
struct mem_cgroup;
+struct hmm;
/*
* Each physical page in the system has a struct page associated with
@@ -500,6 +501,10 @@ struct mm_struct {
atomic_long_t hugetlb_usage;
#endif
struct work_struct async_put_work;
+#if IS_ENABLED(CONFIG_HMM)
+ /* HMM needs to track a few things per mm */
+ struct hmm *hmm;
+#endif
};
extern struct mm_struct init_mm;
diff --git a/kernel/fork.c b/kernel/fork.c
index 81347bd..65a3e94f 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -37,6 +37,7 @@
#include <linux/binfmts.h>
#include <linux/mman.h>
#include <linux/mmu_notifier.h>
+#include <linux/hmm.h>
#include <linux/fs.h>
#include <linux/mm.h>
#include <linux/vmacache.h>
@@ -891,6 +892,7 @@ void __mmdrop(struct mm_struct *mm)
BUG_ON(mm == &init_mm);
mm_free_pgd(mm);
destroy_context(mm);
+ hmm_mm_destroy(mm);
mmu_notifier_mm_destroy(mm);
check_mm(mm);
put_user_ns(mm->user_ns);
diff --git a/mm/Kconfig b/mm/Kconfig
index ad4bd50..90025b3 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -289,6 +289,19 @@ config MIGRATION
config ARCH_ENABLE_HUGEPAGE_MIGRATION
bool
+config ARCH_HAS_HMM
+ bool
+ default y
+ depends on X86_64
+ depends on ZONE_DEVICE
+ depends on MMU && 64BIT
+ depends on MEMORY_HOTPLUG
+ depends on MEMORY_HOTREMOVE
+ depends on SPARSEMEM_VMEMMAP
+
+config HMM
+ bool
+
config PHYS_ADDR_T_64BIT
def_bool 64BIT || ARCH_PHYS_ADDR_T_64BIT
diff --git a/mm/Makefile b/mm/Makefile
index 026f6a8..9eb4121 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -75,6 +75,7 @@ obj-$(CONFIG_FAILSLAB) += failslab.o
obj-$(CONFIG_MEMORY_HOTPLUG) += memory_hotplug.o
obj-$(CONFIG_MEMTEST) += memtest.o
obj-$(CONFIG_MIGRATION) += migrate.o
+obj-$(CONFIG_HMM) += hmm.o
obj-$(CONFIG_QUICKLIST) += quicklist.o
obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
obj-$(CONFIG_PAGE_COUNTER) += page_counter.o
diff --git a/mm/hmm.c b/mm/hmm.c
new file mode 100644
index 0000000..acadb49
--- /dev/null
+++ b/mm/hmm.c
@@ -0,0 +1,71 @@
+/*
+ * Copyright 2013 Red Hat Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * Authors: Jérôme Glisse <jglisse@redhat.com>
+ */
+/*
+ * Refer to include/linux/hmm.h for information about heterogeneous memory
+ * management or HMM for short.
+ */
+#include <linux/mm.h>
+#include <linux/hmm.h>
+#include <linux/slab.h>
+#include <linux/sched.h>
+
+/*
+ * struct hmm - HMM per mm struct
+ *
+ * @mm: mm struct this HMM struct is bound to
+ */
+struct hmm {
+ struct mm_struct *mm;
+};
+
+/*
+ * hmm_register - register HMM against an mm (HMM internal)
+ *
+ * @mm: mm struct to attach to
+ *
+ * This is not intended to be used directly by device drivers. It allocates an
+ * HMM struct if mm does not have one, and initializes it.
+ */
+static struct hmm *hmm_register(struct mm_struct *mm)
+{
+ if (!mm->hmm) {
+ struct hmm *hmm = NULL;
+
+ hmm = kmalloc(sizeof(*hmm), GFP_KERNEL);
+ if (!hmm)
+ return NULL;
+ hmm->mm = mm;
+
+ spin_lock(&mm->page_table_lock);
+ if (!mm->hmm)
+ mm->hmm = hmm;
+ else
+ kfree(hmm);
+ spin_unlock(&mm->page_table_lock);
+ }
+
+ /*
+ * The hmm struct can only be freed once the mm_struct goes away,
+ * hence we should always have pre-allocated an new hmm struct
+ * above.
+ */
+ return mm->hmm;
+}
+
+void hmm_mm_destroy(struct mm_struct *mm)
+{
+ kfree(mm->hmm);
+}
--
2.9.3
[toc] | [prev] | [next] | [standalone]
| From | Jérôme Glisse <jglisse@redhat.com> |
|---|---|
| Date | 2017-04-22 06:10 +0200 |
| Subject | [HMM 08/15] mm/hmm/mirror: mirror process address space on device with HMM helpers v2 |
| Message-ID | <tyUwV-6q7-3@gated-at.bofh.it> |
| In reply to | #1628767 |
This is a heterogeneous memory management (HMM) process address space
mirroring. In a nutshell this provide an API to mirror process address
space on a device. This boils down to keeping CPU and device page table
synchronize (we assume that both device and CPU are cache coherent like
PCIe device can be).
This patch provide a simple API for device driver to achieve address
space mirroring thus avoiding each device driver to grow its own CPU
page table walker and its own CPU page table synchronization mechanism.
This is useful for NVidia GPU >= Pascal, Mellanox IB >= mlx5 and more
hardware in the future.
Changed since v1:
- Kconfig logic (depend on x86-64 and use ARCH_HAS pattern)
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
Signed-off-by: Evgeny Baskakov <ebaskakov@nvidia.com>
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
Signed-off-by: Mark Hairgrove <mhairgrove@nvidia.com>
Signed-off-by: Sherry Cheung <SCheung@nvidia.com>
Signed-off-by: Subhash Gutti <sgutti@nvidia.com>
---
include/linux/hmm.h | 110 ++++++++++++++++++++++++++++++++++
mm/Kconfig | 12 ++++
mm/hmm.c | 170 +++++++++++++++++++++++++++++++++++++++++++++++-----
3 files changed, 277 insertions(+), 15 deletions(-)
diff --git a/include/linux/hmm.h b/include/linux/hmm.h
index 93b363d..6668a1b 100644
--- a/include/linux/hmm.h
+++ b/include/linux/hmm.h
@@ -72,6 +72,7 @@
#if IS_ENABLED(CONFIG_HMM)
+struct hmm;
/*
* hmm_pfn_t - HMM uses its own pfn type to keep several flags per page
@@ -134,6 +135,115 @@ static inline hmm_pfn_t hmm_pfn_t_from_pfn(unsigned long pfn)
}
+#if IS_ENABLED(CONFIG_HMM_MIRROR)
+/*
+ * Mirroring: how to synchronize device page table with CPU page table.
+ *
+ * A device driver that is participating in HMM mirroring must always
+ * synchronize with CPU page table updates. For this, device drivers can either
+ * directly use mmu_notifier APIs or they can use the hmm_mirror API. Device
+ * drivers can decide to register one mirror per device per process, or just
+ * one mirror per process for a group of devices. The pattern is:
+ *
+ * int device_bind_address_space(..., struct mm_struct *mm, ...)
+ * {
+ * struct device_address_space *das;
+ *
+ * // Device driver specific initialization, and allocation of das
+ * // which contains an hmm_mirror struct as one of its fields.
+ * ...
+ *
+ * ret = hmm_mirror_register(&das->mirror, mm, &device_mirror_ops);
+ * if (ret) {
+ * // Cleanup on error
+ * return ret;
+ * }
+ *
+ * // Other device driver specific initialization
+ * ...
+ * }
+ *
+ * Once an hmm_mirror is registered for an address space, the device driver
+ * will get callbacks through sync_cpu_device_pagetables() operation (see
+ * hmm_mirror_ops struct).
+ *
+ * Device driver must not free the struct containing the hmm_mirror struct
+ * before calling hmm_mirror_unregister(). The expected usage is to do that when
+ * the device driver is unbinding from an address space.
+ *
+ *
+ * void device_unbind_address_space(struct device_address_space *das)
+ * {
+ * // Device driver specific cleanup
+ * ...
+ *
+ * hmm_mirror_unregister(&das->mirror);
+ *
+ * // Other device driver specific cleanup, and now das can be freed
+ * ...
+ * }
+ */
+
+struct hmm_mirror;
+
+/*
+ * enum hmm_update_type - type of update
+ * @HMM_UPDATE_INVALIDATE: invalidate range (no indication as to why)
+ */
+enum hmm_update_type {
+ HMM_UPDATE_INVALIDATE,
+};
+
+/*
+ * struct hmm_mirror_ops - HMM mirror device operations callback
+ *
+ * @update: callback to update range on a device
+ */
+struct hmm_mirror_ops {
+ /* sync_cpu_device_pagetables() - synchronize page tables
+ *
+ * @mirror: pointer to struct hmm_mirror
+ * @update_type: type of update that occurred to the CPU page table
+ * @start: virtual start address of the range to update
+ * @end: virtual end address of the range to update
+ *
+ * This callback ultimately originates from mmu_notifiers when the CPU
+ * page table is updated. The device driver must update its page table
+ * in response to this callback. The update argument tells what action
+ * to perform.
+ *
+ * The device driver must not return from this callback until the device
+ * page tables are completely updated (TLBs flushed, etc); this is a
+ * synchronous call.
+ */
+ void (*sync_cpu_device_pagetables)(struct hmm_mirror *mirror,
+ enum hmm_update_type update_type,
+ unsigned long start,
+ unsigned long end);
+};
+
+/*
+ * struct hmm_mirror - mirror struct for a device driver
+ *
+ * @hmm: pointer to struct hmm (which is unique per mm_struct)
+ * @ops: device driver callback for HMM mirror operations
+ * @list: for list of mirrors of a given mm
+ *
+ * Each address space (mm_struct) being mirrored by a device must register one
+ * instance of an hmm_mirror struct with HMM. HMM will track the list of all
+ * mirrors for each mm_struct.
+ */
+struct hmm_mirror {
+ struct hmm *hmm;
+ const struct hmm_mirror_ops *ops;
+ struct list_head list;
+};
+
+int hmm_mirror_register(struct hmm_mirror *mirror, struct mm_struct *mm);
+void hmm_mirror_unregister(struct hmm_mirror *mirror);
+#endif /* IS_ENABLED(CONFIG_HMM_MIRROR) */
+
+
/* Below are for HMM internal use only! Not to be used by device driver! */
void hmm_mm_destroy(struct mm_struct *mm);
diff --git a/mm/Kconfig b/mm/Kconfig
index 90025b3..3c1ffb1 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -302,6 +302,18 @@ config ARCH_HAS_HMM
config HMM
bool
+config HMM_MIRROR
+ bool "HMM mirror CPU page table into a device page table"
+ depends on ARCH_HAS_HMM
+ select MMU_NOTIFIER
+ select HMM
+ help
+ Select HMM_MIRROR if you want to mirror range of the CPU page table of a
+ process into a device page table. Here, mirror means "keep synchronized".
+ Prerequisites: the device must provide the ability to write-protect its
+ page tables (at PAGE_SIZE granularity), and must be able to recover from
+ the resulting potential page faults.
+
config PHYS_ADDR_T_64BIT
def_bool 64BIT || ARCH_PHYS_ADDR_T_64BIT
diff --git a/mm/hmm.c b/mm/hmm.c
index acadb49..7ed4b4c 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -21,14 +21,26 @@
#include <linux/hmm.h>
#include <linux/slab.h>
#include <linux/sched.h>
+#include <linux/mmu_notifier.h>
+
+static const struct mmu_notifier_ops hmm_mmu_notifier_ops;
+
/*
* struct hmm - HMM per mm struct
*
* @mm: mm struct this HMM struct is bound to
+ * @sequence: we track updates to the CPU page table with a sequence number
+ * @mirrors: list of mirrors for this mm
+ * @mmu_notifier: mmu notifier to track updates to CPU page table
+ * @mirrors_sem: read/write semaphore protecting the mirrors list
*/
struct hmm {
struct mm_struct *mm;
+ atomic_t sequence;
+ struct list_head mirrors;
+ struct mmu_notifier mmu_notifier;
+ struct rw_semaphore mirrors_sem;
};
/*
@@ -41,27 +53,48 @@ struct hmm {
*/
static struct hmm *hmm_register(struct mm_struct *mm)
{
- if (!mm->hmm) {
- struct hmm *hmm = NULL;
-
- hmm = kmalloc(sizeof(*hmm), GFP_KERNEL);
- if (!hmm)
- return NULL;
- hmm->mm = mm;
-
- spin_lock(&mm->page_table_lock);
- if (!mm->hmm)
- mm->hmm = hmm;
- else
- kfree(hmm);
- spin_unlock(&mm->page_table_lock);
- }
+ struct hmm *hmm = READ_ONCE(mm->hmm);
+ bool cleanup = false;
/*
* The hmm struct can only be freed once the mm_struct goes away,
* hence we should always have pre-allocated an new hmm struct
* above.
*/
+ if (hmm)
+ return hmm;
+
+ hmm = kmalloc(sizeof(*hmm), GFP_KERNEL);
+ if (!hmm)
+ return NULL;
+ INIT_LIST_HEAD(&hmm->mirrors);
+ init_rwsem(&hmm->mirrors_sem);
+ atomic_set(&hmm->sequence, 0);
+ hmm->mmu_notifier.ops = NULL;
+ hmm->mm = mm;
+
+ /*
+ * We should only get here if hold the mmap_sem in write mode ie on
+ * registration of first mirror through hmm_mirror_register()
+ */
+ hmm->mmu_notifier.ops = &hmm_mmu_notifier_ops;
+ if (__mmu_notifier_register(&hmm->mmu_notifier, mm)) {
+ kfree(hmm);
+ return NULL;
+ }
+
+ spin_lock(&mm->page_table_lock);
+ if (!mm->hmm)
+ mm->hmm = hmm;
+ else
+ cleanup = true;
+ spin_unlock(&mm->page_table_lock);
+
+ if (cleanup) {
+ mmu_notifier_unregister(&hmm->mmu_notifier, mm);
+ kfree(hmm);
+ }
+
return mm->hmm;
}
@@ -69,3 +102,110 @@ void hmm_mm_destroy(struct mm_struct *mm)
{
kfree(mm->hmm);
}
+
+
+#if IS_ENABLED(CONFIG_HMM_MIRROR)
+static void hmm_invalidate_range(struct hmm *hmm,
+ enum hmm_update_type action,
+ unsigned long start,
+ unsigned long end)
+{
+ struct hmm_mirror *mirror;
+
+ down_read(&hmm->mirrors_sem);
+ list_for_each_entry(mirror, &hmm->mirrors, list)
+ mirror->ops->sync_cpu_device_pagetables(mirror, action,
+ start, end);
+ up_read(&hmm->mirrors_sem);
+}
+
+static void hmm_invalidate_page(struct mmu_notifier *mn,
+ struct mm_struct *mm,
+ unsigned long addr)
+{
+ unsigned long start = addr & PAGE_MASK;
+ unsigned long end = start + PAGE_SIZE;
+ struct hmm *hmm = mm->hmm;
+
+ VM_BUG_ON(!hmm);
+
+ atomic_inc(&hmm->sequence);
+ hmm_invalidate_range(mm->hmm, HMM_UPDATE_INVALIDATE, start, end);
+}
+
+static void hmm_invalidate_range_start(struct mmu_notifier *mn,
+ struct mm_struct *mm,
+ unsigned long start,
+ unsigned long end)
+{
+ struct hmm *hmm = mm->hmm;
+
+ VM_BUG_ON(!hmm);
+
+ atomic_inc(&hmm->sequence);
+}
+
+static void hmm_invalidate_range_end(struct mmu_notifier *mn,
+ struct mm_struct *mm,
+ unsigned long start,
+ unsigned long end)
+{
+ struct hmm *hmm = mm->hmm;
+
+ VM_BUG_ON(!hmm);
+
+ hmm_invalidate_range(mm->hmm, HMM_UPDATE_INVALIDATE, start, end);
+}
+
+static const struct mmu_notifier_ops hmm_mmu_notifier_ops = {
+ .invalidate_page = hmm_invalidate_page,
+ .invalidate_range_start = hmm_invalidate_range_start,
+ .invalidate_range_end = hmm_invalidate_range_end,
+};
+
+/*
+ * hmm_mirror_register() - register a mirror against an mm
+ *
+ * @mirror: new mirror struct to register
+ * @mm: mm to register against
+ *
+ * To start mirroring a process address space, the device driver must register
+ * an HMM mirror struct.
+ *
+ * THE mm->mmap_sem MUST BE HELD IN WRITE MODE !
+ */
+int hmm_mirror_register(struct hmm_mirror *mirror, struct mm_struct *mm)
+{
+ /* Sanity check */
+ if (!mm || !mirror || !mirror->ops)
+ return -EINVAL;
+
+ mirror->hmm = hmm_register(mm);
+ if (!mirror->hmm)
+ return -ENOMEM;
+
+ down_write(&mirror->hmm->mirrors_sem);
+ list_add(&mirror->list, &mirror->hmm->mirrors);
+ up_write(&mirror->hmm->mirrors_sem);
+
+ return 0;
+}
+EXPORT_SYMBOL(hmm_mirror_register);
+
+/*
+ * hmm_mirror_unregister() - unregister a mirror
+ *
+ * @mirror: new mirror struct to register
+ *
+ * Stop mirroring a process address space, and cleanup.
+ */
+void hmm_mirror_unregister(struct hmm_mirror *mirror)
+{
+ struct hmm *hmm = mirror->hmm;
+
+ down_write(&hmm->mirrors_sem);
+ list_del(&mirror->list);
+ up_write(&hmm->mirrors_sem);
+}
+EXPORT_SYMBOL(hmm_mirror_unregister);
+#endif /* IS_ENABLED(CONFIG_HMM_MIRROR) */
--
2.9.3
[toc] | [prev] | [next] | [standalone]
| From | Jérôme Glisse <jglisse@redhat.com> |
|---|---|
| Date | 2017-04-22 07:00 +0200 |
| Subject | [HMM 14/15] mm/hmm/devmem: dummy HMM device for ZONE_DEVICE memory v3 |
| Message-ID | <tyVjj-6NC-1@gated-at.bofh.it> |
| In reply to | #1628767 |
This introduce a dummy HMM device class so device driver can use it to
create hmm_device for the sole purpose of registering device memory.
It is useful to device driver that want to manage multiple physical
device memory under same struct device umbrella.
Changed since v2:
- use device_initcall() and drop everything that is module specific
Changed since v1:
- Improve commit message
- Add drvdata parameter to set on struct device
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
Signed-off-by: Evgeny Baskakov <ebaskakov@nvidia.com>
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
Signed-off-by: Mark Hairgrove <mhairgrove@nvidia.com>
Signed-off-by: Sherry Cheung <SCheung@nvidia.com>
Signed-off-by: Subhash Gutti <sgutti@nvidia.com>
---
include/linux/hmm.h | 22 +++++++++++++-
mm/hmm.c | 88 +++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 109 insertions(+), 1 deletion(-)
diff --git a/include/linux/hmm.h b/include/linux/hmm.h
index 50a1115..374e5fd 100644
--- a/include/linux/hmm.h
+++ b/include/linux/hmm.h
@@ -72,11 +72,11 @@
#if IS_ENABLED(CONFIG_HMM)
+#include <linux/device.h>
#include <linux/migrate.h>
#include <linux/memremap.h>
#include <linux/completion.h>
-
struct hmm;
/*
@@ -433,6 +433,26 @@ static inline unsigned long hmm_devmem_page_get_drvdata(struct page *page)
return drvdata[1];
}
+
+
+/*
+ * struct hmm_device - fake device to hang device memory onto
+ *
+ * @device: device struct
+ * @minor: device minor number
+ */
+struct hmm_device {
+ struct device device;
+ unsigned int minor;
+};
+
+/*
+ * A device driver that wants to handle multiple devices memory through a
+ * single fake device can use hmm_device to do so. This is purely a helper and
+ * it is not strictly needed, in order to make use of any HMM functionality.
+ */
+struct hmm_device *hmm_device_new(void *drvdata);
+void hmm_device_put(struct hmm_device *hmm_device);
#endif /* IS_ENABLED(CONFIG_HMM_DEVMEM) */
diff --git a/mm/hmm.c b/mm/hmm.c
index 5d882e6..8b6b4c6 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -19,6 +19,7 @@
*/
#include <linux/mm.h>
#include <linux/hmm.h>
+#include <linux/init.h>
#include <linux/rmap.h>
#include <linux/swap.h>
#include <linux/slab.h>
@@ -1112,4 +1113,91 @@ int hmm_devmem_fault_range(struct hmm_devmem *devmem,
return 0;
}
EXPORT_SYMBOL(hmm_devmem_fault_range);
+
+/*
+ * A device driver that wants to handle multiple devices memory through a
+ * single fake device can use hmm_device to do so. This is purely a helper
+ * and it is not needed to make use of any HMM functionality.
+ */
+#define HMM_DEVICE_MAX 256
+
+static DECLARE_BITMAP(hmm_device_mask, HMM_DEVICE_MAX);
+static DEFINE_SPINLOCK(hmm_device_lock);
+static struct class *hmm_device_class;
+static dev_t hmm_device_devt;
+
+static void hmm_device_release(struct device *device)
+{
+ struct hmm_device *hmm_device;
+
+ hmm_device = container_of(device, struct hmm_device, device);
+ spin_lock(&hmm_device_lock);
+ clear_bit(hmm_device->minor, hmm_device_mask);
+ spin_unlock(&hmm_device_lock);
+
+ kfree(hmm_device);
+}
+
+struct hmm_device *hmm_device_new(void *drvdata)
+{
+ struct hmm_device *hmm_device;
+ int ret;
+
+ hmm_device = kzalloc(sizeof(*hmm_device), GFP_KERNEL);
+ if (!hmm_device)
+ return ERR_PTR(-ENOMEM);
+
+ ret = alloc_chrdev_region(&hmm_device->device.devt, 0, 1, "hmm_device");
+ if (ret < 0) {
+ kfree(hmm_device);
+ return NULL;
+ }
+
+ spin_lock(&hmm_device_lock);
+ hmm_device->minor = find_first_zero_bit(hmm_device_mask, HMM_DEVICE_MAX);
+ if (hmm_device->minor >= HMM_DEVICE_MAX) {
+ spin_unlock(&hmm_device_lock);
+ kfree(hmm_device);
+ return NULL;
+ }
+ set_bit(hmm_device->minor, hmm_device_mask);
+ spin_unlock(&hmm_device_lock);
+
+ dev_set_name(&hmm_device->device, "hmm_device%d", hmm_device->minor);
+ hmm_device->device.devt = MKDEV(MAJOR(hmm_device_devt),
+ hmm_device->minor);
+ hmm_device->device.release = hmm_device_release;
+ dev_set_drvdata(&hmm_device->device, drvdata);
+ hmm_device->device.class = hmm_device_class;
+ device_initialize(&hmm_device->device);
+
+ return hmm_device;
+}
+EXPORT_SYMBOL(hmm_device_new);
+
+void hmm_device_put(struct hmm_device *hmm_device)
+{
+ put_device(&hmm_device->device);
+}
+EXPORT_SYMBOL(hmm_device_put);
+
+static int __init hmm_init(void)
+{
+ int ret;
+
+ ret = alloc_chrdev_region(&hmm_device_devt, 0,
+ HMM_DEVICE_MAX,
+ "hmm_device");
+ if (ret)
+ return ret;
+
+ hmm_device_class = class_create(THIS_MODULE, "hmm_device");
+ if (IS_ERR(hmm_device_class)) {
+ unregister_chrdev_region(hmm_device_devt, HMM_DEVICE_MAX);
+ return PTR_ERR(hmm_device_class);
+ }
+ return 0;
+}
+
+device_initcall(hmm_init);
#endif /* IS_ENABLED(CONFIG_HMM_DEVMEM) */
--
2.9.3
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web