Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1231150 > unrolled thread
| Started by | Dan Williams <dan.j.williams@intel.com> |
|---|---|
| First post | 2015-09-23 06:50 +0200 |
| Last post | 2015-09-23 07:00 +0200 |
| Articles | 3 — 1 participant |
Back to article view | Back to linux.kernel
[PATCH 00/15] get_user_pages() for dax mappings Dan Williams <dan.j.williams@intel.com> - 2015-09-23 06:50 +0200
[PATCH 10/15] block, dax: fix lifetime of in-kernel dax mappings Dan Williams <dan.j.williams@intel.com> - 2015-09-23 06:50 +0200
[PATCH 02/15] hugetlb: fix compile error on tile Dan Williams <dan.j.williams@intel.com> - 2015-09-23 07:00 +0200
| From | Dan Williams <dan.j.williams@intel.com> |
|---|---|
| Date | 2015-09-23 06:50 +0200 |
| Subject | [PATCH 00/15] get_user_pages() for dax mappings |
| Message-ID | <qbKae-20h-7@gated-at.bofh.it> |
To date, we have implemented two I/O usage models for persistent memory,
PMEM (a persistent "ram disk") and DAX (mmap persistent memory into
userspace). This series adds a third, DAX-GUP, that allows DAX mappings
to be the target of direct-i/o. It allows userspace to coordinate
DMA/RDMA from/to persitent memory.
The implementation leverages the ZONE_DEVICE mm-zone that went into
4.3-rc1 to flag pages that are owned and dynamically mapped by a device
driver. The pmem driver, after mapping a persistent memory range into
the system memmap via devm_memremap_pages(), arranges for DAX to
distinguish pfn-only versus page-backed pmem-pfns via flags in the new
__pfn_t type. The DAX code, upon seeing a PFN_DEV+PFN_MAP flagged pfn,
flags the resulting pte(s) inserted into the process page tables with a
new _PAGE_DEVMAP flag. Later, when get_user_pages() is walking ptes it
keys off _PAGE_DEVMAP to pin the device hosting the page range active.
Finally, get_page() and put_page() are modified to take references
against the device driver established page mapping.
Next step, more testing specifically DAX-get_user_pages() vs truncate.
Patches 1 - 3 are general compilation fixups from 0day-kbuild reports
while developing this series.
Patches 4 - 7 are minor cleanups and reworks of the devm_memremap_* api.
Patches 8 - 10 add a reference counter for pinning the pmem driver
active while it is in use. It turns out, prior to these changes, you
can reliably crash the kernel on shutdown if the pmem device is unbound
while hosting a mounted filesystem.
Patches 11 - 15 use __pfn_t and the _PAGE_DEVMAP flag to implement the
dax-gup path.
This series is built on 4.3-rc2 plus the __dax_pmd_fault fix from Ross:
https://patchwork.kernel.org/patch/7244961/
---
Dan Williams (15):
avr32: convert to asm-generic/memory_model.h
hugetlb: fix compile error on tile
frv: fix compiler warning from definition of __pmd()
x86, mm: quiet arch_add_memory()
pmem: kill memremap_pmem()
devm_memunmap: use devres_release()
devm_memremap: convert to return ERR_PTR
block, dax, pmem: reference counting infrastructure
block, pmem: fix null pointer de-reference on shutdown, check for queue death
block, dax: fix lifetime of in-kernel dax mappings
mm, dax, pmem: introduce __pfn_t
mm, dax, gpu: convert vm_insert_mixed to __pfn_t, introduce _PAGE_DEVMAP
mm, dax: convert vmf_insert_pfn_pmd() to __pfn_t
mm, dax, pmem: introduce {get|put}_dev_pagemap() for dax-gup
mm, x86: get_user_pages() for dax mappings
arch/alpha/include/asm/pgtable.h | 1
arch/avr32/include/asm/page.h | 8 +
arch/frv/include/asm/page.h | 2
arch/ia64/include/asm/pgtable.h | 1
arch/m68k/include/asm/page_no.h | 1
arch/parisc/include/asm/pgtable.h | 1
arch/powerpc/include/asm/pgtable.h | 1
arch/powerpc/sysdev/axonram.c | 10 +
arch/sparc/include/asm/pgtable_64.h | 2
arch/tile/include/asm/pgtable.h | 1
arch/um/include/asm/pgtable-3level.h | 1
arch/x86/include/asm/pgtable.h | 24 ++++
arch/x86/include/asm/pgtable_types.h | 7 +
arch/x86/mm/gup.c | 56 ++++++++
arch/x86/mm/init.c | 4 -
arch/x86/mm/init_64.c | 4 -
arch/x86/mm/pat.c | 4 -
block/blk-core.c | 86 ++++++++++++-
block/blk-mq-sysfs.c | 2
block/blk-mq.c | 48 ++-----
block/blk-sysfs.c | 9 +
block/blk.h | 3
drivers/block/brd.c | 6 -
drivers/gpu/drm/exynos/exynos_drm_gem.c | 3
drivers/gpu/drm/gma500/framebuffer.c | 2
drivers/gpu/drm/msm/msm_gem.c | 3
drivers/gpu/drm/omapdrm/omap_gem.c | 6 +
drivers/gpu/drm/ttm/ttm_bo_vm.c | 3
drivers/nvdimm/pmem.c | 57 +++++---
drivers/s390/block/dcssblk.c | 12 +-
fs/block_dev.c | 2
fs/dax.c | 140 +++++++++++++-------
include/asm-generic/pgtable.h | 6 +
include/linux/blkdev.h | 24 +++-
include/linux/huge_mm.h | 2
include/linux/hugetlb.h | 1
include/linux/io.h | 17 --
include/linux/mm.h | 212 +++++++++++++++++++++++++++++--
include/linux/mm_types.h | 6 +
include/linux/pfn.h | 9 +
include/linux/pmem.h | 26 ----
kernel/memremap.c | 78 +++++++++++
mm/gup.c | 11 +-
mm/huge_memory.c | 10 +
mm/hugetlb.c | 18 ++-
mm/memory.c | 17 +-
mm/swap.c | 15 ++
47 files changed, 729 insertions(+), 233 deletions(-)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [next] | [standalone]
| From | Dan Williams <dan.j.williams@intel.com> |
|---|---|
| Date | 2015-09-23 06:50 +0200 |
| Subject | [PATCH 10/15] block, dax: fix lifetime of in-kernel dax mappings |
| Message-ID | <qbKaf-20h-33@gated-at.bofh.it> |
| In reply to | #1231150 |
The DAX implementation needs to protect new calls to ->direct_access()
and usage of its return value against unbind of the underlying block
device. Use blk_dax_{get|put}() to either prevent blk_cleanup_queue()
from proceeding, or fail the dax_map_bh() if the request_queue is being
torn down.
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Boaz Harrosh <boaz@plexistor.com>
Cc: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
fs/dax.c | 131 ++++++++++++++++++++++++++++++++++++++++----------------------
1 file changed, 84 insertions(+), 47 deletions(-)
diff --git a/fs/dax.c b/fs/dax.c
index bcfb14bfc1e4..358eea39e982 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -63,12 +63,43 @@ int dax_clear_blocks(struct inode *inode, sector_t block, long size)
}
EXPORT_SYMBOL_GPL(dax_clear_blocks);
-static long dax_get_addr(struct buffer_head *bh, void __pmem **addr,
- unsigned blkbits)
+static void __pmem *__dax_map_bh(const struct buffer_head *bh, unsigned blkbits,
+ unsigned long *pfn, long *len)
{
- unsigned long pfn;
+ long rc;
+ void __pmem *addr;
+ struct block_device *bdev = bh->b_bdev;
+ struct request_queue *q = bdev->bd_queue;
sector_t sector = bh->b_blocknr << (blkbits - 9);
- return bdev_direct_access(bh->b_bdev, sector, addr, &pfn, bh->b_size);
+
+ rc = blk_dax_get(q);
+ if (rc < 0)
+ return (void __pmem *) ERR_PTR(rc);
+ rc = bdev_direct_access(bdev, sector, &addr, pfn, bh->b_size);
+ if (len)
+ *len = rc;
+ if (rc < 0) {
+ blk_dax_put(q);
+ return (void __pmem *) ERR_PTR(rc);
+ }
+ return addr;
+}
+
+static void __pmem *dax_map_bh(const struct buffer_head *bh, unsigned blkbits)
+{
+ unsigned long pfn;
+
+ return __dax_map_bh(bh, blkbits, &pfn, NULL);
+}
+
+static void dax_unmap_bh(const struct buffer_head *bh, void __pmem *addr)
+{
+ struct block_device *bdev = bh->b_bdev;
+ struct request_queue *q = bdev->bd_queue;
+
+ if (IS_ERR(addr))
+ return;
+ blk_dax_put(q);
}
/* the clear_pmem() calls are ordered by a wmb_pmem() in the caller */
@@ -104,15 +135,16 @@ static ssize_t dax_io(struct inode *inode, struct iov_iter *iter,
loff_t start, loff_t end, get_block_t get_block,
struct buffer_head *bh)
{
- ssize_t retval = 0;
- loff_t pos = start;
- loff_t max = start;
- loff_t bh_max = start;
- void __pmem *addr;
+ loff_t pos = start, max = start, bh_max = start;
+ int rw = iov_iter_rw(iter), rc;
+ long map_len = 0;
+ unsigned long pfn;
+ void __pmem *addr = NULL;
+ void __pmem *kmap = (void __pmem *) ERR_PTR(-EIO);
bool hole = false;
bool need_wmb = false;
- if (iov_iter_rw(iter) != WRITE)
+ if (rw == READ)
end = min(end, i_size_read(inode));
while (pos < end) {
@@ -127,9 +159,8 @@ static ssize_t dax_io(struct inode *inode, struct iov_iter *iter,
if (pos == bh_max) {
bh->b_size = PAGE_ALIGN(end - pos);
bh->b_state = 0;
- retval = get_block(inode, block, bh,
- iov_iter_rw(iter) == WRITE);
- if (retval)
+ rc = get_block(inode, block, bh, rw == WRITE);
+ if (rc)
break;
if (!buffer_size_valid(bh))
bh->b_size = 1 << blkbits;
@@ -141,21 +172,25 @@ static ssize_t dax_io(struct inode *inode, struct iov_iter *iter,
bh->b_size -= done;
}
- hole = iov_iter_rw(iter) != WRITE && !buffer_written(bh);
+ hole = rw == READ && !buffer_written(bh);
if (hole) {
addr = NULL;
size = bh->b_size - first;
} else {
- retval = dax_get_addr(bh, &addr, blkbits);
- if (retval < 0)
+ dax_unmap_bh(bh, kmap);
+ kmap = __dax_map_bh(bh, blkbits, &pfn, &map_len);
+ if (IS_ERR(kmap)) {
+ rc = PTR_ERR(kmap);
break;
+ }
+ addr = kmap;
if (buffer_unwritten(bh) || buffer_new(bh)) {
- dax_new_buf(addr, retval, first, pos,
- end);
+ dax_new_buf(addr, map_len, first, pos,
+ end);
need_wmb = true;
}
addr += first;
- size = retval - first;
+ size = map_len - first;
}
max = min(pos + size, end);
}
@@ -178,8 +213,9 @@ static ssize_t dax_io(struct inode *inode, struct iov_iter *iter,
if (need_wmb)
wmb_pmem();
+ dax_unmap_bh(bh, kmap);
- return (pos == start) ? retval : pos - start;
+ return (pos == start) ? rc : pos - start;
}
/**
@@ -274,21 +310,22 @@ static int copy_user_bh(struct page *to, struct buffer_head *bh,
void __pmem *vfrom;
void *vto;
- if (dax_get_addr(bh, &vfrom, blkbits) < 0)
- return -EIO;
+ vfrom = dax_map_bh(bh, blkbits);
+ if (IS_ERR(vfrom))
+ return PTR_ERR(vfrom);
vto = kmap_atomic(to);
copy_user_page(vto, (void __force *)vfrom, vaddr, to);
kunmap_atomic(vto);
+ dax_unmap_bh(bh, vfrom);
return 0;
}
static int dax_insert_mapping(struct inode *inode, struct buffer_head *bh,
struct vm_area_struct *vma, struct vm_fault *vmf)
{
- sector_t sector = bh->b_blocknr << (inode->i_blkbits - 9);
unsigned long vaddr = (unsigned long)vmf->virtual_address;
- void __pmem *addr;
unsigned long pfn;
+ void __pmem *addr;
pgoff_t size;
int error;
@@ -305,11 +342,9 @@ static int dax_insert_mapping(struct inode *inode, struct buffer_head *bh,
goto out;
}
- error = bdev_direct_access(bh->b_bdev, sector, &addr, &pfn, bh->b_size);
- if (error < 0)
- goto out;
- if (error < PAGE_SIZE) {
- error = -EIO;
+ addr = __dax_map_bh(bh, inode->i_blkbits, &pfn, NULL);
+ if (IS_ERR(addr)) {
+ error = PTR_ERR(addr);
goto out;
}
@@ -317,6 +352,7 @@ static int dax_insert_mapping(struct inode *inode, struct buffer_head *bh,
clear_pmem(addr, PAGE_SIZE);
wmb_pmem();
}
+ dax_unmap_bh(bh, addr);
error = vm_insert_mixed(vma, vaddr, pfn);
@@ -528,11 +564,8 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
unsigned blkbits = inode->i_blkbits;
unsigned long pmd_addr = address & PMD_MASK;
bool write = flags & FAULT_FLAG_WRITE;
- long length;
- void __pmem *kaddr;
pgoff_t size, pgoff;
- sector_t block, sector;
- unsigned long pfn;
+ sector_t block;
int result = 0;
/* Fall back to PTEs if we're going to COW */
@@ -557,8 +590,7 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
bh.b_size = PMD_SIZE;
i_mmap_lock_write(mapping);
- length = get_block(inode, block, &bh, write);
- if (length)
+ if (get_block(inode, block, &bh, write) != 0)
return VM_FAULT_SIGBUS;
/*
@@ -569,17 +601,17 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
if (!buffer_size_valid(&bh) || bh.b_size < PMD_SIZE)
goto fallback;
- sector = bh.b_blocknr << (blkbits - 9);
-
if (buffer_unwritten(&bh) || buffer_new(&bh)) {
int i;
+ long length;
+ unsigned long pfn;
+ void __pmem *kaddr = __dax_map_bh(&bh, blkbits, &pfn, &length);
- length = bdev_direct_access(bh.b_bdev, sector, &kaddr, &pfn,
- bh.b_size);
- if (length < 0) {
+ if (IS_ERR(kaddr)) {
result = VM_FAULT_SIGBUS;
goto out;
}
+
if ((length < PMD_SIZE) || (pfn & PG_PMD_COLOUR))
goto fallback;
@@ -589,6 +621,7 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
count_vm_event(PGMAJFAULT);
mem_cgroup_count_vm_event(vma->vm_mm, PGMAJFAULT);
result |= VM_FAULT_MAJOR;
+ dax_unmap_bh(&bh, kaddr);
}
/*
@@ -635,12 +668,15 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
result = VM_FAULT_NOPAGE;
spin_unlock(ptl);
} else {
- length = bdev_direct_access(bh.b_bdev, sector, &kaddr, &pfn,
- bh.b_size);
- if (length < 0) {
+ long length;
+ unsigned long pfn;
+ void __pmem *kaddr = __dax_map_bh(&bh, blkbits, &pfn, &length);
+
+ if (IS_ERR(kaddr)) {
result = VM_FAULT_SIGBUS;
goto out;
}
+ dax_unmap_bh(&bh, kaddr);
if ((length < PMD_SIZE) || (pfn & PG_PMD_COLOUR))
goto fallback;
@@ -746,12 +782,13 @@ int dax_zero_page_range(struct inode *inode, loff_t from, unsigned length,
if (err < 0)
return err;
if (buffer_written(&bh)) {
- void __pmem *addr;
- err = dax_get_addr(&bh, &addr, inode->i_blkbits);
- if (err < 0)
- return err;
+ void __pmem *addr = dax_map_bh(&bh, inode->i_blkbits);
+
+ if (IS_ERR(addr))
+ return PTR_ERR(addr);
clear_pmem(addr + offset, length);
wmb_pmem();
+ dax_unmap_bh(&bh, addr);
}
return 0;
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Dan Williams <dan.j.williams@intel.com> |
|---|---|
| Date | 2015-09-23 07:00 +0200 |
| Subject | [PATCH 02/15] hugetlb: fix compile error on tile |
| Message-ID | <qbKjU-2bt-1@gated-at.bofh.it> |
| In reply to | #1231150 |
Inlude asm/pgtable.h to get the definition for pud_t to fix: include/linux/hugetlb.h:203:29: error: unknown type name 'pud_t' Signed-off-by: Dan Williams <dan.j.williams@intel.com> --- include/linux/hugetlb.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 5e35379f58a5..ad5539cf52bf 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -8,6 +8,7 @@ #include <linux/cgroup.h> #include <linux/list.h> #include <linux/kref.h> +#include <asm/pgtable.h> struct ctl_table; struct user_struct; -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web