Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1662522 > unrolled thread

[RFC v4 00/20] Speculative page faults

Back to article view | Back to linux.kernel

Contents

  [RFC v4 00/20] Speculative page faults Laurent Dufour <ldufour@linux.vnet.ibm.com> - 2017-06-09 16:30 +0200
    [RFC v4 15/20] mm/spf: protect mremap() against speculative pf Laurent Dufour <ldufour@linux.vnet.ibm.com> - 2017-06-09 16:30 +0200
    [RFC v4 06/20] mm: Provide speculative fault infrastructure Laurent Dufour <ldufour@linux.vnet.ibm.com> - 2017-06-09 16:30 +0200
    Re: [RFC v4 00/20] Speculative page faults Michal Hocko <mhocko@kernel.org> - 2017-06-09 17:10 +0200
      Re: [RFC v4 00/20] Speculative page faults Laurent Dufour <ldufour@linux.vnet.ibm.com> - 2017-06-09 17:30 +0200
        Re: [RFC v4 00/20] Speculative page faults Michal Hocko <mhocko@kernel.org> - 2017-06-09 18:40 +0200
          Re: [RFC v4 00/20] Speculative page faults Tim Chen <tim.c.chen@linux.intel.com> - 2017-06-09 19:10 +0200
            Re: [RFC v4 00/20] Speculative page faults Laurent Dufour <ldufour@linux.vnet.ibm.com> - 2017-06-13 12:30 +0200
          Re: [RFC v4 00/20] Speculative page faults Laurent Dufour <ldufour@linux.vnet.ibm.com> - 2017-06-13 12:00 +0200
    Re: [RFC v4 00/20] Speculative page faults "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-06-09 21:00 +0200
    Re: [RFC v4 00/20] Speculative page faults Jan Kara <jack@suse.cz> - 2017-06-12 12:30 +0200
      Re: [RFC v4 00/20] Speculative page faults Laurent Dufour <ldufour@linux.vnet.ibm.com> - 2017-06-13 12:30 +0200

#1662522 — [RFC v4 00/20] Speculative page faults

This is a port on kernel 4.12 of the work done by Peter Zijlstra to
handle page fault without holding the mm semaphore.

http://linux-kernel.2935.n7.nabble.com/RFC-PATCH-0-6-Another-go-at-speculative-page-faults-tt965642.html#none

Compared to the Peter initial work, this series introduce a try spin
lock when dealing with speculative page fault. This is required to
avoid dead lock when handling a page fault while a TLB invalidate is
requested by an other CPU holding the PTE. Another change due to a
lock dependency issue with mapping->i_mmap_rwsem.

This series also protect changes to VMA's data which are read or
change by the page fault handler. The protections is done through the
VMA's sequence number.

This series is functional on x86 and PowerPC.

It's building on top of v4.12-rc4 and relies on the change done by
Paul McKenney to the SRCU code allowing better performance by
maintaining per-CPU callback lists:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=da915ad5cf25b5f5d358dd3670c3378d8ae8c03e

Tests have been made using a large commercial in-memory database on a
PowerPC system with 752 CPUs. The results are very encouraging since
the loading of the 2TB database was faster by 20% with the speculative
page fault.

Since tests are encouraging and running test suite didn't raise any
issue, I'd like this request for comment series to move to a patch
series soon. So please feel free to comment.

Changes since V3:
 - support for the 5-level paging.
 - abort speculative path before entering userfault code
 - support for PowerPC architecture
 - reorder the patch to fix build test errors.

Laurent Dufour (14):
  mm: Introduce pte_spinlock
  mm/spf: Try spin lock in speculative path
  mm/spf: Fix fe.sequence init in __handle_mm_fault()
  mm/spf: don't set fault entry's fields if locking failed
  mm/spf; fix lock dependency against mapping->i_mmap_rwsem
  mm/spf: Protect changes to vm_flags
  mm/spf Protect vm_policy's changes against speculative pf
  mm/spf: Add check on the VMA's flags
  mm/spf: protect madvise vs speculative pf
  mm/spf: protect mremap() against speculative pf
  mm/spf: Don't call user fault callback in the speculative path
  x86/mm: Update the handle_speculative_fault's path
  powerpc/mm: Add speculative page fault
  mm/spf: Clear FAULT_FLAG_KILLABLE in the speculative path

Peter Zijlstra (6):
  mm: Dont assume page-table invariance during faults
  mm: Prepare for FAULT_FLAG_SPECULATIVE
  mm: VMA sequence count
  mm: RCU free VMAs
  mm: Provide speculative fault infrastructure
  x86/mm: Add speculative pagefault handling

 arch/powerpc/mm/fault.c  |  25 +++-
 arch/x86/mm/fault.c      |  14 +++
 fs/proc/task_mmu.c       |   2 +
 include/linux/mm.h       |   4 +
 include/linux/mm_types.h |   3 +
 kernel/fork.c            |   1 +
 mm/init-mm.c             |   1 +
 mm/internal.h            |  20 ++++
 mm/madvise.c             |   4 +
 mm/memory.c              | 291 +++++++++++++++++++++++++++++++++++++++--------
 mm/mempolicy.c           |  10 +-
 mm/mlock.c               |   9 +-
 mm/mmap.c                | 123 +++++++++++++++-----
 mm/mprotect.c            |   2 +
 mm/mremap.c              |   7 ++
 15 files changed, 435 insertions(+), 81 deletions(-)

-- 
2.7.4

[toc] | [next] | [standalone]

#1662524 — [RFC v4 15/20] mm/spf: protect mremap() against speculative pf

mremap() is modifying the VMA layout and thus must be protected against
the speculative page fault handler.

Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
---
 mm/mremap.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/mm/mremap.c b/mm/mremap.c
index cd8a1b199ef9..9c7f69c9e80f 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -300,6 +300,10 @@ static unsigned long move_vma(struct vm_area_struct *vma,
 	if (!new_vma)
 		return -ENOMEM;
 
+	write_seqcount_begin(&vma->vm_sequence);
+	write_seqcount_begin_nested(&new_vma->vm_sequence,
+				    SINGLE_DEPTH_NESTING);
+
 	moved_len = move_page_tables(vma, old_addr, new_vma, new_addr, old_len,
 				     need_rmap_locks);
 	if (moved_len < old_len) {
@@ -316,6 +320,7 @@ static unsigned long move_vma(struct vm_area_struct *vma,
 		 */
 		move_page_tables(new_vma, new_addr, vma, old_addr, moved_len,
 				 true);
+		write_seqcount_end(&vma->vm_sequence);
 		vma = new_vma;
 		old_len = new_len;
 		old_addr = new_addr;
@@ -324,7 +329,9 @@ static unsigned long move_vma(struct vm_area_struct *vma,
 		mremap_userfaultfd_prep(new_vma, uf);
 		arch_remap(mm, old_addr, old_addr + old_len,
 			   new_addr, new_addr + new_len);
+		write_seqcount_end(&vma->vm_sequence);
 	}
+	write_seqcount_end(&new_vma->vm_sequence);
 
 	/* Conceal VM_ACCOUNT so old reservation is not undone */
 	if (vm_flags & VM_ACCOUNT) {
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1662525 — [RFC v4 06/20] mm: Provide speculative fault infrastructure

From: Peter Zijlstra <peterz@infradead.org>

Provide infrastructure to do a speculative fault (not holding
mmap_sem).

The not holding of mmap_sem means we can race against VMA
change/removal and page-table destruction. We use the SRCU VMA freeing
to keep the VMA around. We use the VMA seqcount to detect change
(including umapping / page-table deletion) and we use gup_fast() style
page-table walking to deal with page-table races.

Once we've obtained the page and are ready to update the PTE, we
validate if the state we started the fault with is still valid, if
not, we'll fail the fault with VM_FAULT_RETRY, otherwise we update the
PTE and we're done.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
[Fix newly introduced pte_spinlock() for speculative page fault]
[Rename vma_is_dead() to vma_has_changed()]
[Call p4d_alloc() as it is safe since pgd is valid]
[Call pud_alloc() as it is safe since p4d is valid]
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
---
 include/linux/mm.h |   3 ++
 mm/memory.c        | 148 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 148 insertions(+), 3 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 6b7ec2a76953..671541e00d26 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -315,6 +315,7 @@ struct vm_fault {
 	gfp_t gfp_mask;			/* gfp mask to be used for allocations */
 	pgoff_t pgoff;			/* Logical page offset based on vma */
 	unsigned long address;		/* Faulting virtual address */
+	unsigned int sequence;
 	pmd_t *pmd;			/* Pointer to pmd entry matching
 					 * the 'address' */
 	pud_t *pud;			/* Pointer to pud entry matching
@@ -1286,6 +1287,8 @@ int invalidate_inode_page(struct page *page);
 #ifdef CONFIG_MMU
 extern int handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
 		unsigned int flags);
+extern int handle_speculative_fault(struct mm_struct *mm,
+				    unsigned long address, unsigned int flags);
 extern int fixup_user_fault(struct task_struct *tsk, struct mm_struct *mm,
 			    unsigned long address, unsigned int fault_flags,
 			    bool *unlocked);
diff --git a/mm/memory.c b/mm/memory.c
index 8f0468d099fc..8c43895e9310 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2244,15 +2244,66 @@ static inline void wp_page_reuse(struct vm_fault *vmf)
 
 static bool pte_spinlock(struct vm_fault *vmf)
 {
+	bool ret = false;
+
+	/* Check if vma is still valid */
+	if (!(vmf->flags & FAULT_FLAG_SPECULATIVE)) {
+		vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
+		spin_lock(vmf->ptl);
+		return true;
+	}
+
+	local_irq_disable();
+	if (vma_has_changed(vmf->vma, vmf->sequence))
+		goto out;
+
 	vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
 	spin_lock(vmf->ptl);
-	return true;
+
+	if (vma_has_changed(vmf->vma, vmf->sequence)) {
+		spin_unlock(vmf->ptl);
+		goto out;
+	}
+
+	ret = true;
+out:
+	local_irq_enable();
+	return ret;
 }
 
 static bool pte_map_lock(struct vm_fault *vmf)
 {
-	vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl);
-	return true;
+	bool ret = false;
+
+	if (!(vmf->flags & FAULT_FLAG_SPECULATIVE)) {
+		vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
+					       vmf->address, &vmf->ptl);
+		return true;
+	}
+
+	/*
+	 * The first vma_has_changed() guarantees the page-tables are still
+	 * valid, having IRQs disabled ensures they stay around, hence the
+	 * second vma_has_changed() to make sure they are still valid once
+	 * we've got the lock. After that a concurrent zap_pte_range() will
+	 * block on the PTL and thus we're safe.
+	 */
+	local_irq_disable();
+	if (vma_has_changed(vmf->vma, vmf->sequence))
+		goto out;
+
+	vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
+				       vmf->address, &vmf->ptl);
+
+	if (vma_has_changed(vmf->vma, vmf->sequence)) {
+		pte_unmap_unlock(vmf->pte, vmf->ptl);
+		goto out;
+	}
+
+	ret = true;
+out:
+	local_irq_enable();
+	return ret;
 }
 
 /*
@@ -2684,6 +2735,7 @@ int do_swap_page(struct vm_fault *vmf)
 	entry = pte_to_swp_entry(vmf->orig_pte);
 	if (unlikely(non_swap_entry(entry))) {
 		if (is_migration_entry(entry)) {
+			/* XXX fe->pmd might be dead */
 			migration_entry_wait(vma->vm_mm, vmf->pmd,
 					     vmf->address);
 		} else if (is_hwpoison_entry(entry)) {
@@ -3868,6 +3920,96 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
 	return handle_pte_fault(&vmf);
 }
 
+int handle_speculative_fault(struct mm_struct *mm, unsigned long address,
+			     unsigned int flags)
+{
+	struct vm_fault vmf = {
+		.address = address,
+		.flags = flags | FAULT_FLAG_SPECULATIVE,
+	};
+	pgd_t *pgd;
+	p4d_t *p4d;
+	pud_t *pud;
+	pmd_t *pmd;
+	int dead, seq, idx, ret = VM_FAULT_RETRY;
+	struct vm_area_struct *vma;
+
+	idx = srcu_read_lock(&vma_srcu);
+	vma = find_vma_srcu(mm, address);
+	if (!vma)
+		goto unlock;
+
+	/*
+	 * Validate the VMA found by the lockless lookup.
+	 */
+	dead = RB_EMPTY_NODE(&vma->vm_rb);
+	seq = raw_read_seqcount(&vma->vm_sequence); /* rmb <-> seqlock,vma_rb_erase() */
+	if ((seq & 1) || dead) /* XXX wait for !&1 instead? */
+		goto unlock;
+
+	if (address < vma->vm_start || vma->vm_end <= address)
+		goto unlock;
+
+	/*
+	 * We need to re-validate the VMA after checking the bounds, otherwise
+	 * we might have a false positive on the bounds.
+	 */
+	if (read_seqcount_retry(&vma->vm_sequence, seq))
+		goto unlock;
+
+	/*
+	 * Do a speculative lookup of the PTE entry.
+	 */
+	local_irq_disable();
+	pgd = pgd_offset(mm, address);
+	if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd)))
+		goto out_walk;
+
+	p4d = p4d_alloc(mm, pgd, address);
+	if (p4d_none(*p4d) || unlikely(p4d_bad(*p4d)))
+		goto out_walk;
+
+	pud = pud_alloc(mm, p4d, address);
+	if (pud_none(*pud) || unlikely(pud_bad(*pud)))
+		goto out_walk;
+
+	pmd = pmd_offset(pud, address);
+	if (pmd_none(*pmd) || unlikely(pmd_bad(*pmd)))
+		goto out_walk;
+
+	/*
+	 * The above does not allocate/instantiate page-tables because doing so
+	 * would lead to the possibility of instantiating page-tables after
+	 * free_pgtables() -- and consequently leaking them.
+	 *
+	 * The result is that we take at least one !speculative fault per PMD
+	 * in order to instantiate it.
+	 *
+	 * XXX try and fix that.. should be possible somehow.
+	 */
+
+	if (pmd_huge(*pmd)) /* XXX no huge support */
+		goto out_walk;
+
+	vmf.vma = vma;
+	vmf.pmd = pmd;
+	vmf.pgoff = linear_page_index(vma, address);
+	vmf.gfp_mask = __get_fault_gfp_mask(vma);
+	vmf.sequence = seq;
+
+	local_irq_enable();
+
+	ret = handle_pte_fault(&vmf);
+
+unlock:
+	srcu_read_unlock(&vma_srcu, idx);
+	return ret;
+
+out_walk:
+	local_irq_enable();
+	goto unlock;
+}
+
 /*
  * By the time we get here, we already hold the mm semaphore
  *
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1662546

On Fri 09-06-17 16:20:49, Laurent Dufour wrote:
> This is a port on kernel 4.12 of the work done by Peter Zijlstra to
> handle page fault without holding the mm semaphore.
> 
> http://linux-kernel.2935.n7.nabble.com/RFC-PATCH-0-6-Another-go-at-speculative-page-faults-tt965642.html#none
> 
> Compared to the Peter initial work, this series introduce a try spin
> lock when dealing with speculative page fault. This is required to
> avoid dead lock when handling a page fault while a TLB invalidate is
> requested by an other CPU holding the PTE. Another change due to a
> lock dependency issue with mapping->i_mmap_rwsem.
> 
> This series also protect changes to VMA's data which are read or
> change by the page fault handler. The protections is done through the
> VMA's sequence number.
> 
> This series is functional on x86 and PowerPC.
> 
> It's building on top of v4.12-rc4 and relies on the change done by
> Paul McKenney to the SRCU code allowing better performance by
> maintaining per-CPU callback lists:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=da915ad5cf25b5f5d358dd3670c3378d8ae8c03e
> 
> Tests have been made using a large commercial in-memory database on a
> PowerPC system with 752 CPUs. The results are very encouraging since
> the loading of the 2TB database was faster by 20% with the speculative
> page fault.
> 
> Since tests are encouraging and running test suite didn't raise any
> issue, I'd like this request for comment series to move to a patch
> series soon. So please feel free to comment.

What other testing have you done? Other benchmarks (some numbers)? What
about some standard worklaods like kbench? This is a pretty invasive
change so I would expect much more numbers.

It would also help to describe the highlevel design of the change here
in the cover letter. This would make the review of specifics much
easier.

Thanks!
-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1662554

On 09/06/2017 17:01, Michal Hocko wrote:
> On Fri 09-06-17 16:20:49, Laurent Dufour wrote:
>> This is a port on kernel 4.12 of the work done by Peter Zijlstra to
>> handle page fault without holding the mm semaphore.
>>
>> http://linux-kernel.2935.n7.nabble.com/RFC-PATCH-0-6-Another-go-at-speculative-page-faults-tt965642.html#none
>>
>> Compared to the Peter initial work, this series introduce a try spin
>> lock when dealing with speculative page fault. This is required to
>> avoid dead lock when handling a page fault while a TLB invalidate is
>> requested by an other CPU holding the PTE. Another change due to a
>> lock dependency issue with mapping->i_mmap_rwsem.
>>
>> This series also protect changes to VMA's data which are read or
>> change by the page fault handler. The protections is done through the
>> VMA's sequence number.
>>
>> This series is functional on x86 and PowerPC.
>>
>> It's building on top of v4.12-rc4 and relies on the change done by
>> Paul McKenney to the SRCU code allowing better performance by
>> maintaining per-CPU callback lists:
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=da915ad5cf25b5f5d358dd3670c3378d8ae8c03e
>>
>> Tests have been made using a large commercial in-memory database on a
>> PowerPC system with 752 CPUs. The results are very encouraging since
>> the loading of the 2TB database was faster by 20% with the speculative
>> page fault.
>>
>> Since tests are encouraging and running test suite didn't raise any
>> issue, I'd like this request for comment series to move to a patch
>> series soon. So please feel free to comment.
> 
> What other testing have you done? Other benchmarks (some numbers)? What
> about some standard worklaods like kbench? This is a pretty invasive
> change so I would expect much more numbers.

Thanks Michal for your feedback.

I mostly focused on this database workload since this is the one where
we hit the mmap_sem bottleneck when running on big node. On my usual
victim node, I checked for basic usage like kernel build time, but I
agree that's clearly not enough.

I try to find details about the 'kbench' you mentioned, but I didn't get
any valid entry.
Would you please point me on this or any other bench tool you think will
be useful here ?

> 
> It would also help to describe the highlevel design of the change here
> in the cover letter. This would make the review of specifics much
> easier.

You're right, I'll try to make a highlevel design.

Thanks,
Laurent.

[toc] | [prev] | [next] | [standalone]

#1662614

On Fri 09-06-17 17:25:51, Laurent Dufour wrote:
[...]
> Thanks Michal for your feedback.
> 
> I mostly focused on this database workload since this is the one where
> we hit the mmap_sem bottleneck when running on big node. On my usual
> victim node, I checked for basic usage like kernel build time, but I
> agree that's clearly not enough.
> 
> I try to find details about the 'kbench' you mentioned, but I didn't get
> any valid entry.
> Would you please point me on this or any other bench tool you think will
> be useful here ?

Sorry I meant kernbech (aka parallel kernel build). Other highly threaded
workloads doing a lot of page faults and address space modification
would be good to see as well. I wish I could give you much more
comprehensive list but I am not very good at benchmarks.

-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1662636

On 06/09/2017 09:35 AM, Michal Hocko wrote:
> On Fri 09-06-17 17:25:51, Laurent Dufour wrote:
> [...]
>> Thanks Michal for your feedback.
>>
>> I mostly focused on this database workload since this is the one where
>> we hit the mmap_sem bottleneck when running on big node. On my usual
>> victim node, I checked for basic usage like kernel build time, but I
>> agree that's clearly not enough.
>>
>> I try to find details about the 'kbench' you mentioned, but I didn't get
>> any valid entry.
>> Would you please point me on this or any other bench tool you think will
>> be useful here ?
> 
> Sorry I meant kernbech (aka parallel kernel build). Other highly threaded
> workloads doing a lot of page faults and address space modification
> would be good to see as well. I wish I could give you much more
> comprehensive list but I am not very good at benchmarks.
> 

Laurent,

Have you tried running the multi-fault microbenchmark by Kamezawa?
It does threaded page faults in parallel.
Peter ran that when he posted his specualtive page faults patches.
https://lkml.org/lkml/2010/1/6/28

Thanks.

Tim

[toc] | [prev] | [next] | [standalone]

#1664717

On 09/06/2017 18:59, Tim Chen wrote:
> On 06/09/2017 09:35 AM, Michal Hocko wrote:
>> On Fri 09-06-17 17:25:51, Laurent Dufour wrote:
>> [...]
>>> Thanks Michal for your feedback.
>>>
>>> I mostly focused on this database workload since this is the one where
>>> we hit the mmap_sem bottleneck when running on big node. On my usual
>>> victim node, I checked for basic usage like kernel build time, but I
>>> agree that's clearly not enough.
>>>
>>> I try to find details about the 'kbench' you mentioned, but I didn't get
>>> any valid entry.
>>> Would you please point me on this or any other bench tool you think will
>>> be useful here ?
>>
>> Sorry I meant kernbech (aka parallel kernel build). Other highly threaded
>> workloads doing a lot of page faults and address space modification
>> would be good to see as well. I wish I could give you much more
>> comprehensive list but I am not very good at benchmarks.
>>
> 
> Laurent,
> 
> Have you tried running the multi-fault microbenchmark by Kamezawa?
> It does threaded page faults in parallel.
> Peter ran that when he posted his specualtive page faults patches.
> https://lkml.org/lkml/2010/1/6/28


Thanks Tim to remind me about this, I downloaded and built it a time ago
and forget about it. I'll give it another try !

[toc] | [prev] | [next] | [standalone]

#1664658

On 09/06/2017 18:35, Michal Hocko wrote:
> On Fri 09-06-17 17:25:51, Laurent Dufour wrote:
> [...]
>> Thanks Michal for your feedback.
>>
>> I mostly focused on this database workload since this is the one where
>> we hit the mmap_sem bottleneck when running on big node. On my usual
>> victim node, I checked for basic usage like kernel build time, but I
>> agree that's clearly not enough.
>>
>> I try to find details about the 'kbench' you mentioned, but I didn't get
>> any valid entry.
>> Would you please point me on this or any other bench tool you think will
>> be useful here ?
> 
> Sorry I meant kernbech (aka parallel kernel build). Other highly threaded
> workloads doing a lot of page faults and address space modification
> would be good to see as well. I wish I could give you much more
> comprehensive list but I am not very good at benchmarks.
> 

Thanks Michal, I found kernbench 0.5, I will give it a try.

[toc] | [prev] | [next] | [standalone]

#1662688

On Fri, Jun 09, 2017 at 04:20:49PM +0200, Laurent Dufour wrote:
> This is a port on kernel 4.12 of the work done by Peter Zijlstra to
> handle page fault without holding the mm semaphore.
> 
> http://linux-kernel.2935.n7.nabble.com/RFC-PATCH-0-6-Another-go-at-speculative-page-faults-tt965642.html#none
> 
> Compared to the Peter initial work, this series introduce a try spin
> lock when dealing with speculative page fault. This is required to
> avoid dead lock when handling a page fault while a TLB invalidate is
> requested by an other CPU holding the PTE. Another change due to a
> lock dependency issue with mapping->i_mmap_rwsem.
> 
> This series also protect changes to VMA's data which are read or
> change by the page fault handler. The protections is done through the
> VMA's sequence number.
> 
> This series is functional on x86 and PowerPC.
> 
> It's building on top of v4.12-rc4 and relies on the change done by
> Paul McKenney to the SRCU code allowing better performance by
> maintaining per-CPU callback lists:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=da915ad5cf25b5f5d358dd3670c3378d8ae8c03e
> 
> Tests have been made using a large commercial in-memory database on a
> PowerPC system with 752 CPUs. The results are very encouraging since
> the loading of the 2TB database was faster by 20% with the speculative
> page fault.
> 
> Since tests are encouraging and running test suite didn't raise any
> issue, I'd like this request for comment series to move to a patch
> series soon. So please feel free to comment.

For whatever it is worth, this series passes moderate rcutorture testing,
30 minutes on each of sixteen scenarios.  Not that rcutorture is set up
to find mmap_sem issues, but it does place some stress on the kernel as
a whole.

Tested-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

							Thanx, Paul

> Changes since V3:
>  - support for the 5-level paging.
>  - abort speculative path before entering userfault code
>  - support for PowerPC architecture
>  - reorder the patch to fix build test errors.
> 
> Laurent Dufour (14):
>   mm: Introduce pte_spinlock
>   mm/spf: Try spin lock in speculative path
>   mm/spf: Fix fe.sequence init in __handle_mm_fault()
>   mm/spf: don't set fault entry's fields if locking failed
>   mm/spf; fix lock dependency against mapping->i_mmap_rwsem
>   mm/spf: Protect changes to vm_flags
>   mm/spf Protect vm_policy's changes against speculative pf
>   mm/spf: Add check on the VMA's flags
>   mm/spf: protect madvise vs speculative pf
>   mm/spf: protect mremap() against speculative pf
>   mm/spf: Don't call user fault callback in the speculative path
>   x86/mm: Update the handle_speculative_fault's path
>   powerpc/mm: Add speculative page fault
>   mm/spf: Clear FAULT_FLAG_KILLABLE in the speculative path
> 
> Peter Zijlstra (6):
>   mm: Dont assume page-table invariance during faults
>   mm: Prepare for FAULT_FLAG_SPECULATIVE
>   mm: VMA sequence count
>   mm: RCU free VMAs
>   mm: Provide speculative fault infrastructure
>   x86/mm: Add speculative pagefault handling
> 
>  arch/powerpc/mm/fault.c  |  25 +++-
>  arch/x86/mm/fault.c      |  14 +++
>  fs/proc/task_mmu.c       |   2 +
>  include/linux/mm.h       |   4 +
>  include/linux/mm_types.h |   3 +
>  kernel/fork.c            |   1 +
>  mm/init-mm.c             |   1 +
>  mm/internal.h            |  20 ++++
>  mm/madvise.c             |   4 +
>  mm/memory.c              | 291 +++++++++++++++++++++++++++++++++++++++--------
>  mm/mempolicy.c           |  10 +-
>  mm/mlock.c               |   9 +-
>  mm/mmap.c                | 123 +++++++++++++++-----
>  mm/mprotect.c            |   2 +
>  mm/mremap.c              |   7 ++
>  15 files changed, 435 insertions(+), 81 deletions(-)
> 
> -- 
> 2.7.4
> 

[toc] | [prev] | [next] | [standalone]

#1663404

Hello,

On Fri 09-06-17 16:20:49, Laurent Dufour wrote:
> This is a port on kernel 4.12 of the work done by Peter Zijlstra to
> handle page fault without holding the mm semaphore.
> 
> http://linux-kernel.2935.n7.nabble.com/RFC-PATCH-0-6-Another-go-at-speculative-page-faults-tt965642.html#none
> 
> Compared to the Peter initial work, this series introduce a try spin
> lock when dealing with speculative page fault. This is required to
> avoid dead lock when handling a page fault while a TLB invalidate is
> requested by an other CPU holding the PTE. Another change due to a
> lock dependency issue with mapping->i_mmap_rwsem.
> 
> This series also protect changes to VMA's data which are read or
> change by the page fault handler. The protections is done through the
> VMA's sequence number.
> 
> This series is functional on x86 and PowerPC.
> 
> It's building on top of v4.12-rc4 and relies on the change done by
> Paul McKenney to the SRCU code allowing better performance by
> maintaining per-CPU callback lists:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=da915ad5cf25b5f5d358dd3670c3378d8ae8c03e
> 
> Tests have been made using a large commercial in-memory database on a
> PowerPC system with 752 CPUs. The results are very encouraging since
> the loading of the 2TB database was faster by 20% with the speculative
> page fault.
> 
> Since tests are encouraging and running test suite didn't raise any
> issue, I'd like this request for comment series to move to a patch
> series soon. So please feel free to comment.

I had a look at the series and I have one comment regarding the whole
structure of the series: Instead of taking original Peter's patches and
then fixing up various problems with them, either fold the fixes into
original patches which introduced problems (this would make sense for
example for the lock inversion issue you fix) or just put these changes
to a place in the series where they logically belong - e.g. VMA is
protected by the sequence counter in patch 4 and then you add various
places that were missed later in the series. Instead of this just handle
sequence count protection in consecutive logical steps like vma_adjust()
changes, mremap() changes, munmap() changes, vma->flags protection, ...

Also amount of 'XXX' comments seems to be a bit to high and these should be
addressed.

								Honza
> 
> Changes since V3:
>  - support for the 5-level paging.
>  - abort speculative path before entering userfault code
>  - support for PowerPC architecture
>  - reorder the patch to fix build test errors.
> 
> Laurent Dufour (14):
>   mm: Introduce pte_spinlock
>   mm/spf: Try spin lock in speculative path
>   mm/spf: Fix fe.sequence init in __handle_mm_fault()
>   mm/spf: don't set fault entry's fields if locking failed
>   mm/spf; fix lock dependency against mapping->i_mmap_rwsem
>   mm/spf: Protect changes to vm_flags
>   mm/spf Protect vm_policy's changes against speculative pf
>   mm/spf: Add check on the VMA's flags
>   mm/spf: protect madvise vs speculative pf
>   mm/spf: protect mremap() against speculative pf
>   mm/spf: Don't call user fault callback in the speculative path
>   x86/mm: Update the handle_speculative_fault's path
>   powerpc/mm: Add speculative page fault
>   mm/spf: Clear FAULT_FLAG_KILLABLE in the speculative path
> 
> Peter Zijlstra (6):
>   mm: Dont assume page-table invariance during faults
>   mm: Prepare for FAULT_FLAG_SPECULATIVE
>   mm: VMA sequence count
>   mm: RCU free VMAs
>   mm: Provide speculative fault infrastructure
>   x86/mm: Add speculative pagefault handling
> 
>  arch/powerpc/mm/fault.c  |  25 +++-
>  arch/x86/mm/fault.c      |  14 +++
>  fs/proc/task_mmu.c       |   2 +
>  include/linux/mm.h       |   4 +
>  include/linux/mm_types.h |   3 +
>  kernel/fork.c            |   1 +
>  mm/init-mm.c             |   1 +
>  mm/internal.h            |  20 ++++
>  mm/madvise.c             |   4 +
>  mm/memory.c              | 291 +++++++++++++++++++++++++++++++++++++++--------
>  mm/mempolicy.c           |  10 +-
>  mm/mlock.c               |   9 +-
>  mm/mmap.c                | 123 +++++++++++++++-----
>  mm/mprotect.c            |   2 +
>  mm/mremap.c              |   7 ++
>  15 files changed, 435 insertions(+), 81 deletions(-)
> 
> -- 
> 2.7.4
> 
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[toc] | [prev] | [next] | [standalone]

#1664722

On 12/06/2017 12:20, Jan Kara wrote:
> Hello,
> 
> On Fri 09-06-17 16:20:49, Laurent Dufour wrote:
>> This is a port on kernel 4.12 of the work done by Peter Zijlstra to
>> handle page fault without holding the mm semaphore.
>>
>> http://linux-kernel.2935.n7.nabble.com/RFC-PATCH-0-6-Another-go-at-speculative-page-faults-tt965642.html#none
>>
>> Compared to the Peter initial work, this series introduce a try spin
>> lock when dealing with speculative page fault. This is required to
>> avoid dead lock when handling a page fault while a TLB invalidate is
>> requested by an other CPU holding the PTE. Another change due to a
>> lock dependency issue with mapping->i_mmap_rwsem.
>>
>> This series also protect changes to VMA's data which are read or
>> change by the page fault handler. The protections is done through the
>> VMA's sequence number.
>>
>> This series is functional on x86 and PowerPC.
>>
>> It's building on top of v4.12-rc4 and relies on the change done by
>> Paul McKenney to the SRCU code allowing better performance by
>> maintaining per-CPU callback lists:
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=da915ad5cf25b5f5d358dd3670c3378d8ae8c03e
>>
>> Tests have been made using a large commercial in-memory database on a
>> PowerPC system with 752 CPUs. The results are very encouraging since
>> the loading of the 2TB database was faster by 20% with the speculative
>> page fault.
>>
>> Since tests are encouraging and running test suite didn't raise any
>> issue, I'd like this request for comment series to move to a patch
>> series soon. So please feel free to comment.
> 
> I had a look at the series and I have one comment regarding the whole
> structure of the series: Instead of taking original Peter's patches and
> then fixing up various problems with them, either fold the fixes into
> original patches which introduced problems (this would make sense for
> example for the lock inversion issue you fix) or just put these changes
> to a place in the series where they logically belong - e.g. VMA is
> protected by the sequence counter in patch 4 and then you add various
> places that were missed later in the series. Instead of this just handle
> sequence count protection in consecutive logical steps like vma_adjust()
> changes, mremap() changes, munmap() changes, vma->flags protection, ...

Thanks Jan for the review,

I tried to keep the Peter's patches intact, but I agree, that may not be
a good idea, and I tend to split this series in too much small patches.

I think I'll fold the changes I made into the original patches, this may
also fix some build test issue raised earlier.

> Also amount of 'XXX' comments seems to be a bit to high and these should be
> addressed.

Right, I'll get rid of them !

Thanks,
Laurent.

> 
> 								Honza
>>
>> Changes since V3:
>>  - support for the 5-level paging.
>>  - abort speculative path before entering userfault code
>>  - support for PowerPC architecture
>>  - reorder the patch to fix build test errors.
>>
>> Laurent Dufour (14):
>>   mm: Introduce pte_spinlock
>>   mm/spf: Try spin lock in speculative path
>>   mm/spf: Fix fe.sequence init in __handle_mm_fault()
>>   mm/spf: don't set fault entry's fields if locking failed
>>   mm/spf; fix lock dependency against mapping->i_mmap_rwsem
>>   mm/spf: Protect changes to vm_flags
>>   mm/spf Protect vm_policy's changes against speculative pf
>>   mm/spf: Add check on the VMA's flags
>>   mm/spf: protect madvise vs speculative pf
>>   mm/spf: protect mremap() against speculative pf
>>   mm/spf: Don't call user fault callback in the speculative path
>>   x86/mm: Update the handle_speculative_fault's path
>>   powerpc/mm: Add speculative page fault
>>   mm/spf: Clear FAULT_FLAG_KILLABLE in the speculative path
>>
>> Peter Zijlstra (6):
>>   mm: Dont assume page-table invariance during faults
>>   mm: Prepare for FAULT_FLAG_SPECULATIVE
>>   mm: VMA sequence count
>>   mm: RCU free VMAs
>>   mm: Provide speculative fault infrastructure
>>   x86/mm: Add speculative pagefault handling
>>
>>  arch/powerpc/mm/fault.c  |  25 +++-
>>  arch/x86/mm/fault.c      |  14 +++
>>  fs/proc/task_mmu.c       |   2 +
>>  include/linux/mm.h       |   4 +
>>  include/linux/mm_types.h |   3 +
>>  kernel/fork.c            |   1 +
>>  mm/init-mm.c             |   1 +
>>  mm/internal.h            |  20 ++++
>>  mm/madvise.c             |   4 +
>>  mm/memory.c              | 291 +++++++++++++++++++++++++++++++++++++++--------
>>  mm/mempolicy.c           |  10 +-
>>  mm/mlock.c               |   9 +-
>>  mm/mmap.c                | 123 +++++++++++++++-----
>>  mm/mprotect.c            |   2 +
>>  mm/mremap.c              |   7 ++
>>  15 files changed, 435 insertions(+), 81 deletions(-)
>>
>> -- 
>> 2.7.4
>>

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web