Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1693318 > unrolled thread

Re: [HMM 12/15] mm/migrate: new memory migration helper for use with device memory v4

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [HMM 12/15] mm/migrate: new memory migration helper for use with  device memory v4 Evgeny Baskakov <ebaskakov@nvidia.com> - 2017-07-21 03:10 +0200
    Re: [HMM 12/15] mm/migrate: new memory migration helper for use with  device memory v4 Jerome Glisse <jglisse@redhat.com> - 2017-07-21 03:40 +0200
      Re: [HMM 12/15] mm/migrate: new memory migration helper for use with  device memory v4 Evgeny Baskakov <ebaskakov@nvidia.com> - 2017-07-22 00:10 +0200
      Re: [HMM 12/15] mm/migrate: new memory migration helper for use with  device memory v4 Evgeny Baskakov <ebaskakov@nvidia.com> - 2017-07-26 00:50 +0200
        Re: [HMM 12/15] mm/migrate: new memory migration helper for use with  device memory v4 Jerome Glisse <jglisse@redhat.com> - 2017-07-26 21:20 +0200

#1693318 — Re: [HMM 12/15] mm/migrate: new memory migration helper for use with device memory v4

[Multipart message — attachments visible in raw view] — view raw

On 7/14/17 5:55 PM, Jerome Glisse wrote:

> ...
>
> Cheers,
> Jérôme

Hi Jerome,

I think I just found a couple of new issues, now related to fork/execve.

1) With a fork() followed by execve(), the child process makes a copy of 
the parent mm_struct object, including the "hmm" pointer. Later on, an 
execve() syscall in the child process frees the old mm_struct, and 
destroys the "hmm" object - which apparently it shouldn't do, because 
the "hmm" object is shared between the parent and child processes:

(gdb) bt
#0  hmm_mm_destroy (mm=0xffff88080757aa40) at mm/hmm.c:134
#1  0xffffffff81058567 in __mmdrop (mm=0xffff88080757aa40) at 
kernel/fork.c:889
#2  0xffffffff8105904f in mmdrop (mm=<optimized out>) at 
./include/linux/sched/mm.h:42
#3  __mmput (mm=<optimized out>) at kernel/fork.c:916
#4  mmput (mm=0xffff88080757aa40) at kernel/fork.c:927
#5  0xffffffff811c5a68 in exec_mmap (mm=<optimized out>) at fs/exec.c:1057
#6  flush_old_exec (bprm=<optimized out>) at fs/exec.c:1284
#7  0xffffffff81214460 in load_elf_binary (bprm=0xffff8808133b1978) at 
fs/binfmt_elf.c:855
#8  0xffffffff811c4fce in search_binary_handler 
(bprm=0xffff88081b40cb78) at fs/exec.c:1625
#9  0xffffffff811c6bbf in exec_binprm (bprm=<optimized out>) at 
fs/exec.c:1667
#10 do_execveat_common (fd=<optimized out>, filename=0xffff88080a101200, 
flags=0x0, argv=..., envp=...) at fs/exec.c:1789
#11 0xffffffff811c6fda in do_execve (__envp=<optimized out>, 
__argv=<optimized out>, filename=<optimized out>) at fs/exec.c:1833
#12 SYSC_execve (envp=<optimized out>, argv=<optimized out>, 
filename=<optimized out>) at fs/exec.c:1914
#13 SyS_execve (filename=<optimized out>, argv=0x7f4e5c2aced0, 
envp=0x7f4e5c2aceb0) at fs/exec.c:1909
#14 0xffffffff810018dd in do_syscall_64 (regs=0xffff88081b40cb78) at 
arch/x86/entry/common.c:284
#15 0xffffffff819e2c06 in entry_SYSCALL_64 () at 
arch/x86/entry/entry_64.S:245

This leads to a sporadic memory corruption in the parent process:

Thread 200 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 3685]
0xffffffff811a3efe in __mmu_notifier_invalidate_range_start 
(mm=0xffff880807579000, start=0x7f4e5c62f000, end=0x7f4e5c66f000) at 
mm/mmu_notifier.c:199
199            if (mn->ops->invalidate_range_start)
(gdb) bt
#0  0xffffffff811a3efe in __mmu_notifier_invalidate_range_start 
(mm=0xffff880807579000, start=0x7f4e5c62f000, end=0x7f4e5c66f000) at 
mm/mmu_notifier.c:199
#1  0xffffffff811ae471 in mmu_notifier_invalidate_range_start 
(end=<optimized out>, start=<optimized out>, mm=<optimized out>) at 
./include/linux/mmu_notifier.h:282
#2  migrate_vma_collect (migrate=0xffffc90003ca3940) at mm/migrate.c:2280
#3  0xffffffff811b04a7 in migrate_vma (ops=<optimized out>, 
vma=0x7f4e5c62f000, start=0x7f4e5c62f000, end=0x7f4e5c66f000, 
src=0xffffc90003ca39d0, dst=0xffffc90003ca39d0, 
private=0xffffc90003ca39c0) at mm/migrate.c:2819
(gdb) p mn->ops
$2 = (const struct mmu_notifier_ops *) 0x6b6b6b6b6b6b6b6b

Please see attached a reproducer (sanity_rmem004_fork.tgz). Use 
"./build.sh; sudo ./kload.sh; ./run.sh" to recreate the issue on your end.


2) A slight modification of the affected application does not use 
fork(). Instead, an execve() call from a parallel thread replaces the 
original process. This is a particularly interesting case, because at 
that point the process is busy migrating pages to/from device.

Here's what happens:

0xffffffff811b9879 in commit_charge (page=<optimized out>, 
lrucare=<optimized out>, memcg=<optimized out>) at mm/memcontrol.c:2060
2060        VM_BUG_ON_PAGE(page->mem_cgroup, page);
(gdb) bt
#0  0xffffffff811b9879 in commit_charge (page=<optimized out>, 
lrucare=<optimized out>, memcg=<optimized out>) at mm/memcontrol.c:2060
#1  0xffffffff811b93d6 in commit_charge (lrucare=<optimized out>, 
memcg=<optimized out>, page=<optimized out>) at 
./include/linux/page-flags.h:149
#2  mem_cgroup_commit_charge (page=0xffff88081b68cb70, 
memcg=0xffff88081b051548, lrucare=<optimized out>, compound=<optimized 
out>) at mm/memcontrol.c:5468
#3  0xffffffff811b10d4 in migrate_vma_insert_page (migrate=<optimized 
out>, dst=<optimized out>, src=<optimized out>, page=<optimized out>, 
addr=<optimized out>) at mm/migrate.c:2605
#4  migrate_vma_pages (migrate=<optimized out>) at mm/migrate.c:2647
#5  migrate_vma (ops=<optimized out>, vma=<optimized out>, 
start=<optimized out>, end=<optimized out>, src=<optimized out>, 
dst=<optimized out>, private=0xffffc900037439c0) at mm/migrate.c:2844


Please find another reproducer attached (sanity_rmem004_execve.tgz) for 
this issue.

Thanks!

-- 
Evgeny Baskakov
NVIDIA

[toc] | [next] | [standalone]

#1693326

On Thu, Jul 20, 2017 at 06:00:08PM -0700, Evgeny Baskakov wrote:
> On 7/14/17 5:55 PM, Jerome Glisse wrote:
> Hi Jerome,
> 
> I think I just found a couple of new issues, now related to fork/execve.
> 
> 1) With a fork() followed by execve(), the child process makes a copy of the
> parent mm_struct object, including the "hmm" pointer. Later on, an execve()
> syscall in the child process frees the old mm_struct, and destroys the "hmm"
> object - which apparently it shouldn't do, because the "hmm" object is
> shared between the parent and child processes:
> 
> (gdb) bt
> #0  hmm_mm_destroy (mm=0xffff88080757aa40) at mm/hmm.c:134
> #1  0xffffffff81058567 in __mmdrop (mm=0xffff88080757aa40) at
> kernel/fork.c:889
> #2  0xffffffff8105904f in mmdrop (mm=<optimized out>) at
> ./include/linux/sched/mm.h:42
> #3  __mmput (mm=<optimized out>) at kernel/fork.c:916
> #4  mmput (mm=0xffff88080757aa40) at kernel/fork.c:927
> #5  0xffffffff811c5a68 in exec_mmap (mm=<optimized out>) at fs/exec.c:1057
> #6  flush_old_exec (bprm=<optimized out>) at fs/exec.c:1284
> #7  0xffffffff81214460 in load_elf_binary (bprm=0xffff8808133b1978) at
> fs/binfmt_elf.c:855
> #8  0xffffffff811c4fce in search_binary_handler (bprm=0xffff88081b40cb78) at
> fs/exec.c:1625
> #9  0xffffffff811c6bbf in exec_binprm (bprm=<optimized out>) at
> fs/exec.c:1667
> #10 do_execveat_common (fd=<optimized out>, filename=0xffff88080a101200,
> flags=0x0, argv=..., envp=...) at fs/exec.c:1789
> #11 0xffffffff811c6fda in do_execve (__envp=<optimized out>,
> __argv=<optimized out>, filename=<optimized out>) at fs/exec.c:1833
> #12 SYSC_execve (envp=<optimized out>, argv=<optimized out>,
> filename=<optimized out>) at fs/exec.c:1914
> #13 SyS_execve (filename=<optimized out>, argv=0x7f4e5c2aced0,
> envp=0x7f4e5c2aceb0) at fs/exec.c:1909
> #14 0xffffffff810018dd in do_syscall_64 (regs=0xffff88081b40cb78) at
> arch/x86/entry/common.c:284
> #15 0xffffffff819e2c06 in entry_SYSCALL_64 () at
> arch/x86/entry/entry_64.S:245
> 
> This leads to a sporadic memory corruption in the parent process:
> 
> Thread 200 received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 3685]
> 0xffffffff811a3efe in __mmu_notifier_invalidate_range_start
> (mm=0xffff880807579000, start=0x7f4e5c62f000, end=0x7f4e5c66f000) at
> mm/mmu_notifier.c:199
> 199            if (mn->ops->invalidate_range_start)
> (gdb) bt
> #0  0xffffffff811a3efe in __mmu_notifier_invalidate_range_start
> (mm=0xffff880807579000, start=0x7f4e5c62f000, end=0x7f4e5c66f000) at
> mm/mmu_notifier.c:199
> #1  0xffffffff811ae471 in mmu_notifier_invalidate_range_start
> (end=<optimized out>, start=<optimized out>, mm=<optimized out>) at
> ./include/linux/mmu_notifier.h:282
> #2  migrate_vma_collect (migrate=0xffffc90003ca3940) at mm/migrate.c:2280
> #3  0xffffffff811b04a7 in migrate_vma (ops=<optimized out>,
> vma=0x7f4e5c62f000, start=0x7f4e5c62f000, end=0x7f4e5c66f000,
> src=0xffffc90003ca39d0, dst=0xffffc90003ca39d0, private=0xffffc90003ca39c0)
> at mm/migrate.c:2819
> (gdb) p mn->ops
> $2 = (const struct mmu_notifier_ops *) 0x6b6b6b6b6b6b6b6b
> 
> Please see attached a reproducer (sanity_rmem004_fork.tgz). Use "./build.sh;
> sudo ./kload.sh; ./run.sh" to recreate the issue on your end.
> 
> 
> 2) A slight modification of the affected application does not use fork().
> Instead, an execve() call from a parallel thread replaces the original
> process. This is a particularly interesting case, because at that point the
> process is busy migrating pages to/from device.
> 
> Here's what happens:
> 
> 0xffffffff811b9879 in commit_charge (page=<optimized out>,
> lrucare=<optimized out>, memcg=<optimized out>) at mm/memcontrol.c:2060
> 2060        VM_BUG_ON_PAGE(page->mem_cgroup, page);
> (gdb) bt
> #0  0xffffffff811b9879 in commit_charge (page=<optimized out>,
> lrucare=<optimized out>, memcg=<optimized out>) at mm/memcontrol.c:2060
> #1  0xffffffff811b93d6 in commit_charge (lrucare=<optimized out>,
> memcg=<optimized out>, page=<optimized out>) at
> ./include/linux/page-flags.h:149
> #2  mem_cgroup_commit_charge (page=0xffff88081b68cb70,
> memcg=0xffff88081b051548, lrucare=<optimized out>, compound=<optimized out>)
> at mm/memcontrol.c:5468
> #3  0xffffffff811b10d4 in migrate_vma_insert_page (migrate=<optimized out>,
> dst=<optimized out>, src=<optimized out>, page=<optimized out>,
> addr=<optimized out>) at mm/migrate.c:2605
> #4  migrate_vma_pages (migrate=<optimized out>) at mm/migrate.c:2647
> #5  migrate_vma (ops=<optimized out>, vma=<optimized out>, start=<optimized
> out>, end=<optimized out>, src=<optimized out>, dst=<optimized out>,
> private=0xffffc900037439c0) at mm/migrate.c:2844
> 
> 
> Please find another reproducer attached (sanity_rmem004_execve.tgz) for this
> issue.
> 

So i pushed an updated hmm-next branch it should have all fixes so far, including
something that should fix this issue. I still want to go over all emails again
to make sure i am not forgetting anything.

Cheers,
Jérôme

[toc] | [prev] | [next] | [standalone]

#1694011

On 7/20/17 6:33 PM, Jerome Glisse wrote:

> So i pushed an updated hmm-next branch it should have all fixes so far, including
> something that should fix this issue. I still want to go over all emails again
> to make sure i am not forgetting anything.
>
> Cheers,
> Jérôme

Hi Jerome,

The issues I observed seem to be gone!

I am still running my stress tests, though. I will let you know if there 
is anything else that needs to be addressed.

Thanks!

-- 
Evgeny Baskakov
NVIDIA

[toc] | [prev] | [next] | [standalone]

#1696645

On 7/20/17 6:33 PM, Jerome Glisse wrote:

> So i pushed an updated hmm-next branch it should have all fixes so far, including
> something that should fix this issue. I still want to go over all emails again
> to make sure i am not forgetting anything.
>
> Cheers,
> Jérôme

Hi Jerome,

Thanks for updating the documentation for hmm_devmem_ops.

I have an inquiry about the "fault" callback, though. The documentation 
says "Returns: 0 on success", but can the driver set any VM_FAULT_* 
flags? For instance, the driver might want to set the VM_FAULT_MAJOR 
flag to indicate that a heavy-weight page migration has happened on the 
page fault.

If that is possible, can you please update the documentation and list 
the flags that are permitted in the callback's return value?

Thanks!

-- 
Evgeny Baskakov
NVIDIA

[toc] | [prev] | [next] | [standalone]

#1697498

On Tue, Jul 25, 2017 at 03:45:14PM -0700, Evgeny Baskakov wrote:
> On 7/20/17 6:33 PM, Jerome Glisse wrote:
> 
> > So i pushed an updated hmm-next branch it should have all fixes so far, including
> > something that should fix this issue. I still want to go over all emails again
> > to make sure i am not forgetting anything.
> > 
> > Cheers,
> > Jérôme
> 
> Hi Jerome,
> 
> Thanks for updating the documentation for hmm_devmem_ops.
> 
> I have an inquiry about the "fault" callback, though. The documentation says
> "Returns: 0 on success", but can the driver set any VM_FAULT_* flags? For
> instance, the driver might want to set the VM_FAULT_MAJOR flag to indicate
> that a heavy-weight page migration has happened on the page fault.
> 
> If that is possible, can you please update the documentation and list the
> flags that are permitted in the callback's return value?

Yes you can.

Jérôme

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web