Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1684694 > unrolled thread

Re: [HMM 12/15] mm/migrate: new memory migration helper for use with device memory v4

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [HMM 12/15] mm/migrate: new memory migration helper for use with  device memory v4 Evgeny Baskakov <ebaskakov@nvidia.com> - 2017-07-11 01:50 +0200
    Re: [HMM 12/15] mm/migrate: new memory migration helper for use with  device memory v4 Jerome Glisse <jglisse@redhat.com> - 2017-07-11 20:30 +0200
      Re: [HMM 12/15] mm/migrate: new memory migration helper for use with  device memory v4 Evgeny Baskakov <ebaskakov@nvidia.com> - 2017-07-11 20:50 +0200
        Re: [HMM 12/15] mm/migrate: new memory migration helper for use with  device memory v4 Jerome Glisse <jglisse@redhat.com> - 2017-07-11 20:50 +0200
          Re: [HMM 12/15] mm/migrate: new memory migration helper for use with  device memory v4 Evgeny Baskakov <ebaskakov@nvidia.com> - 2017-07-11 21:40 +0200
            Re: [HMM 12/15] mm/migrate: new memory migration helper for use with  device memory v4 Jerome Glisse <jglisse@redhat.com> - 2017-07-13 22:20 +0200
              Re: [HMM 12/15] mm/migrate: new memory migration helper for use with  device memory v4 Evgeny Baskakov <ebaskakov@nvidia.com> - 2017-07-14 21:50 +0200
                Re: [HMM 12/15] mm/migrate: new memory migration helper for use with  device memory v4 Jerome Glisse <jglisse@redhat.com> - 2017-07-15 03:00 +0200
                  Re: [HMM 12/15] mm/migrate: new memory migration helper for use with  device memory v4 Evgeny Baskakov <ebaskakov@nvidia.com> - 2017-07-15 07:10 +0200

#1684694 — Re: [HMM 12/15] mm/migrate: new memory migration helper for use with device memory v4

[Multipart message — attachments visible in raw view] — view raw

On 6/30/17 5:57 PM, Jerome Glisse wrote:

...

Hi Jerome,

I am working on a sporadic data corruption seen in highly contented use 
cases. So far, I've been able to re-create a sporadic hang that happens 
when multiple threads compete to migrate the same page to and from 
device memory. The reproducer uses only the dummy driver from hmm-next.

Please find attached. This is how it hangs on my 12-core Intel i7-5930K 
SMT system:

&&& 2 migrate threads, 2 read threads: STARTING
(EE:84) hmm_buffer_mirror_read error -1
&&& 2 migrate threads, 2 read threads: PASSED
&&& 2 migrate threads, 3 read threads: STARTING
&&& 2 migrate threads, 3 read threads: PASSED
&&& 2 migrate threads, 4 read threads: STARTING
&&& 2 migrate threads, 4 read threads: PASSED
&&& 3 migrate threads, 2 read threads: STARTING

The kernel log (also attached) shows multiple threads blocked in 
hmm_vma_fault() and migrate_vma():

[  139.054907] sanity_rmem004  D13528  3997   3818 0x00000000
[  139.054912] Call Trace:
[  139.054914]  __schedule+0x20b/0x6c0
[  139.054916]  schedule+0x36/0x80
[  139.054920]  io_schedule+0x16/0x40
[  139.054923]  __lock_page+0xf2/0x130
[  139.054929]  migrate_vma+0x48a/0xee0
[  139.054933]  dummy_migrate.isra.10+0xd9/0x110 [hmm_dmirror]
[  139.054945]  dummy_fops_unlocked_ioctl+0x1e8/0x330 [hmm_dmirror]
[  139.054954]  do_vfs_ioctl+0x96/0x5a0
[  139.054957]  SyS_ioctl+0x79/0x90
[  139.054960]  entry_SYSCALL_64_fastpath+0x13/0x94
...
[  139.055067] sanity_rmem004  D13136  3999   3818 0x00000000
[  139.055072] Call Trace:
[  139.055074]  __schedule+0x20b/0x6c0
[  139.055076]  schedule+0x36/0x80
[  139.055079]  io_schedule+0x16/0x40
[  139.055083]  wait_on_page_bit+0xee/0x120
[  139.055089]  __migration_entry_wait+0xe8/0x190
[  139.055091]  migration_entry_wait+0x5f/0x70
[  139.055094]  do_swap_page+0x4c7/0x4e0
[  139.055096]  __handle_mm_fault+0x347/0x9d0
[  139.055099]  handle_mm_fault+0x88/0x150
[  139.055103]  hmm_vma_walk_clear+0x8f/0xd0
[  139.055105]  hmm_vma_walk_pmd+0x1ba/0x250
[  139.055109]  __walk_page_range+0x1e8/0x420
[  139.055112]  walk_page_range+0x73/0xf0
[  139.055114]  hmm_vma_fault+0x180/0x260
[  139.055121]  dummy_fault+0xda/0x1f0 [hmm_dmirror]
[  139.055138]  dummy_fops_unlocked_ioctl+0x12c/0x330 [hmm_dmirror]
[  139.055142]  do_vfs_ioctl+0x96/0x5a0
[  139.055145]  SyS_ioctl+0x79/0x90
[  139.055148]  entry_SYSCALL_64_fastpath+0x13/0x94

Please compile and run the attached program this way:

$ ./build.sh
$ sudo ./kload.sh
$ sudo ./run.sh

Thanks!

Evgeny Baskakov
NVIDIA

[toc] | [next] | [standalone]

#1685278

[Multipart message — attachments visible in raw view] — view raw

On Mon, Jul 10, 2017 at 04:44:38PM -0700, Evgeny Baskakov wrote:
> On 6/30/17 5:57 PM, Jerome Glisse wrote:
> 
> ...
> 
> Hi Jerome,
> 
> I am working on a sporadic data corruption seen in highly contented use
> cases. So far, I've been able to re-create a sporadic hang that happens when
> multiple threads compete to migrate the same page to and from device memory.
> The reproducer uses only the dummy driver from hmm-next.
> 
> Please find attached. This is how it hangs on my 12-core Intel i7-5930K SMT
> system:
> 

Can you test if attached patch helps ? I am having trouble reproducing this
from inside a vm.

My theory is that 2 concurrent CPU page fault happens. First one manage to
start the migration back to system memory but second one see the migration
special entry and call migration_entry_wait() which increase page refcount
and this happen before first one check page refcount are ok for migration.

For regular migration such scenario is ok as the migration bails out and
because page is CPU accessible there is no need to kick again the migration
for other thread that CPU fault to migrate.

I am looking into how i can change migration_entry_wait() not to refcount
pages. Let me know if the attached patch helps.

Thank you
Jérôme

[toc] | [prev] | [next] | [standalone]

#1685289

[Multipart message — attachments visible in raw view] — view raw

On 7/11/17 11:29 AM, Jerome Glisse wrote:
> Can you test if attached patch helps ? I am having trouble reproducing 
> this
> from inside a vm.
>
> My theory is that 2 concurrent CPU page fault happens. First one manage to
> start the migration back to system memory but second one see the migration
> special entry and call migration_entry_wait() which increase page refcount
> and this happen before first one check page refcount are ok for migration.
>
> For regular migration such scenario is ok as the migration bails out and
> because page is CPU accessible there is no need to kick again the migration
> for other thread that CPU fault to migrate.
>
> I am looking into how i can change migration_entry_wait() not to refcount
> pages. Let me know if the attached patch helps.
>
> Thank you
> Jérôme

Hi Jerome,

Thanks for the update.

Unfortunately, the patch does not help. I just applied it and recompiled 
the kernel. Please find attached a new kernel log and an app log.

-- 
Evgeny Baskakov
NVIDIA

[toc] | [prev] | [next] | [standalone]

#1685290

On Tue, Jul 11, 2017 at 11:42:20AM -0700, Evgeny Baskakov wrote:
> On 7/11/17 11:29 AM, Jerome Glisse wrote:
> > Can you test if attached patch helps ? I am having trouble reproducing
> > this
> > from inside a vm.
> > 
> > My theory is that 2 concurrent CPU page fault happens. First one manage to
> > start the migration back to system memory but second one see the migration
> > special entry and call migration_entry_wait() which increase page refcount
> > and this happen before first one check page refcount are ok for migration.
> > 
> > For regular migration such scenario is ok as the migration bails out and
> > because page is CPU accessible there is no need to kick again the migration
> > for other thread that CPU fault to migrate.
> > 
> > I am looking into how i can change migration_entry_wait() not to refcount
> > pages. Let me know if the attached patch helps.
> > 
> > Thank you
> > Jérôme
> 
> Hi Jerome,
> 
> Thanks for the update.
> 
> Unfortunately, the patch does not help. I just applied it and recompiled the
> kernel. Please find attached a new kernel log and an app log.
> 

What are the symptoms ? The program just stop making any progress and you
trigger a sysrequest to dump current states of each threads ? In this
log i don't see migration_entry_wait() anymore but it seems to be waiting
on page lock so there might be 2 issues here.

Jérôme

[toc] | [prev] | [next] | [standalone]

#1685309

On 7/11/17 11:49 AM, Jerome Glisse wrote:

>
> What are the symptoms ? The program just stop making any progress and you
> trigger a sysrequest to dump current states of each threads ? In this
> log i don't see migration_entry_wait() anymore but it seems to be waiting
> on page lock so there might be 2 issues here.
>
> Jérôme

That is correct, the program is not making any progress.

The stack traces in the kernel log are produced by a "sysrq w" (blocked 
tasks) command.

Thanks,

-- 
Evgeny Baskakov
NVIDIA

[toc] | [prev] | [next] | [standalone]

#1686881

On Tue, Jul 11, 2017 at 12:35:03PM -0700, Evgeny Baskakov wrote:
> On 7/11/17 11:49 AM, Jerome Glisse wrote:
> 
> > 
> > What are the symptoms ? The program just stop making any progress and you
> > trigger a sysrequest to dump current states of each threads ? In this
> > log i don't see migration_entry_wait() anymore but it seems to be waiting
> > on page lock so there might be 2 issues here.
> > 
> > Jérôme
> 
> That is correct, the program is not making any progress.
> 
> The stack traces in the kernel log are produced by a "sysrq w" (blocked
> tasks) command.
> 

I updated hmm-next with a patch that might fix some other issues but i am
still trying to get this dead lock you are seing. Does it happens quickly
with the test program ?

I can't see how it dead lock on the page lock bit. Going over and over all
code path we always unlock page once we are done or when we back off from
migration. So far i haven't been able to reproduce thought i haven't had
much time to test as other thing kept me busy. I should be back looking into
that tomorrow.

https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-next

Cheers,
Jérôme

[toc] | [prev] | [next] | [standalone]

#1687627

[Multipart message — attachments visible in raw view] — view raw

On 7/13/17 1:16 PM, Jerome Glisse wrote:

> ...
>

Hi Jerome,

I have hit another kind of hang. Briefly, if a not yet allocated page 
faults on CPU during migration to device memory, any subsequent 
migration will fail for such page. Such a situation can trigger if a CPU 
page fault happens just immediately after migrate_vma() starts unmapping 
pages to migrate.

Please find attached a reproducer based on the sample driver. In the 
hmm_test() function, an HMM_DMIRROR_MIGRATE request is triggered from a 
separate thread for not yet allocated pages (coming from malloc). In the 
same time, a HMM_DMIRROR_READ request is made for the same pages. This 
results in a sporadic app-side hang, because random number of pages 
never migrate to device memory.

Note that if the pages are touched (initialized with data) prior to 
that, everything works as expected: all HMM_DMIRROR_READ and 
HMM_DMIRROR_MIGRATE requests eventually succeed. See comments in the 
hmm_test() function.

Thanks!

-- 
Evgeny Baskakov
NVIDIA

[toc] | [prev] | [next] | [standalone]

#1687735

On Fri, Jul 14, 2017 at 12:43:51PM -0700, Evgeny Baskakov wrote:
> On 7/13/17 1:16 PM, Jerome Glisse wrote:
> Hi Jerome,
> 
> I have hit another kind of hang. Briefly, if a not yet allocated page faults
> on CPU during migration to device memory, any subsequent migration will fail
> for such page. Such a situation can trigger if a CPU page fault happens just
> immediately after migrate_vma() starts unmapping pages to migrate.
> 
> Please find attached a reproducer based on the sample driver. In the
> hmm_test() function, an HMM_DMIRROR_MIGRATE request is triggered from a
> separate thread for not yet allocated pages (coming from malloc). In the
> same time, a HMM_DMIRROR_READ request is made for the same pages. This
> results in a sporadic app-side hang, because random number of pages never
> migrate to device memory.
> 
> Note that if the pages are touched (initialized with data) prior to that,
> everything works as expected: all HMM_DMIRROR_READ and HMM_DMIRROR_MIGRATE
> requests eventually succeed. See comments in the hmm_test() function.
> 

So pushed an updated hmm-next branch this should fix all issues you had.
Thought i am not sure about the test in this mail, all i see is that it
continously spit error messages but it does not hang (i let it run 20min
or so). Dunno if that is what expected. Let me know if this is still an
issue and if so what should be the expected output of this test program.

Cheers,
Jérôme

[toc] | [prev] | [next] | [standalone]

#1687859

On 7/14/17 5:55 PM, Jerome Glisse wrote:
> So pushed an updated hmm-next branch this should fix all issues you had.
> Thought i am not sure about the test in this mail, all i see is that it
> continously spit error messages but it does not hang (i let it run 20min
> or so). Dunno if that is what expected. Let me know if this is still an
> issue and if so what should be the expected output of this test program.
>
> Cheers,
> Jérôme

Thanks, Jerome. The kernel hang indeed seems to be fixed.

Regarding the last issue I reported. It still persists. The test program 
should eventually exit. Instead, it loops indefinitely (sorry, I was not 
clear when I called it 'app-side hang').

This is what's expected. The number of error messages can be random, but 
must be finite; the program must print "OK" at the end:

$ ./run.sh
&&& 1 migrate threads: STARTING
iteration 0
thread 0 is migrating 10000 pages starting from 0x7f6abce79000
migrate_thread_func:87: failed to migrate pages at 0x7f6abce79000 
(migrate.npages (tid 0): 9725 != npages: 10000)
thread 0 is migrating 10000 pages starting from 0x7f6abce79000
thread 0 is migrating 10000 pages starting from 0x7f6abce79000
&&& 1 migrate threads: PASSED
(OK)[./sanity_rmem004] anon migration read test

Thanks,

-- 
Evgeny Baskakov
NVIDIA

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web