Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1738928 > unrolled thread
| Started by | Will Deacon <will.deacon@arm.com> |
|---|---|
| First post | 2017-09-25 13:00 +0200 |
| Last post | 2017-09-26 16:30 +0200 |
| Articles | 4 — 2 participants |
Back to article view | Back to linux.kernel
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
Re: ARM64: kernel panics in DABT in sys_msync path Will Deacon <will.deacon@arm.com> - 2017-09-25 13:00 +0200
Re: ARM64: kernel panics in DABT in sys_msync path "Ruigrok, Richard" <rruigrok@codeaurora.org> - 2017-09-25 23:00 +0200
Re: ARM64: kernel panics in DABT in sys_msync path Will Deacon <will.deacon@arm.com> - 2017-09-26 12:30 +0200
Re: ARM64: kernel panics in DABT in sys_msync path "Ruigrok, Richard" <rruigrok@codeaurora.org> - 2017-09-26 16:30 +0200
| From | Will Deacon <will.deacon@arm.com> |
|---|---|
| Date | 2017-09-25 13:00 +0200 |
| Subject | Re: ARM64: kernel panics in DABT in sys_msync path |
| Message-ID | <utzhg-5f9-15@gated-at.bofh.it> |
Hi Yury, Thanks for the report. On Mon, Sep 25, 2017 at 12:36:22AM +0300, Yury Norov wrote: > Hi all, > > I found that running with qemu-10 with '-smp 4' option kernel v4.13 and > v4.14-rc1 panics with LTP test rwtest03: > rwtest -N rwtest03 -c -q -i 60s -n 2 -f buffered -s mmread,mmwrite -m random -Dv 10%25000:mm-buff-$$ > [ 2068.307587] Unable to handle kernel paging request at virtual address ffffffffc0000d68 > [ 2068.308195] swapper pgtable: 4k pages, 48-bit VAs, pgd = ffff00000901f000 > [ 2068.308387] [ffffffffc0000d68] *pgd=0000000000000000 > [ 2068.308643] Internal error: Oops: 96000004 [#1] PREEMPT SMP > [ 2068.308865] Modules linked in: > [ 2068.309013] CPU: 0 PID: 9861 Comm: doio Not tainted 4.13.0-00027-g2fdc18baa2ae #196 > [ 2068.309205] Hardware name: linux,dummy-virt (DT) > [ 2068.309331] task: ffff80000300d400 task.stack: ffff80003d28c000 > [ 2068.309728] PC is at check_pte+0x8/0x130 > [ 2068.309848] LR is at page_vma_mapped_walk+0x240/0x498 > [ 2068.309995] pc : [<ffff0000081c5268>] lr : [<ffff0000081c55d0>] pstate: 00000145 > > [...] > > [ 2068.338791] [<ffff0000081c5268>] check_pte+0x8/0x130 > [ 2068.339070] [<ffff0000081c66c0>] page_mkclean_one+0xa0/0x258 > [ 2068.339209] [<ffff0000081c6a70>] rmap_walk_file+0xe8/0x238 > [ 2068.339331] [<ffff0000081c88c8>] rmap_walk+0x48/0x70 > [ 2068.339436] [<ffff0000081c8ae8>] page_mkclean+0x80/0x98 > [ 2068.339592] [<ffff00000819178c>] clear_page_dirty_for_io+0xac/0x298 > [ 2068.339770] [<ffff0000082a36cc>] mpage_submit_page+0x2c/0x90 > [ 2068.340004] [<ffff0000082a3864>] mpage_process_page_bufs+0x134/0x140 > [ 2068.340261] [<ffff0000082a398c>] mpage_prepare_extent_to_map+0x11c/0x270 > [ 2068.340438] [<ffff0000082a9058>] ext4_writepages+0x2f0/0xb30 > [ 2068.340600] [<ffff000008193b78>] do_writepages+0x60/0x90 > [ 2068.340742] [<ffff000008185a44>] __filemap_fdatawrite_range+0xa4/0xf0 > [ 2068.340908] [<ffff0000081861c8>] file_write_and_wait_range+0x50/0xb8 > [ 2068.341071] [<ffff000008299b40>] ext4_sync_file+0x80/0x340 > [ 2068.341222] [<ffff00000823f668>] vfs_fsync_range+0x48/0xc8 > [ 2068.341425] [<ffff0000081c51f4>] SyS_msync+0x1bc/0x228 > [ 2068.341572] [<ffff00000808375c>] el0_svc_naked+0x20/0x24 > > The bug is reproducible for ilp32 and lp64 binaries. For kernel 4.12 > and for all kernels if '-smp 1' is passed to qemu, everything works > fine. If no ideas, I think I'm able bisect it. I tried to reproduce this on hardware, but failed to do so. Our nightly tests are also coming back fine for rwtest03. I just built Qemu v2.10.0 and that also passes the test with -smp 4 for me, so I'm a bit stuck. Could you share: * Your kernel .config * Your QEMU command line * Details of your userspace please? Thanks, Will
[toc] | [next] | [standalone]
| From | "Ruigrok, Richard" <rruigrok@codeaurora.org> |
|---|---|
| Date | 2017-09-25 23:00 +0200 |
| Message-ID | <utIDU-37w-19@gated-at.bofh.it> |
| In reply to | #1738928 |
On 9/25/2017 1:04 PM, Yury Norov wrote: > On Mon, Sep 25, 2017 at 05:02:40PM +0300, Yury Norov wrote: >> Hi Will, >> >>>> The bug is reproducible for ilp32 and lp64 binaries. For kernel 4.12 >>>> and for all kernels if '-smp 1' is passed to qemu, everything works >>>> fine. If no ideas, I think I'm able bisect it. >>> I tried to reproduce this on hardware, but failed to do so. Our nightly >>> tests are also coming back fine for rwtest03. I just built Qemu v2.10.0 >>> and that also passes the test with -smp 4 for me, so I'm a bit stuck. Hi Will, I also found this issue with kernels from 4.11 through 4.13, reproduced on Hardware. In my tests, I found that it reproduces only with 4K page and Transparent Huge Pages. With 64K page I was not able to reproduce. RH also reported it here: https://bugzilla.redhat.com/show_bug.cgi?id=1491504 and Linaro reported on the 4.12 based RPK kernel. https://bugs.linaro.org/show_bug.cgi?id=3191 https://bugs.linaro.org/show_bug.cgi?id=3068. I was able to bisect down to a specific commit: f27176cfc363 mm: convert page_mkclean_one() to use page_vma_mapped_walk() Ran with LTP 20170516 release, rwtest: ./runltp -p -f fs -s rwtest To validate bisecting (good points), I ran 30 iterations. Usually it reproduces in 5-10 iterations. LMK if you have any suggestions for instrumentation, when running with ftrace I could not repro. I can run tests on 4.13 or on 4.11 at the above bisect point. I have not tried any 4.14-rc yet. >> I also see the test passed sometimes. I run it in endless cycle and >> leave for a while. 5-10 iterations are usually enough. >> >>> Could you share: >>> >>> * Your kernel .config >>> * Your QEMU command line >>> * Details of your userspace >> Qemu configure command: >> ./configure --target-list=aarch64-softmmu --enable-fdt --enable-vhost-net --enable-kvm >> >> And run command: >> /home/yury/work/qemu-2.10.0/aarch64-softmmu/qemu-system-aarch64 \ >> -machine virtualization=true -machine gic-version=3 \ >> -machine virt -cpu cortex-a57 -nographic -smp 4 -m 1024 \ >> -global virtio-blk-device.scsi=off -device virtio-scsi-device,id=scsi \ >> -drive file=img/ubuntu-core-14.04.1-core-arm64.img,id=coreimg,cache=unsafe,if=none -device scsi-hd,drive=coreimg \ >> -kernel /home/yury/work/linux/arch/arm64/boot/Image \ >> --append "console=ttyAMA0 root=/dev/sda" \ >> -initrd initrd.img-3.13.0-62-generic \ >> $NETWORK \ >> -redir tcp:2222::22 \ >> -s \ >> $@ >> >> My userspace is Ubuntu 14. I build lp64 tests with default Ubuntu >> toolchain, and ilp32 tests with Linaro cross-toolchain. >> >> The config is attached, and the branch is vanilla 4.13 kernel, or this >> one: >> https://github.com/norov/linux/tree/ilp32-4.13 >> >> Later today I will share the whole qemu environment I use. > https://drive.google.com/file/d/0B07VUB3kjLD8Mm5XN21qTTBfbnc/view > >> Yury > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel -- Qualcomm Datacenter Technologies as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
[toc] | [prev] | [next] | [standalone]
| From | Will Deacon <will.deacon@arm.com> |
|---|---|
| Date | 2017-09-26 12:30 +0200 |
| Message-ID | <utVhM-3uw-23@gated-at.bofh.it> |
| In reply to | #1738928 |
On Mon, Sep 25, 2017 at 01:54:57PM -0600, Ruigrok, Richard wrote: > I also found this issue with kernels from 4.11 through 4.13. In my tests, I > found that it reproduces only with 4K page and Transparent Huge Pages. With 64K > page I was not able to reproduce. RH also reported it here: https:// > bugzilla.redhat.com/show_bug.cgi?id=1491504 Linaro reported on the RPK kernel > (4.12) on Centriq2400 and ThunderX > > > https://bugs.linaro.org/show_bug.cgi?id=3191 > > https://bugs.linaro.org/show_bug.cgi?id=3068. These two aren't the same bug (that's a forward progress issue that we're currently working on). I don't have permission to look at the redhat one, but is it just an RCU stall or actually the Oops reported by Yury? > I was able to bisect down to a specific commit. I think we're chasing two different things here, so not sure I trust the bisect! Will > First bad commit is: > commit f27176cfc363d395eea8dc5c4a26e5d6d7d65eaf > Author: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> > Date: Fri Feb 24 14:57:57 2017 -0800 > > mm: convert page_mkclean_one() to use page_vma_mapped_walk() > > For consistency, it worth converting all page_check_address() to > page_vma_mapped_walk(), so we could drop the former. > > PMD handling here is future-proofing, we don't have users yet. ext4 > with huge pages will be the first. > > I did not use virtualization, simply booting kernel and running the LTP > rwtest: ./runltp -p -f fs -s rwtest > To validate bisecting (good points), I ran 30 iterations. Usually it > reproduces in 5-10 iterations. > > If you have any suggestions for instrumentation I can run tests, we can work > with 4.13 or on 4.11 at the above bisect point. > I have not tried the 4.14-rc's yet.
[toc] | [prev] | [next] | [standalone]
| From | "Ruigrok, Richard" <rruigrok@codeaurora.org> |
|---|---|
| Date | 2017-09-26 16:30 +0200 |
| Message-ID | <utZ22-5UA-23@gated-at.bofh.it> |
| In reply to | #1739771 |
On 9/26/2017 4:23 AM, Will Deacon wrote:
> On Mon, Sep 25, 2017 at 01:54:57PM -0600, Ruigrok, Richard wrote:
>> I also found this issue with kernels from 4.11 through 4.13. In my tests, I
>> found that it reproduces only with 4K page and Transparent Huge Pages. With 64K
>> page I was not able to reproduce. RH also reported it here: https://
>> bugzilla.redhat.com/show_bug.cgi?id=1491504 Linaro reported on the RPK kernel
>> (4.12) on Centriq2400 and ThunderX
>>
>>
>> https://bugs.linaro.org/show_bug.cgi?id=3191
>>
>> https://bugs.linaro.org/show_bug.cgi?id=3068.
> These two aren't the same bug (that's a forward progress issue that we're
> currently working on). I don't have permission to look at the redhat one,
> but is it just an RCU stall or actually the Oops reported by Yury?
>
>> I was able to bisect down to a specific commit.
> I think we're chasing two different things here, so not sure I trust the
> bisect!
>
> Will
The RCU stall is side effect. The issue I'm seeing has the same stack trace and same stimulus (rwtest). Following are the details.
I agree the bisect needs to be verified. Yury could you test commits before and at the bisect point I provided. I did extensive test on our platform and bisect converged consistently to the same commit.
Details:
When running ARM64 kernel configured with THP enabled:
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
And 4k page (CONFIG_ARM64_4K_PAGES=y)
Running ltp release 20170516-182-g738dbdb rwtest: runltp -p -f fs -s rwtest
An unhandled page fault occurs in the mm code, when PC hits line at mm/page_vma_mapped.c
http://elixir.free-electrons.com/linux/v4.13/source/mm/page_vma_mapped.c#L163
When an invalid pvmw pointer is passed to check_pte, in addition to the unhandled page fault, the entire system is brought down since the core on which the page fault occurs halts while holding the spinlock: spin_lock(pvmw->ptl);
From <http://elixir.free-electrons.com/linux/v4.13/source/mm/page_vma_mapped.c#L163>
All other cores will show: NMI watchdog: BUG: soft lockup - CPU#<n> stuck for 22s! [doio:4152]
list *( 0xffff0000081b9210 +0x70)
(gdb) list *( 0xffff0000081b9210 +0x70)
0xffff0000081b9280 is in page_mkclean_one (mm/rmap.c:1028).
1023 .address = address,
1024 .flags = PVMW_SYNC,
1025 };
1026 int *cleaned = arg;
1027
1028 while (page_vma_mapped_walk(&pvmw)) {
1029 int ret = 0;
1030 address = pvmw.address;
1031 if (pvmw.pte) {
1032 pte_t entry;
(gdb)
Dump of assembler code for function check_pte:
0xffff0000081b80c0 <+0>: ldr w1, [x0,#48]
list *(0xffff0000081b80c0 + 0x68)
(gdb) list *(0xffff0000081b80c0 + 0x68)
0xffff0000081b8128 is in check_pte (mm/page_vma_mapped.c:63).
58 return false;
59 #else
60 WARN_ON_ONCE(1);
61 #endif
62 } else {
63 if (!pte_present(*pvmw->pte))
64 return false;
65
66 /* THP can be referenced by any subpage */
67 if (pte_page(*pvmw->pte) - pvmw->page >=
[ 544.799399] Unable to handle kernel paging request at virtual address ffff800000000c10
[ 544.806371] pgd = ffff8007d4d7b000
[ 544.809753] [ffff800000000c10] *pgd=0000000000000000
[ 544.814695] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[ 544.820248] Modules linked in:
[ 544.823287] CPU: 2 PID: 4153 Comm: doio Not tainted 4.10.0-dev-0907-t64-09623-g726c7c0 #93
[ 544.831526] Hardware name: Qualcomm Qualcomm Centriq(TM) 2400 Development Platform/ABW|SYS|CVR,1DPC|V3 , BIOS XBL.DF.2.0.R1-00542 QDF2400_REL CR
[ 544.845328] task: ffff8007d8428d00 task.stack: ffff8007db4ac000
[ 544.851248] PC is at check_pte+0x68/0x150
[ 544.855231] LR is at page_vma_mapped_walk+0x260/0x3d8
[ 544.860259] pc : [<ffff0000081b8128>] lr : [<ffff0000081b8470>] pstate: 00400145
[ 544.867637] sp : ffff8007db4af8a0
[ 544.870942] x29: ffff8007db4af8a0 x28: 0000000000000714
[ 544.876231] x27: 0088000000000000 x26: ff77ffffffffffff
[ 544.881526] x25: 0400000000000001 x24: 0040000000000041
[ 544.886821] x23: ffff8007d77f7000 x22: ffff8007db4afa34
[ 544.892116] x21: ffff000009276000 x20: ffff7e001f292600
[ 544.897411] x19: ffff8007db4af958 x18: 0000000000000a03
[ 544.902706] x17: 0000ffff945fb1a0 x16: ffff0000081b7ee8
[ 544.908001] x15: ffff8007bd6a6b48 x14: 0000000000000040
[ 544.913297] x13: 0000000000000000 x12: 0000000000000002
[ 544.918592] x11: 0000000000000230 x10: 0000000000001200
[ 544.923887] x9 : ffff7e001f2925c0 x8 : 0000000000001200
[ 544.929182] x7 : 0000000000000001 x6 : 0000000000000c35
[ 544.934477] x5 : 0000000000000001 x4 : 0000000000000182
[ 544.939772] x3 : 0400000000000001 x2 : ffff800000000c10
[ 544.945067] x1 : 0000000000000000 x0 : ffff8007db4af958
[ 545.425022] Call trace: [10/44993]
[ 545.427453] Exception stack(0xffff8007db4af6d0 to 0xffff8007db4af800)
[ 545.433870] f6c0: ffff8007db4af958 0001000000000000
[ 545.441683] f6e0: ffff8007db4af8a0 ffff0000081b8128 ffff8007db4af710 ffff0000081dc514
[ 545.449495] f700: 0000000000000000 ffff0000091ef000 ffff8007db4af770 ffff0000087f0444
[ 545.457308] f720: ffff8007d9f1e148 ffff0000095ad000 ffff8007d80eb000 0000000001011200
[ 545.465120] f740: ffff8007db4af7a0 ffff00000817cf40 0000000000000000 ffff8007d8e7f700
[ 545.472933] f760: 0000000001091220 ffff0000080fd998 ffff8007db4af958 0000000000000000
[ 545.480745] f780: ffff800000000c10 0400000000000001 0000000000000182 0000000000000001
[ 545.488558] f7a0: 0000000000000c35 0000000000000001 0000000000001200 ffff7e001f2925c0
[ 545.496370] f7c0: 0000000000001200 0000000000000230 0000000000000002 0000000000000000
[ 545.504183] f7e0: 0000000000000040 ffff8007bd6a6b48 ffff0000081b7ee8 0000ffff945fb1a0
[ 545.512008] [<ffff0000081b8128>] check_pte+0x68/0x150
[ 545.517043] [<ffff0000081b9280>] page_mkclean_one+0x70/0x1a0
[ 545.522672] [<ffff0000081b94dc>] rmap_walk_file+0xe4/0x290
[ 545.528141] [<ffff0000081bb788>] rmap_walk+0x48/0x70
[ 545.533089] [<ffff0000081bb9a8>] page_mkclean+0x88/0xa0
[ 545.538313] [<ffff0000081866dc>] clear_page_dirty_for_io+0x9c/0x200
[ 545.544564] [<ffff000008280a20>] mpage_submit_page+0x48/0x98
[ 545.550190] [<ffff000008280bb8>] mpage_process_page_bufs+0x148/0x158
[ 545.556526] [<ffff000008280d0c>] mpage_prepare_extent_to_map+0x144/0x270
[ 545.563217] [<ffff000008284f20>] ext4_writepages+0x3b0/0xa00
[ 545.568853] [<ffff000008188ccc>] do_writepages+0x24/0x48
[ 545.574161] [<ffff00000817b454>] __filemap_fdatawrite_range+0x9c/0xe8
[ 545.580571] [<ffff00000817b5b4>] filemap_write_and_wait_range+0x2c/0x88
[ 545.587175] [<ffff00000827c540>] ext4_sync_file+0x58/0x300
[ 545.592652] [<ffff00000822b46c>] vfs_fsync_range+0x44/0xc0
[ 545.598107] [<ffff0000081b806c>] SyS_msync+0x184/0x1d8
[ 545.603242] [<ffff000008082f30>] el0_svc_naked+0x24/0x28
[ 545.608530] Code: f9401002 d2800023 f2e08003 52800001 (f9400042)
[ 545.614630] ---[ end trace 065a200dac27fe87 ]---
[ 545.619213] note: doio[4153] exited with preempt_count 1
[ 569.734898] NMI watchdog: BUG: soft lockup - CPU#27 stuck for 22s! [doio:4152]
[ 569.741155] Modules linked in:
[ 569.744193]
[ 569.745671] CPU: 27 PID: 4152 Comm: doio Tainted: G D 4.10.0-dev-0907-t64-09623-g726c7c0 #93
[ 569.755218] Hardware name: Qualcomm Qualcomm Centriq(TM) 2400 Development Platform/ABW|SYS|CVR,1DPC|V3 , BIOS XBL.DF.2.0.R1-00542 QDF2400_REL CR
[ 569.769020] task: ffff8007d842ce00 task.stack: ffff8007d8280000
[ 569.774938] PC is at _raw_spin_lock+0x34/0x48
[ 569.779279] LR is at alloc_set_pte+0x438/0x560
Thanks,
Richard.
>> First bad commit is:
>> commit f27176cfc363d395eea8dc5c4a26e5d6d7d65eaf
>> Author: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
>> Date: Fri Feb 24 14:57:57 2017 -0800
>>
>> mm: convert page_mkclean_one() to use page_vma_mapped_walk()
>>
>> For consistency, it worth converting all page_check_address() to
>> page_vma_mapped_walk(), so we could drop the former.
>>
>> PMD handling here is future-proofing, we don't have users yet. ext4
>> with huge pages will be the first.
>>
>> I did not use virtualization, simply booting kernel and running the LTP
>> rwtest: ./runltp -p -f fs -s rwtest
>> To validate bisecting (good points), I ran 30 iterations. Usually it
>> reproduces in 5-10 iterations.
>>
>> If you have any suggestions for instrumentation I can run tests, we can work
>> with 4.13 or on 4.11 at the above bisect point.
>> I have not tried the 4.14-rc's yet.
--
Qualcomm Datacenter Technologies as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the
Code Aurora Forum, a Linux Foundation Collaborative Project.
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web