Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1292453 > unrolled thread
| Started by | Michal Hocko <mhocko@kernel.org> |
|---|---|
| First post | 2015-12-15 19:40 +0100 |
| Last post | 2015-12-25 12:50 +0100 |
| Articles | 3 on this page of 23 — 6 participants |
Back to article view | Back to linux.kernel
[PATCH 1/2] mm, oom: introduce oom reaper Michal Hocko <mhocko@kernel.org> - 2015-12-15 19:40 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Andrew Morton <akpm@linux-foundation.org> - 2015-12-17 02:00 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Michal Hocko <mhocko@kernel.org> - 2015-12-17 14:10 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Linus Torvalds <torvalds@linux-foundation.org> - 2015-12-17 21:00 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Andrew Morton <akpm@linux-foundation.org> - 2015-12-17 21:10 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Michal Hocko <mhocko@kernel.org> - 2015-12-18 13:00 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Andrew Morton <akpm@linux-foundation.org> - 2015-12-18 22:20 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Michal Hocko <mhocko@kernel.org> - 2015-12-21 09:40 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Andrew Morton <akpm@linux-foundation.org> - 2015-12-17 22:20 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Michal Hocko <mhocko@kernel.org> - 2015-12-18 13:20 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> - 2015-12-18 13:20 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> - 2015-12-20 08:20 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Andrew Morton <akpm@linux-foundation.org> - 2015-12-18 01:20 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Michal Hocko <mhocko@kernel.org> - 2015-12-18 12:50 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Paul Gortmaker <paul.gortmaker@windriver.com> - 2015-12-21 21:40 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Ross Zwisler <zwisler@gmail.com> - 2015-12-24 00:10 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Michal Hocko <mhocko@kernel.org> - 2015-12-24 10:50 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> - 2015-12-24 12:10 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Ross Zwisler <zwisler@gmail.com> - 2015-12-24 22:00 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Michal Hocko <mhocko@kernel.org> - 2015-12-25 12:50 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Ross Zwisler <zwisler@gmail.com> - 2015-12-24 21:50 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Michal Hocko <mhocko@kernel.org> - 2015-12-25 12:40 +0100
Re: [PATCH 1/2] mm, oom: introduce oom reaper Michal Hocko <mhocko@kernel.org> - 2015-12-25 12:50 +0100
Page 2 of 2 — ← Prev page 1 [2]
| From | Ross Zwisler <zwisler@gmail.com> |
|---|---|
| Date | 2015-12-24 21:50 +0100 |
| Message-ID | <qJkZI-2WE-5@gated-at.bofh.it> |
| In reply to | #1297798 |
On Thu, Dec 24, 2015 at 2:47 AM, Michal Hocko <mhocko@kernel.org> wrote:
> On Wed 23-12-15 16:00:09, Ross Zwisler wrote:
> [...]
>> While running xfstests on next-20151223 I hit a pair of kernel BUGs
>> that bisected to this commit:
>>
>> 1eb3a80d8239 ("mm, oom: introduce oom reaper")
>
> Thank you for the report and the bisection.
>
>> Here is a BUG produced by generic/029 when run against XFS:
>>
>> [ 235.751723] ------------[ cut here ]------------
>> [ 235.752194] kernel BUG at mm/filemap.c:208!
>
> This is VM_BUG_ON_PAGE(page_mapped(page), page), right? Could you attach
> the full kernel log? It all smells like a race when OOM reaper tears
> down the mapping and there is a truncate still in progress. But hitting
> the BUG_ON just because of that doesn't make much sense to me. OOM
> reaper is essentially MADV_DONTNEED. I have to think about this some
> more, though, but I am in a holiday mode until early next year so please
> bear with me.
The two stack traces were gathered with next-20151223, so the line numbers
may have moved around a bit when compared to the actual "mm, oom: introduce
oom reaper" commit.
> [...]
>> [ 235.765638] Call Trace:
>> [ 235.765903] [<ffffffff811c8493>] delete_from_page_cache+0x63/0xd0
>> [ 235.766513] [<ffffffff811dc3e5>] truncate_inode_page+0xa5/0x120
>> [ 235.767088] [<ffffffff811dc648>] truncate_inode_pages_range+0x1a8/0x7f0
>> [ 235.767725] [<ffffffff81021459>] ? sched_clock+0x9/0x10
>> [ 235.768239] [<ffffffff810db37c>] ? local_clock+0x1c/0x20
>> [ 235.768779] [<ffffffff811feba4>] ? unmap_mapping_range+0x64/0x130
>> [ 235.769385] [<ffffffff811febb4>] ? unmap_mapping_range+0x74/0x130
>> [ 235.770010] [<ffffffff810f5c3f>] ? up_write+0x1f/0x40
>> [ 235.770501] [<ffffffff811febb4>] ? unmap_mapping_range+0x74/0x130
>> [ 235.771092] [<ffffffff811dcd58>] truncate_pagecache+0x48/0x70
>> [ 235.771646] [<ffffffff811dcdb2>] truncate_setsize+0x32/0x40
>> [ 235.772276] [<ffffffff8148e972>] xfs_setattr_size+0x232/0x470
>> [ 235.772839] [<ffffffff8148ec64>] xfs_vn_setattr+0xb4/0xc0
>> [ 235.773369] [<ffffffff8127af87>] notify_change+0x237/0x350
>> [ 235.773945] [<ffffffff81257c87>] do_truncate+0x77/0xc0
>> [ 235.774446] [<ffffffff8125800f>] do_sys_ftruncate.constprop.15+0xef/0x150
>> [ 235.775156] [<ffffffff812580ae>] SyS_ftruncate+0xe/0x10
>> [ 235.775650] [<ffffffff81a527b2>] entry_SYSCALL_64_fastpath+0x12/0x76
>> [ 235.776257] Code: 5f 5d c3 48 8b 43 20 48 8d 78 ff a8 01 48 0f 44
>> fb 8b 47 48 85 c0 0f 88 2b 01 00 00 48 c7 c6 a8 57 f0 81 48 89 df e8
>> fa 1a 03 00 <0f> 0b 4c 89 ce 44 89 fa 4c 89 e7 4c 89 45 b0 4c 89 4d b8
>> e8 32
>> [ 235.778695] RIP [<ffffffff811c81f6>] __delete_from_page_cache+0x206/0x440
>> [ 235.779350] RSP <ffff8800bab83b60>
>> [ 235.779694] ---[ end trace fac9dd65c4cdd828 ]---
>>
>> And a different BUG produced by generic/095, also with XFS:
>>
>> [ 609.398897] ------------[ cut here ]------------
>> [ 609.399843] kernel BUG at mm/truncate.c:629!
>
> Hmm, I do not see any BUG_ON at this line. But there is
> BUG_ON(page_mapped(page)) at line 620.
Ditto - check out next-20151223 for real line numbers.
>> [ 609.400666] invalid opcode: 0000 [#1] SMP
>> [ 609.401512] Modules linked in: nd_pmem nd_btt nd_e820 libnvdimm
>> [ 609.402719] CPU: 4 PID: 26782 Comm: fio Tainted: G W
>
> There was a warning before this triggered. The full kernel log would be
> helpful as well.
Sure, I can gather full kernel logs, but it'll probably after the new year.
> [...]
>> [ 609.425325] Call Trace:
>> [ 609.425797] [<ffffffff811dc307>] invalidate_inode_pages2+0x17/0x20
>> [ 609.426971] [<ffffffff81482167>] xfs_file_read_iter+0x297/0x300
>> [ 609.428097] [<ffffffff81259ac9>] __vfs_read+0xc9/0x100
>> [ 609.429073] [<ffffffff8125a319>] vfs_read+0x89/0x130
>> [ 609.430010] [<ffffffff8125b418>] SyS_read+0x58/0xd0
>> [ 609.430943] [<ffffffff81a527b2>] entry_SYSCALL_64_fastpath+0x12/0x76
>> [ 609.432139] Code: 85 d8 fe ff ff 01 00 00 00 f6 c4 40 0f 84 59 ff
>> ff ff 49 8b 47 20 48 8d 78 ff a8 01 49 0f 44 ff 8b 47 48 85 c0 0f 88
>> bd 01 00 00 <0f> 0b 4d 3b 67 08 0f 85 70 ff ff ff 49 f7 07 00 18 00 00
>> 74 15
> [...]
>> My test setup is a qemu guest machine with a pair of 4 GiB PMEM
>> ramdisk test devices, one for the xfstest test disk and one for the
>> scratch disk.
>
> Is this just a plain ramdisk device or it needs a special configuration?
Just a plain PMEM ram disk with DAX turned off. Configuration instructions
for PMEM can be found here:
https://nvdimm.wiki.kernel.org/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Michal Hocko <mhocko@kernel.org> |
|---|---|
| Date | 2015-12-25 12:40 +0100 |
| Message-ID | <qJySZ-2Wh-3@gated-at.bofh.it> |
| In reply to | #1298042 |
On Thu 24-12-15 13:44:03, Ross Zwisler wrote:
> On Thu, Dec 24, 2015 at 2:47 AM, Michal Hocko <mhocko@kernel.org> wrote:
> > On Wed 23-12-15 16:00:09, Ross Zwisler wrote:
> > [...]
> >> While running xfstests on next-20151223 I hit a pair of kernel BUGs
> >> that bisected to this commit:
> >>
> >> 1eb3a80d8239 ("mm, oom: introduce oom reaper")
> >
> > Thank you for the report and the bisection.
> >
> >> Here is a BUG produced by generic/029 when run against XFS:
> >>
> >> [ 235.751723] ------------[ cut here ]------------
> >> [ 235.752194] kernel BUG at mm/filemap.c:208!
> >
> > This is VM_BUG_ON_PAGE(page_mapped(page), page), right? Could you attach
> > the full kernel log? It all smells like a race when OOM reaper tears
> > down the mapping and there is a truncate still in progress. But hitting
> > the BUG_ON just because of that doesn't make much sense to me. OOM
> > reaper is essentially MADV_DONTNEED. I have to think about this some
> > more, though, but I am in a holiday mode until early next year so please
> > bear with me.
>
> The two stack traces were gathered with next-20151223, so the line numbers
> may have moved around a bit when compared to the actual "mm, oom: introduce
> oom reaper" commit.
I was looking at the same next tree, I believe
$ git describe
next-20151223
[...]
> > There was a warning before this triggered. The full kernel log would be
> > helpful as well.
>
> Sure, I can gather full kernel logs, but it'll probably after the new year.
OK, I will wait for the logs. It is really interesting to see what was
the timing between OOM killer invocation and this trace.
> > [...]
> >> [ 609.425325] Call Trace:
> >> [ 609.425797] [<ffffffff811dc307>] invalidate_inode_pages2+0x17/0x20
> >> [ 609.426971] [<ffffffff81482167>] xfs_file_read_iter+0x297/0x300
> >> [ 609.428097] [<ffffffff81259ac9>] __vfs_read+0xc9/0x100
> >> [ 609.429073] [<ffffffff8125a319>] vfs_read+0x89/0x130
> >> [ 609.430010] [<ffffffff8125b418>] SyS_read+0x58/0xd0
> >> [ 609.430943] [<ffffffff81a527b2>] entry_SYSCALL_64_fastpath+0x12/0x76
> >> [ 609.432139] Code: 85 d8 fe ff ff 01 00 00 00 f6 c4 40 0f 84 59 ff
> >> ff ff 49 8b 47 20 48 8d 78 ff a8 01 49 0f 44 ff 8b 47 48 85 c0 0f 88
> >> bd 01 00 00 <0f> 0b 4d 3b 67 08 0f 85 70 ff ff ff 49 f7 07 00 18 00 00
> >> 74 15
> > [...]
> >> My test setup is a qemu guest machine with a pair of 4 GiB PMEM
> >> ramdisk test devices, one for the xfstest test disk and one for the
> >> scratch disk.
> >
> > Is this just a plain ramdisk device or it needs a special configuration?
>
> Just a plain PMEM ram disk with DAX turned off. Configuration instructions
> for PMEM can be found here:
>
> https://nvdimm.wiki.kernel.org/
Thanks I will try to reproduce early next year. But so far I think this
is just a general issue of MADV_DONTNEED vs. truncate and oom_reaper is
just lucky to trigger it. There shouldn't be anything oom_reaper
specific here. Maybe there is some additional locking missing?
--
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Michal Hocko <mhocko@kernel.org> |
|---|---|
| Date | 2015-12-25 12:50 +0100 |
| Message-ID | <qJz2G-30l-3@gated-at.bofh.it> |
| In reply to | #1298103 |
On Fri 25-12-15 12:35:37, Michal Hocko wrote: [...] > Thanks I will try to reproduce early next year. But so far I think this > is just a general issue of MADV_DONTNEED vs. truncate and oom_reaper is > just lucky to trigger it. There shouldn't be anything oom_reaper > specific here. Maybe there is some additional locking missing? Hmm, scratch that. I think Tetsuo has nailed it. It seems like the missing initialization of details structure during unmap is the culprit. So there most probably was on OOM killing invoked. It is just a side effect of the patch and missing http://marc.info/?l=linux-mm&m=145068666428057 follow up fix. -- Michal Hocko SUSE Labs -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [standalone]
Page 2 of 2 — ← Prev page 1 [2]
Back to top | Article view | linux.kernel
csiph-web