Re: [BUG] Slab corruption during XFS writeback under memory pressure

From	Dave Chinner <david@fromorbit.com>
Newsgroups	linux.kernel
Subject	Re: [BUG] Slab corruption during XFS writeback under memory pressure
Date	2016-07-20 01:00 +0200
Message-ID	<rWM9A-1xV-15@gated-at.bofh.it> (permalink)
References	<rVHOF-24a-5@gated-at.bofh.it> <rVHOF-24a-3@gated-at.bofh.it> <rW9UB-2uF-5@gated-at.bofh.it> <rWM9A-1xV-17@gated-at.bofh.it> <rWM9A-1xV-19@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

On Tue, Jul 19, 2016 at 02:22:47PM -0700, Calvin Owens wrote:
> On 07/18/2016 07:05 PM, Calvin Owens wrote:
> >On 07/17/2016 11:02 PM, Dave Chinner wrote:
> >>On Sun, Jul 17, 2016 at 10:00:03AM +1000, Dave Chinner wrote:
> >>>On Fri, Jul 15, 2016 at 05:18:02PM -0700, Calvin Owens wrote:
> >>>>Hello all,
> >>>>
> >>>>I've found a nasty source of slab corruption. Based on seeing similar symptoms
> >>>>on boxes at Facebook, I suspect it's been around since at least 3.10.
> >>>>
> >>>>It only reproduces under memory pressure so far as I can tell: the issue seems
> >>>>to be that XFS reclaims pages from buffers that are still in use by
> >>>>scsi/block. I'm not sure which side the bug lies on, but I've only observed it
> >>>>with XFS.
> >>[....]
> >>>But this indicates that the page is under writeback at this point,
> >>>so that tends to indicate that the above freeing was incorrect.
> >>>
> >>>Hmmm - it's clear we've got direct reclaim involved here, and the
> >>>suspicion of a dirty page that has had it's bufferheads cleared.
> >>>Are there any other warnings in the log from XFS prior to kasan
> >>>throwing the error?
> >>
> >>Can you try the patch below?
> >
> >Thanks for getting this out so quickly :)
> >
> >So far so good: I booted Linus' tree as of this morning and reproduced the ASAN
> >splat. After applying your patch I haven't triggered it.
> >
> >I'm a bit wary since it was hard to trigger reliably in the first place... so I
> >lined up a few dozen boxes to run the test case overnight. I'll confirm in the
> >morning (-0700) they look good.
> 
> All right, my testcase ran 2099 times overnight without triggering anything.
> 
> For the overnight tests, I booted the boxes with "mem=" to artificially limit RAM,
> which makes my repro *much* more reliable (I feel silly for not thinking of that
> in the first place). With that setup, I hit the ASAN splat 21 times in 98 runs on
> vanilla 4.7-rc7. So I'm sold.
> 
> Tested-by: Calvin Owens <calvinowens@fb.com>

Thanks for testing, Calvin. I'll update the patch and get it
reviewed and committed.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Back to linux.kernel | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Re: [BUG] Slab corruption during XFS writeback under memory pressure Dave Chinner <david@fromorbit.com> - 2016-07-17 02:10 +0200
  Re: [BUG] Slab corruption during XFS writeback under memory pressure Dave Chinner <david@fromorbit.com> - 2016-07-18 08:10 +0200
    Re: [BUG] Slab corruption during XFS writeback under memory pressure Dave Chinner <david@fromorbit.com> - 2016-07-20 01:00 +0200

csiph-web