Re: [4.10, panic, regression] iscsi: null pointer deref at iscsi_tcp_segment_done+0x20d/0x2e0

From	Johannes Weiner <hannes@cmpxchg.org>
Newsgroups	linux.kernel
Subject	Re: [4.10, panic, regression] iscsi: null pointer deref at iscsi_tcp_segment_done+0x20d/0x2e0
Date	2016-12-23 08:40 +0100
Message-ID	<sRsCl-3UM-3@gated-at.bofh.it> (permalink)
References	(4 earlier) <sQYuB-1qn-21@gated-at.bofh.it> <sQZh0-1Xw-21@gated-at.bofh.it> <sR3Xj-51C-3@gated-at.bofh.it> <sR4qm-5dV-27@gated-at.bofh.it> <sRi9Y-5GP-25@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

On Thu, Dec 22, 2016 at 12:22:27PM -0800, Hugh Dickins wrote:
> On Wed, 21 Dec 2016, Linus Torvalds wrote:
> > On Wed, Dec 21, 2016 at 9:13 PM, Dave Chinner <david@fromorbit.com> wrote:
> > > I unmounted the fs, mkfs'd it again, ran the
> > > workload again and about a minute in this fired:
> > >
> > > [628867.607417] ------------[ cut here ]------------
> > > [628867.608603] WARNING: CPU: 2 PID: 16925 at mm/workingset.c:461 shadow_lru_isolate+0x171/0x220
> > 
> > Well, part of the changes during the merge window were the shadow
> > entry tracking changes that came in through Andrew's tree. Adding
> > Johannes Weiner to the participants.
> > 
> > > Now, this workload does not touch the page cache at all - it's
> > > entirely an XFS metadata workload, so it should not really be
> > > affecting the working set code.
> > 
> > Well, I suspect that anything that creates memory pressure will end up
> > triggering the working set code, so ..
> > 
> > That said, obviously memory corruption could be involved and result in
> > random issues too, but I wouldn't really expect that in this code.
> > 
> > It would probably be really useful to get more data points - is the
> > problem reliably in this area, or is it going to be random and all
> > over the place.
> 
> Data point: kswapd got WARNING on mm/workingset.c:457 in shadow_lru_isolate,
> soon followed by NULL pointer deref in list_lru_isolate, one time when
> I tried out Sunday's git tree.  Not seen since, I haven't had time to
> investigate, just set it aside as something to worry about if it happens
> again.  But it looks like shadow_lru_isolate() has issues beyond Dave's
> case (I've no XFS and no iscsi), suspect unrelated to his other problems.

This seems consistent with what Dave observed: we encounter regular
pages in radix tree nodes on the shadow LRU that should only contain
nodes full of exceptional shadow entries. It could be an issue in the
new slot replacement code and the node tracking callback.

Looking...

Thread

[4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Dave Chinner <david@fromorbit.com> - 2016-12-14 23:30 +0100
  Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Dave Chinner <david@fromorbit.com> - 2016-12-14 23:40 +0100
    Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Chris Leech <cleech@redhat.com> - 2016-12-16 20:00 +0100
      Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Dave Chinner <david@fromorbit.com> - 2016-12-21 23:30 +0100
        Re: [4.10, panic, regression] iscsi: null pointer deref at iscsi_tcp_segment_done+0x20d/0x2e0 Linus Torvalds <torvalds@linux-foundation.org> - 2016-12-22 00:30 +0100
          Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Chris Leech <cleech@redhat.com> - 2016-12-22 01:20 +0100
            Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Dave Chinner <david@fromorbit.com> - 2016-12-22 06:20 +0100
              Re: [4.10, panic, regression] iscsi: null pointer deref at iscsi_tcp_segment_done+0x20d/0x2e0 Linus Torvalds <torvalds@linux-foundation.org> - 2016-12-22 06:50 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Dave Chinner <david@fromorbit.com> - 2016-12-22 08:00 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Chris Leech <cleech@redhat.com> - 2016-12-22 20:00 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at iscsi_tcp_segment_done+0x20d/0x2e0 Ming Lei <tom.leiming@gmail.com> - 2016-12-23 01:00 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Chris Leech <cleech@redhat.com> - 2016-12-23 01:10 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at         iscsi_tcp_segment_done+0x20d/0x2e0 Christoph Hellwig <hch@lst.de> - 2016-12-23 11:10 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at iscsi_tcp_segment_done+0x20d/0x2e0 Linus Torvalds <torvalds@linux-foundation.org> - 2016-12-23 20:50 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at         iscsi_tcp_segment_done+0x20d/0x2e0 Christoph Hellwig <hch@lst.de> - 2016-12-24 10:50 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at         iscsi_tcp_segment_done+0x20d/0x2e0 Christoph Hellwig <hch@lst.de> - 2016-12-24 11:10 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at         iscsi_tcp_segment_done+0x20d/0x2e0 Christoph Hellwig <hch@lst.de> - 2016-12-24 14:20 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Hannes Reinecke <hare@suse.de> - 2016-12-24 14:20 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Hugh Dickins <hughd@google.com> - 2016-12-22 21:30 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Johannes Weiner <hannes@cmpxchg.org> - 2016-12-23 08:40 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Johannes Weiner <hannes@cmpxchg.org> - 2016-12-23 09:40 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Johannes Weiner <hannes@cmpxchg.org> - 2017-01-02 22:20 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Jan Kara <jack@suse.cz> - 2017-01-03 13:30 +0100
              Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Dave Chinner <david@fromorbit.com> - 2016-12-22 07:30 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at iscsi_tcp_segment_done+0x20d/0x2e0 Linus Torvalds <torvalds@linux-foundation.org> - 2016-12-22 18:30 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Thomas Gleixner <tglx@linutronix.de> - 2016-12-22 21:30 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Dave Chinner <david@fromorbit.com> - 2016-12-22 21:50 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Dave Chinner <david@fromorbit.com> - 2016-12-22 22:10 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at iscsi_tcp_segment_done+0x20d/0x2e0 Linus Torvalds <torvalds@linux-foundation.org> - 2016-12-22 22:20 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Dave Chinner <david@fromorbit.com> - 2016-12-22 23:20 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Dave Chinner <david@fromorbit.com> - 2016-12-22 23:40 +0100
                Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Dave Chinner <david@fromorbit.com> - 2016-12-23 05:00 +0100
          Re: [4.10, panic, regression] iscsi: null pointer deref at         iscsi_tcp_segment_done+0x20d/0x2e0 Christoph Hellwig <hch@lst.de> - 2016-12-22 07:20 +0100
            Re: [4.10, panic, regression] iscsi: null pointer deref at  iscsi_tcp_segment_done+0x20d/0x2e0 Dave Chinner <david@fromorbit.com> - 2016-12-22 07:40 +0100
              Re: [4.10, panic, regression] iscsi: null pointer deref at         iscsi_tcp_segment_done+0x20d/0x2e0 Christoph Hellwig <hch@lst.de> - 2016-12-22 07:40 +0100

csiph-web