Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1576766 > unrolled thread

Re: net: use-after-free in tw_timer_handler

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: net: use-after-free in tw_timer_handler Dmitry Vyukov <dvyukov@google.com> - 2017-02-08 18:40 +0100
    Re: net: use-after-free in tw_timer_handler Eric Dumazet <eric.dumazet@gmail.com> - 2017-02-08 19:40 +0100
      Re: net: use-after-free in tw_timer_handler Dmitry Vyukov <dvyukov@google.com> - 2017-02-08 20:00 +0100
        Re: net: use-after-free in tw_timer_handler Eric Dumazet <eric.dumazet@gmail.com> - 2017-02-08 20:20 +0100
          Re: net: use-after-free in tw_timer_handler Dmitry Vyukov <dvyukov@google.com> - 2017-02-08 20:50 +0100

#1576766 — Re: net: use-after-free in tw_timer_handler

On Tue, Jan 24, 2017 at 4:52 PM, Eric Dumazet <edumazet@google.com> wrote:
> On Tue, Jan 24, 2017 at 7:06 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
>>>
>>> This code was changed a long time ago :
>>>
>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=ed2e923945892a8372ab70d2f61d364b0b6d9054
>>>
>>> So I suspect a recent patch broke the logic.
>>>
>>> You might start a bisection :
>>>
>>> I would check if 4.7 and 4.8 trigger the issue you noticed.
>>
>>
>> It happens with too low rate for bisecting (few times per day). I
>> could add some additional checks into code, but I don't know what
>> checks could be useful.
>
> If you can not tell if 4.7 and/or 4.8 have the problem, I am not sure
> we are able to help.


There are also chances that the problem is older.

Looking at the code, this part of inet_twsk_purge looks fishy:

285                         if (unlikely((tw->tw_family != family) ||
286                                      atomic_read(&twsk_net(tw)->count))) {

It uses net->count == 0 check to find the right sockets. But what if
there are several nets with count == 0 in flight, can't there be
several inet_twsk_purge calls running concurrently freeing each other
sockets? If so it looks like inet_twsk_purge can call
inet_twsk_deschedule_put twice for a socket. Namely, two calls for
different nets discover the socket, check that net->count==0 and both
call inet_twsk_deschedule_put. Shouldn't we just give inet_twsk_purge
net that it needs to purge?

The second issue that I noticed is that tw_refcnt is set to 4 _after_
we schedule the timer. The timer will probably won't fire before we
set tw_refcnt, but if it somehow does it will corrupt the ref count. I
don't think that it's what I am seeing, though. More likely it's the
first issues (if it's real).

Does it make any sense?

[toc] | [next] | [standalone]

#1576812

On Wed, 2017-02-08 at 18:36 +0100, Dmitry Vyukov wrote:
> On Tue, Jan 24, 2017 at 4:52 PM, Eric Dumazet <edumazet@google.com> wrote:
> > On Tue, Jan 24, 2017 at 7:06 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
> >>>
> >>> This code was changed a long time ago :
> >>>
> >>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=ed2e923945892a8372ab70d2f61d364b0b6d9054
> >>>
> >>> So I suspect a recent patch broke the logic.
> >>>
> >>> You might start a bisection :
> >>>
> >>> I would check if 4.7 and 4.8 trigger the issue you noticed.
> >>
> >>
> >> It happens with too low rate for bisecting (few times per day). I
> >> could add some additional checks into code, but I don't know what
> >> checks could be useful.
> >
> > If you can not tell if 4.7 and/or 4.8 have the problem, I am not sure
> > we are able to help.
> 
> 
> There are also chances that the problem is older.
> 
> Looking at the code, this part of inet_twsk_purge looks fishy:
> 
> 285                         if (unlikely((tw->tw_family != family) ||
> 286                                      atomic_read(&twsk_net(tw)->count))) {
> 
> It uses net->count == 0 check to find the right sockets. But what if
> there are several nets with count == 0 in flight, can't there be
> several inet_twsk_purge calls running concurrently freeing each other
> sockets? If so it looks like inet_twsk_purge can call
> inet_twsk_deschedule_put twice for a socket. Namely, two calls for
> different nets discover the socket, check that net->count==0 and both
> call inet_twsk_deschedule_put. Shouldn't we just give inet_twsk_purge
> net that it needs to purge?

Yes, atomic_read() is not a proper sync point.

> The second issue that I noticed is that tw_refcnt is set to 4 _after_
> we schedule the timer. The timer will probably won't fire before we
> set tw_refcnt, but if it somehow does it will corrupt the ref count. I
> don't think that it's what I am seeing, though. More likely it's the
> first issues (if it's real).
> 

Timer is pinned, it cannot fire under us on this cpu.

[toc] | [prev] | [next] | [standalone]

#1576885

On Wed, Feb 8, 2017 at 6:58 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Wed, 2017-02-08 at 18:36 +0100, Dmitry Vyukov wrote:
>> On Tue, Jan 24, 2017 at 4:52 PM, Eric Dumazet <edumazet@google.com> wrote:
>> > On Tue, Jan 24, 2017 at 7:06 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
>> >>>
>> >>> This code was changed a long time ago :
>> >>>
>> >>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=ed2e923945892a8372ab70d2f61d364b0b6d9054
>> >>>
>> >>> So I suspect a recent patch broke the logic.
>> >>>
>> >>> You might start a bisection :
>> >>>
>> >>> I would check if 4.7 and 4.8 trigger the issue you noticed.
>> >>
>> >>
>> >> It happens with too low rate for bisecting (few times per day). I
>> >> could add some additional checks into code, but I don't know what
>> >> checks could be useful.
>> >
>> > If you can not tell if 4.7 and/or 4.8 have the problem, I am not sure
>> > we are able to help.
>>
>>
>> There are also chances that the problem is older.
>>
>> Looking at the code, this part of inet_twsk_purge looks fishy:
>>
>> 285                         if (unlikely((tw->tw_family != family) ||
>> 286                                      atomic_read(&twsk_net(tw)->count))) {
>>
>> It uses net->count == 0 check to find the right sockets. But what if
>> there are several nets with count == 0 in flight, can't there be
>> several inet_twsk_purge calls running concurrently freeing each other
>> sockets? If so it looks like inet_twsk_purge can call
>> inet_twsk_deschedule_put twice for a socket. Namely, two calls for
>> different nets discover the socket, check that net->count==0 and both
>> call inet_twsk_deschedule_put. Shouldn't we just give inet_twsk_purge
>> net that it needs to purge?
>
> Yes, atomic_read() is not a proper sync point.

Do you mean that it does not include read barrier?
I more mean that we can call inet_twsk_deschedule_put twice for the same socket.


>> The second issue that I noticed is that tw_refcnt is set to 4 _after_
>> we schedule the timer. The timer will probably won't fire before we
>> set tw_refcnt, but if it somehow does it will corrupt the ref count. I
>> don't think that it's what I am seeing, though. More likely it's the
>> first issues (if it's real).
>>
>
> Timer is pinned, it cannot fire under us on this cpu.

Ack

[toc] | [prev] | [next] | [standalone]

#1576947

On Wed, 2017-02-08 at 19:55 +0100, Dmitry Vyukov wrote:
> On Wed, Feb 8, 2017 at 6:58 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> > On Wed, 2017-02-08 at 18:36 +0100, Dmitry Vyukov wrote:
> >> On Tue, Jan 24, 2017 at 4:52 PM, Eric Dumazet <edumazet@google.com> wrote:
> >> > On Tue, Jan 24, 2017 at 7:06 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
> >> >>>
> >> >>> This code was changed a long time ago :
> >> >>>
> >> >>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=ed2e923945892a8372ab70d2f61d364b0b6d9054
> >> >>>
> >> >>> So I suspect a recent patch broke the logic.
> >> >>>
> >> >>> You might start a bisection :
> >> >>>
> >> >>> I would check if 4.7 and 4.8 trigger the issue you noticed.
> >> >>
> >> >>
> >> >> It happens with too low rate for bisecting (few times per day). I
> >> >> could add some additional checks into code, but I don't know what
> >> >> checks could be useful.
> >> >
> >> > If you can not tell if 4.7 and/or 4.8 have the problem, I am not sure
> >> > we are able to help.
> >>
> >>
> >> There are also chances that the problem is older.
> >>
> >> Looking at the code, this part of inet_twsk_purge looks fishy:
> >>
> >> 285                         if (unlikely((tw->tw_family != family) ||
> >> 286                                      atomic_read(&twsk_net(tw)->count))) {
> >>
> >> It uses net->count == 0 check to find the right sockets. But what if
> >> there are several nets with count == 0 in flight, can't there be
> >> several inet_twsk_purge calls running concurrently freeing each other
> >> sockets? If so it looks like inet_twsk_purge can call
> >> inet_twsk_deschedule_put twice for a socket. Namely, two calls for
> >> different nets discover the socket, check that net->count==0 and both
> >> call inet_twsk_deschedule_put. Shouldn't we just give inet_twsk_purge
> >> net that it needs to purge?
> >
> > Yes, atomic_read() is not a proper sync point.
> 
> Do you mean that it does not include read barrier?
> I more mean that we can call inet_twsk_deschedule_put twice for the same socket.

I meant that this code assumed RTNL being held.

This might not be the case now, after some old change.

[toc] | [prev] | [next] | [standalone]

#1576996

On Wed, Feb 8, 2017 at 8:17 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Wed, 2017-02-08 at 19:55 +0100, Dmitry Vyukov wrote:
>> On Wed, Feb 8, 2017 at 6:58 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
>> > On Wed, 2017-02-08 at 18:36 +0100, Dmitry Vyukov wrote:
>> >> On Tue, Jan 24, 2017 at 4:52 PM, Eric Dumazet <edumazet@google.com> wrote:
>> >> > On Tue, Jan 24, 2017 at 7:06 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
>> >> >>>
>> >> >>> This code was changed a long time ago :
>> >> >>>
>> >> >>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=ed2e923945892a8372ab70d2f61d364b0b6d9054
>> >> >>>
>> >> >>> So I suspect a recent patch broke the logic.
>> >> >>>
>> >> >>> You might start a bisection :
>> >> >>>
>> >> >>> I would check if 4.7 and 4.8 trigger the issue you noticed.
>> >> >>
>> >> >>
>> >> >> It happens with too low rate for bisecting (few times per day). I
>> >> >> could add some additional checks into code, but I don't know what
>> >> >> checks could be useful.
>> >> >
>> >> > If you can not tell if 4.7 and/or 4.8 have the problem, I am not sure
>> >> > we are able to help.
>> >>
>> >>
>> >> There are also chances that the problem is older.
>> >>
>> >> Looking at the code, this part of inet_twsk_purge looks fishy:
>> >>
>> >> 285                         if (unlikely((tw->tw_family != family) ||
>> >> 286                                      atomic_read(&twsk_net(tw)->count))) {
>> >>
>> >> It uses net->count == 0 check to find the right sockets. But what if
>> >> there are several nets with count == 0 in flight, can't there be
>> >> several inet_twsk_purge calls running concurrently freeing each other
>> >> sockets? If so it looks like inet_twsk_purge can call
>> >> inet_twsk_deschedule_put twice for a socket. Namely, two calls for
>> >> different nets discover the socket, check that net->count==0 and both
>> >> call inet_twsk_deschedule_put. Shouldn't we just give inet_twsk_purge
>> >> net that it needs to purge?
>> >
>> > Yes, atomic_read() is not a proper sync point.
>>
>> Do you mean that it does not include read barrier?
>> I more mean that we can call inet_twsk_deschedule_put twice for the same socket.
>
> I meant that this code assumed RTNL being held.
>
> This might not be the case now, after some old change.


cleanup_net releases rtnl lock right before calling these callbacks.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web