Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1331955 > unrolled thread

Re: Bug 4.1.16: self-detected stall in net/unix/?

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Bug 4.1.16: self-detected stall in net/unix/? Philipp Hahn <pmhahn@pmhahn.de> - 2016-02-11 14:50 +0100
    Re: Bug 4.1.16: self-detected stall in net/unix/? Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2016-02-11 17:00 +0100
      Re: Bug 4.1.16: self-detected stall in net/unix/? Ben Hutchings <ben@decadent.org.uk> - 2016-02-11 18:10 +0100
        Re: Bug 4.1.16: self-detected stall in net/unix/? Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2016-02-11 18:50 +0100
          Re: Bug 4.1.16: self-detected stall in net/unix/? Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2016-02-11 19:00 +0100
          Re: Bug 4.1.16: self-detected stall in net/unix/? Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2016-02-11 19:40 +0100
            [PATCH net] af_unix: Guard against other == sk in unix_dgram_sendmsg Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2016-02-11 20:40 +0100
              Re: [PATCH net] af_unix: Guard against other == sk in  unix_dgram_sendmsg Philipp Hahn <pmhahn@pmhahn.de> - 2016-02-12 10:20 +0100
                Re: [PATCH net] af_unix: Guard against other == sk in unix_dgram_sendmsg Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2016-02-12 14:30 +0100
                  Re: [PATCH net] af_unix: Guard against other == sk in  unix_dgram_sendmsg Ben Hutchings <ben@decadent.org.uk> - 2016-02-12 21:00 +0100
                    Re: [PATCH net] af_unix: Guard against other == sk in unix_dgram_sendmsg Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2016-02-12 21:20 +0100
                      Re: [PATCH net] af_unix: Guard against other == sk in  unix_dgram_sendmsg Ben Hutchings <ben@decadent.org.uk> - 2016-02-12 21:50 +0100
                        Re: [PATCH net] af_unix: Guard against other == sk in unix_dgram_sendmsg Rainer Weikusat <rweikusat@mobileactivedefense.com> - 2016-02-12 22:10 +0100
              Re: [PATCH net] af_unix: Guard against other == sk in  unix_dgram_sendmsg David Miller <davem@davemloft.net> - 2016-02-16 19:00 +0100

#1331955 — Re: Bug 4.1.16: self-detected stall in net/unix/?

[Multipart message — attachments visible in raw view] — view raw

Hi,

Am 05.02.2016 um 16:28 schrieb Philipp Hahn:
> Am 03.02.2016 um 02:43 schrieb Hannes Frederic Sowa:
>> On 02.02.2016 17:25, Philipp Hahn wrote:
>>> we recently updated our kernel to 4.1.16 + patch for "unix: properly
>>> account for FDs passed over unix sockets" and have since then
>>> self-detected stalls triggered by the Samba daemon:
> ...
>>> We have not yet been able to reproduce the hang, but going back to our
>>> previous kernel 4.1.12 makes the problem go away.
>>
>> Can you remove the patch "unix: properly account for FDs passed over
>> unix sockets" and see if the problem still happens?
> 
> I will try.
> The problem is that I can't trigger the bug reliably. It always happens
> to "smbd", but I don't know the triggering condition.

Probably the same bug was also reported to samba-technical by Karolin
Seeger; she filed the bug for 3.19-ckt with Ubuntu:

<https://bugs.launchpad.net/ubuntu/+source/linux-lts-trusty/+bug/1543980>

Running the Samba test suite reproduces the problem; see bug for details.


> I will for now build a new kernel with
>> $ git log --oneline  v4.1.12..v4.1.17 -- net/unix
>> dc6b0ec unix: properly account for FDs passed over unix sockets
>> cc01a0a af_unix: Revert 'lock_interruptible' in stream receive code
>> 5c77e26 unix: avoid use-after-free in ep_remove_wait_queue
> reverted to see if it still happens. The "middle" patch seems harmless,
> as it only changes a code path for STREAMS, while the bug triggers with
> DGRAMS only.
> 
>> The stack trace is rather unreliable, maybe something completely
>> different happend. Do you happend to see better reports?
> 
> So far they look all the same.
> Anything more I can do to prepare for collection better information next
> time I get that bug?

I've enabled more Kernel debug options and got the following:

> [  598.482787] 
> [  598.492559] =====================================
> [  598.502646] [ BUG: bad unlock balance detected! ]
> [  598.512874] 4.1.16+ #24 Not tainted
> [  598.523134] -------------------------------------
> [  598.533592] smbd/8659 is trying to release lock (&(&u->lock)->rlock) at:
> [  598.544429] [<ffffffff815d1319>] spin_unlock+0x9/0x10
> [  598.555148] but there are no more locks to release!
> [  598.565892] 
> [  598.565892] other info that might help us debug this:
> [  598.586936] no locks held by smbd/8659.
> [  598.597478] 
> [  598.597478] stack backtrace:
> [  598.618275] CPU: 3 PID: 8659 Comm: smbd Not tainted 4.1.16+ #24
> [  598.628820] Hardware name: System manufacturer System Product Name/P7F-X Series, BIOS 0703    09/24/2010
> [  598.650020]  ffffffff815d1319 ffff8800b8efbb88 ffffffff8163ee73 0000000000000000
> [  598.661051]  ffff880034fc4110 ffff8800b8efbbb8 ffffffff810db540 ffff880034fc4110
> [  598.671990]  ffff880034fc4110 ffff88023206bd40 ffffffff815d1319 ffff8800b8efbc08
> [  598.682736] Call Trace:
> [  598.693187]  [<ffffffff815d1319>] ? spin_unlock+0x9/0x10
> [  598.703798]  [<ffffffff8163ee73>] dump_stack+0x4c/0x65
> [  598.714223]  [<ffffffff810db540>] print_unlock_imbalance_bug+0x100/0x110
> [  598.724611]  [<ffffffff815d1319>] ? spin_unlock+0x9/0x10
> [  598.734763]  [<ffffffff810e0d8e>] lock_release+0x2be/0x430
> [  598.744636]  [<ffffffff81648303>] _raw_spin_unlock+0x23/0x40
> [  598.754230]  [<ffffffff815d41a8>] ? unix_dgram_sendmsg+0x288/0x6f0
> [  598.763840]  [<ffffffff815d1319>] spin_unlock+0x9/0x10
> [  598.773126]  [<ffffffff815d41e7>] unix_dgram_sendmsg+0x2c7/0x6f0
> [  598.782209]  [<ffffffff814f6c9d>] sock_sendmsg+0x4d/0x60
> [  598.791313]  [<ffffffff814f7c3b>] ___sys_sendmsg+0x2db/0x2f0
> [  598.800369]  [<ffffffff812083c8>] ? kmem_cache_free+0x328/0x360
> [  598.809383]  [<ffffffff8127c1c0>] ? locks_free_lock+0x50/0x60
> [  598.818157]  [<ffffffff814f8649>] __sys_sendmsg+0x49/0x90
> [  598.826742]  [<ffffffff814f86a2>] SyS_sendmsg+0x12/0x20
> [  598.835110]  [<ffffffff816486f2>] system_call_fastpath+0x16/0x7a
> [  598.843546] ------------[ cut here ]------------
> [  598.851999] WARNING: CPU: 3 PID: 8659 at net/core/skbuff.c:691 skb_release_head_state+0xaa/0xb0()

I continued bisecting "v4.1.12..v4.1.17 -- net/unix/" and found:

Reverting the attached patch in 4.1 fixes my problem (for now). The
original patch went into 4.4, but was back-ported to several stable trees:

v3.2: a3b0f6e8a21ef02f69a15abac440572d8cde8c2a
v3.18: 72032798034d921ed565e3bf8dfdc3098f6473e2
v4.1: 5c77e26862ce604edea05b3442ed765e9756fe0f
v4.2: bad967fdd8ecbdd171f5f243657be033d2d081a7
v4.3: 58a6a46a036ce81a2a8ecaa6fc1537c894349e3f
v4.4: 7d267278a9ece963d77eefec61630223fce08c6c

Philipp

[toc] | [next] | [standalone]

#1332177

Philipp Hahn <pmhahn@pmhahn.de> writes:

[...]

> Probably the same bug was also reported to samba-technical by Karolin
> Seeger; she filed the bug for 3.19-ckt with Ubuntu:
>
> <https://bugs.launchpad.net/ubuntu/+source/linux-lts-trusty/+bug/1543980>
>
> Running the Samba test suite reproduces the problem; see bug for
> details.


JFTR: The oops in this bug report is for 3.13.0-77 and the patch you
reverted for 4.1 is not part of that (at least not of the upstream 3.13).


[...]

>> [  598.492559] =====================================
>> [  598.502646] [ BUG: bad unlock balance detected! ]
>> [  598.512874] 4.1.16+ #24 Not tainted
>> [  598.523134] -------------------------------------
>> [  598.533592] smbd/8659 is trying to release lock (&(&u->lock)->rlock) at:
>> [  598.544429] [<ffffffff815d1319>] spin_unlock+0x9/0x10
>> [  598.555148] but there are no more locks to release!
>> [  598.565892] 
>> [  598.565892] other info that might help us debug this:
>> [  598.586936] no locks held by smbd/8659.
>> [  598.597478] 
>> [  598.597478] stack backtrace:
>> [  598.618275] CPU: 3 PID: 8659 Comm: smbd Not tainted 4.1.16+ #24
>> [  598.628820] Hardware name: System manufacturer System Product Name/P7F-X Series, BIOS 0703    09/24/2010
>> [  598.650020]  ffffffff815d1319 ffff8800b8efbb88 ffffffff8163ee73 0000000000000000
>> [  598.661051]  ffff880034fc4110 ffff8800b8efbbb8 ffffffff810db540 ffff880034fc4110
>> [  598.671990]  ffff880034fc4110 ffff88023206bd40 ffffffff815d1319 ffff8800b8efbc08
>> [  598.682736] Call Trace:
>> [  598.693187]  [<ffffffff815d1319>] ? spin_unlock+0x9/0x10
>> [  598.703798]  [<ffffffff8163ee73>] dump_stack+0x4c/0x65
>> [  598.714223]  [<ffffffff810db540>] print_unlock_imbalance_bug+0x100/0x110
>> [  598.724611]  [<ffffffff815d1319>] ? spin_unlock+0x9/0x10
>> [  598.734763]  [<ffffffff810e0d8e>] lock_release+0x2be/0x430
>> [  598.744636]  [<ffffffff81648303>] _raw_spin_unlock+0x23/0x40
>> [  598.754230]  [<ffffffff815d41a8>] ? unix_dgram_sendmsg+0x288/0x6f0
>> [  598.763840]  [<ffffffff815d1319>] spin_unlock+0x9/0x10
>> [  598.773126]  [<ffffffff815d41e7>] unix_dgram_sendmsg+0x2c7/0x6f0
>> [  598.782209]  [<ffffffff814f6c9d>] sock_sendmsg+0x4d/0x60
>> [  598.791313]  [<ffffffff814f7c3b>] ___sys_sendmsg+0x2db/0x2f0
>> [  598.800369]  [<ffffffff812083c8>] ? kmem_cache_free+0x328/0x360
>> [  598.809383]  [<ffffffff8127c1c0>] ? locks_free_lock+0x50/0x60
>> [  598.818157]  [<ffffffff814f8649>] __sys_sendmsg+0x49/0x90
>> [  598.826742]  [<ffffffff814f86a2>] SyS_sendmsg+0x12/0x20
>> [  598.835110]  [<ffffffff816486f2>] system_call_fastpath+0x16/0x7a
>> [  598.843546] ------------[ cut here ]------------
>> [  598.851999] WARNING: CPU: 3 PID: 8659 at net/core/skbuff.c:691 skb_release_head_state+0xaa/0xb0()

Could you send your disassembled unix_dgram_sendmsg (objdump -d)? This
would be enormously helpful wrt determining which unlock is involved
here.

[toc] | [prev] | [next] | [standalone]

#1332238

[Multipart message — attachments visible in raw view] — view raw

On Thu, 2016-02-11 at 15:55 +0000, Rainer Weikusat wrote:
> Philipp Hahn <pmhahn@pmhahn.de> writes:
> 
> [...]
> 
> > Probably the same bug was also reported to samba-technical by Karolin
> > Seeger; she filed the bug for 3.19-ckt with Ubuntu:
> > 
> > 
> > 
> > Running the Samba test suite reproduces the problem; see bug for
> > details.
> 
> 
> JFTR: The oops in this bug report is for 3.13.0-77 and the patch you
> reverted for 4.1 is not part of that (at least not of the upstream 3.13).
[...]

It is in 3.13-ckt and basically all the stable branches.

Does the patch below fix this bug?

Ben.

---
unix: Fix potential double-unlock in unix_dgram_sendmsg()

A datagram socket may be peered with itself, so that sk == other.  We
use unix_state_double_lock() to lock sk and other in the right order,
which also guards against this and only locks the socket once, but we
then end up trying to unlock it twice.  Add the check for sk != other.

Reported-by: Philipp Hahn <pmhahn@pmhahn.de>
Fixes: 7d267278a9ec ("unix: avoid use-after-free in ep_remove_wait_queue")
Cc: stable <stable@vger.kernel.org>
Signed-off-by: Ben Hutchings <ben@decadent.org.uk>
---
 net/unix/af_unix.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c
index c5bf5ef2bf89..b4320d3e3a25 100644
--- a/net/unix/af_unix.c
+++ b/net/unix/af_unix.c
@@ -1810,7 +1810,7 @@ restart_locked:
 		}
 	}
 
-	if (unlikely(sk_locked))
+	if (unlikely(sk_locked) && sk != other)
 		unix_state_unlock(sk);
 
 	if (sock_flag(other, SOCK_RCVTSTAMP))
@@ -1826,7 +1826,7 @@ restart_locked:
 	return len;
 
 out_unlock:
-	if (sk_locked)
+	if (sk_locked && sk != other)
 		unix_state_unlock(sk);
 	unix_state_unlock(other);
 out_free:

-- 
Ben Hutchings
Who are all these weirdos? - David Bowie, reading IRC for the first time

[toc] | [prev] | [next] | [standalone]

#1332268

Ben Hutchings <ben@decadent.org.uk> writes:
> On Thu, 2016-02-11 at 15:55 +0000, Rainer Weikusat wrote:
>> Philipp Hahn <pmhahn@pmhahn.de> writes:
>> 
>> [...]
>> 
>> > Probably the same bug was also reported to samba-technical by Karolin
>> > Seeger; she filed the bug for 3.19-ckt with Ubuntu:
>> > 
>> > 
>> > 
>> > Running the Samba test suite reproduces the problem; see bug for
>> > details.
>> 
>> 
>> JFTR: The oops in this bug report is for 3.13.0-77 and the patch you
>> reverted for 4.1 is not part of that (at least not of the upstream 3.13).
> [...]
>
> It is in 3.13-ckt and basically all the stable branches.
>
> Does the patch below fix this bug?
>
> Ben.
>
> ---
> unix: Fix potential double-unlock in unix_dgram_sendmsg()
>
> A datagram socket may be peered with itself, so that sk == other.  We
> use unix_state_double_lock() to lock sk and other in the right order,
> which also guards against this and only locks the socket once, but we
> then end up trying to unlock it twice.  Add the check for sk != other.

That's a good observation but I think this happens in another way. The
code setting sk_logged is

	if (unlikely(unix_peer(other) != sk && unix_recvq_full(other))) {
		if (timeo) {
			timeo = unix_wait_for_peer(other, timeo);

			err = sock_intr_errno(timeo);
			if (signal_pending(current))
				goto out_free;

			goto restart;
		}

		if (!sk_locked) {
			unix_state_unlock(other);
			unix_state_double_lock(sk, other);
		}

		if (unix_peer(sk) != other ||
		    unix_dgram_peer_wake_me(sk, other)) {
			err = -EAGAIN;
			sk_locked = 1;
			goto out_unlock;
		}

		if (!sk_locked) {
			sk_locked = 1;
			goto restart_locked;
		}
	}

This means it only gets locked if unix_peer(other) != sk and this cannot
happen if other == sk and unix_peer(sk) == other, however, the 2nd
condition isn't guaranteed: other might indeed be == sk and not the peer
of it because someone could be using _sendmsg to send a message via a
socket to an address bound to the same socket. In this case, other was
found via

	if (!other) {
		err = -ECONNRESET;
		if (sunaddr == NULL)
			goto out_free;

		other = unix_find_other(net, sunaddr, namelen, sk->sk_type,
					hash, &err);
		if (other == NULL)
			goto out_free;
	}

and the if-block leading to the double lock should never have been
executed as it's supposed to deal with the case where sk is connect to
other but other not to sk (eg, /dev/log).

If the description correct, the patch below should fix it (as sunaddr
gets cleared if and only if unix_peer(sk) == other.

This is a 'preview', ie, not even compiled. It's provided for
discussion/ testing. I'll try to create a test program for this and a
more formal patch.

---

diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c
index 1975fd8..06259ac 100644
--- a/net/unix/af_unix.c
+++ b/net/unix/af_unix.c
@@ -1698,7 +1698,7 @@ restart_locked:
                        goto out_unlock;
        }
 
-       if (unlikely(unix_peer(other) != sk && unix_recvq_full(other))) {
+       if (!sunaddr && unlikely(unix_peer(other) != sk && unix_recvq_full(other))) {
                if (timeo) {
                        timeo = unix_wait_for_peer(other, timeo);
 

[toc] | [prev] | [next] | [standalone]

#1332270

Rainer Weikusat <rw@doppelsaurus.mobileactivedefense.com> writes:

[...]

> This means it only gets locked if unix_peer(other) != sk and this cannot
> happen if other == sk and unix_peer(sk) == other, however, the 2nd
> condition isn't guaranteed: other might indeed be == sk and not the peer
> of it because someone could be using _sendmsg to send a message via a
> socket to an address bound to the same socket. In this case, other was
> found via

A second way to hits this (probably somewhat difficult to trigger in
practice): sk happened to be connected to itself by the time the
unix_peer_get(sk) was executed but was disconnected before the
unix_state_lock(other) below.

[toc] | [prev] | [next] | [standalone]

#1332286

Rainer Weikusat <rw@doppelsaurus.mobileactivedefense.com> writes:
> Ben Hutchings <ben@decadent.org.uk> writes:

[...]

>> unix: Fix potential double-unlock in unix_dgram_sendmsg()
>>
>> A datagram socket may be peered with itself, so that sk == other.  We
>> use unix_state_double_lock() to lock sk and other in the right order,
>> which also guards against this and only locks the socket once, but we
>> then end up trying to unlock it twice.  Add the check for sk != other.

[...]

> other was found via
>
> 	if (!other) {
> 		err = -ECONNRESET;
> 		if (sunaddr == NULL)
> 			goto out_free;
>
> 		other = unix_find_other(net, sunaddr, namelen, sk->sk_type,
> 					hash, &err);
> 		if (other == NULL)
> 			goto out_free;
> 	}
>
> and the if-block leading to the double lock should never have been
> executed as it's supposed to deal with the case where sk is connect to
> other but other not to sk (eg, /dev/log).

Test program triggering a double unlock (w/o the MSG_DONTWAIT, it blocks
which it isn't supposed to do):

--------
#include <stdio.h>
#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>

#define SUN_NAME	"\0other-is-me"

int main(int argc, char **argv)
{
    struct sockaddr_un sun;
    unsigned max;
    int sk, rc;

    sk = socket(AF_UNIX, SOCK_DGRAM, 0);

    sun.sun_family = AF_UNIX;
    strncpy(sun.sun_path, SUN_NAME, sizeof(sun.sun_path));
    bind(sk, (struct sockaddr *)&sun, sizeof(sun));

    max = 12;
    do 
	sendto(sk, &sun, sizeof(sun), MSG_DONTWAIT,
	       (struct sockaddr *)&sun, sizeof(sun));
    while (--max);

    return 0;
}

[toc] | [prev] | [next] | [standalone]

#1332335 — [PATCH net] af_unix: Guard against other == sk in unix_dgram_sendmsg

The unix_dgram_sendmsg routine use the following test

if (unlikely(unix_peer(other) != sk && unix_recvq_full(other))) {

to determine if sk and other are in an n:1 association (either
established via connect or by using sendto to send messages to an
unrelated socket identified by address). This isn't correct as the
specified address could have been bound to the sending socket itself or
because this socket could have been connected to itself by the time of
the unix_peer_get but disconnected before the unix_state_lock(other). In
both cases, the if-block would be entered despite other == sk which
might either block the sender unintentionally or lead to trying to unlock
the same spin lock twice for a non-blocking send. Add a other != sk
check to guard against this.

Fixes: 7d267278a9ec ("unix: avoid use-after-free in ep_remove_wait_queue")
Reported-By: Philipp Hahn <pmhahn@pmhahn.de>
Signed-off-by: Rainer Weikusat <rweikusat@mobileactivedefense.com>
---
diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c
index 29be035..f1ca279 100644
--- a/net/unix/af_unix.c
+++ b/net/unix/af_unix.c
@@ -1781,7 +1781,12 @@ restart_locked:
 			goto out_unlock;
 	}
 
-	if (unlikely(unix_peer(other) != sk && unix_recvq_full(other))) {
+	/* other == sk && unix_peer(other) != sk if
+	 * - unix_peer(sk) == NULL, destination address bound to sk
+	 * - unix_peer(sk) == sk by time of get but disconnected before lock
+	 */
+	if (other != sk &&
+	    unlikely(unix_peer(other) != sk && unix_recvq_full(other))) {
 		if (timeo) {
 			timeo = unix_wait_for_peer(other, timeo);
 

[toc] | [prev] | [next] | [standalone]

#1332602 — Re: [PATCH net] af_unix: Guard against other == sk in unix_dgram_sendmsg

Hello Rainer,

Am 11.02.2016 um 20:37 schrieb Rainer Weikusat:
> The unix_dgram_sendmsg routine use the following test
> 
> if (unlikely(unix_peer(other) != sk && unix_recvq_full(other))) {
> 
> to determine if sk and other are in an n:1 association (either
> established via connect or by using sendto to send messages to an
> unrelated socket identified by address). This isn't correct as the
> specified address could have been bound to the sending socket itself or
> because this socket could have been connected to itself by the time of
> the unix_peer_get but disconnected before the unix_state_lock(other). In
> both cases, the if-block would be entered despite other == sk which
> might either block the sender unintentionally or lead to trying to unlock
> the same spin lock twice for a non-blocking send. Add a other != sk
> check to guard against this.
> 
> Fixes: 7d267278a9ec ("unix: avoid use-after-free in ep_remove_wait_queue")
> Reported-By: Philipp Hahn <pmhahn@pmhahn.de>
> Signed-off-by: Rainer Weikusat <rweikusat@mobileactivedefense.com>
> ---
> diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c
> index 29be035..f1ca279 100644
> --- a/net/unix/af_unix.c
> +++ b/net/unix/af_unix.c
> @@ -1781,7 +1781,12 @@ restart_locked:
>  			goto out_unlock;
>  	}
>  
> -	if (unlikely(unix_peer(other) != sk && unix_recvq_full(other))) {
> +	/* other == sk && unix_peer(other) != sk if
> +	 * - unix_peer(sk) == NULL, destination address bound to sk
> +	 * - unix_peer(sk) == sk by time of get but disconnected before lock
> +	 */
> +	if (other != sk &&
> +	    unlikely(unix_peer(other) != sk && unix_recvq_full(other))) {
>  		if (timeo) {
>  			timeo = unix_wait_for_peer(other, timeo);
>  
> 

After applying that patch at least my machine running the samba test no
longer crashes. So you might add
Tested-by: Philipp Hahn <pmhahn@pmhahn.de>

Thanks for looking it that issues.

Philipp

[toc] | [prev] | [next] | [standalone]

#1332734 — Re: [PATCH net] af_unix: Guard against other == sk in unix_dgram_sendmsg

Philipp Hahn <pmhahn@pmhahn.de> writes:

> Hello Rainer,
>
> Am 11.02.2016 um 20:37 schrieb Rainer Weikusat:
>> The unix_dgram_sendmsg routine use the following test
>> 
>> if (unlikely(unix_peer(other) != sk && unix_recvq_full(other))) {

[...]

>> This isn't correct as the> specified address could have been bound to
>> the sending socket itself

[...]

> After applying that patch at least my machine running the samba test no
> longer crashes.

There's a possible gotcha in there: Send-to-self used to be limited by
the queue limit. But the rationale for that (IIRC) was that someone
could keep using newly created sockets to queue ever more data to a
single, unrelated receiver. I don't think this should apply when
receiving and sending sockets are identical. But that's just my
opinion. The other option would be to avoid the unix_state_double_lock
for sk == other. I'd be willing to change this accordingly if someone
thinks the queue limit should apply to send-to-self.

[toc] | [prev] | [next] | [standalone]

#1333073 — Re: [PATCH net] af_unix: Guard against other == sk in unix_dgram_sendmsg

[Multipart message — attachments visible in raw view] — view raw

On Fri, 2016-02-12 at 13:25 +0000, Rainer Weikusat wrote:
> Philipp Hahn <pmhahn@pmhahn.de> writes:
> 
> > Hello Rainer,
> > 
> > Am 11.02.2016 um 20:37 schrieb Rainer Weikusat:
> > > The unix_dgram_sendmsg routine use the following test
> > > 
> > > if (unlikely(unix_peer(other) != sk && unix_recvq_full(other))) {
> 
> [...]
> 
> > > This isn't correct as the> specified address could have been bound to
> > > the sending socket itself
> 
> [...]
> 
> > After applying that patch at least my machine running the samba test no
> > longer crashes.
> 
> There's a possible gotcha in there: Send-to-self used to be limited by
> the queue limit. But the rationale for that (IIRC) was that someone
> could keep using newly created sockets to queue ever more data to a
> single, unrelated receiver. I don't think this should apply when
> receiving and sending sockets are identical. But that's just my
> opinion. The other option would be to avoid the unix_state_double_lock
> for sk == other.

Given that unix_state_double_lock() already handles sk == other, I'm
not sure why you think it needs to be avoided.

> I'd be willing to change this accordingly if someone
> thinks the queue limit should apply to send-to-self.

If we don't check the queue limit here, does anything else prevent the
queue growing to the point it's a DoS?

Ben.

-- 
Ben Hutchings
I say we take off; nuke the site from orbit.  It's the only way to be sure.

[toc] | [prev] | [next] | [standalone]

#1333081 — Re: [PATCH net] af_unix: Guard against other == sk in unix_dgram_sendmsg

Ben Hutchings <ben@decadent.org.uk> writes:
> On Fri, 2016-02-12 at 13:25 +0000, Rainer Weikusat wrote:
>> Philipp Hahn <pmhahn@pmhahn.de> writes:
>> > Hello Rainer,
>> > 
>> > Am 11.02.2016 um 20:37 schrieb Rainer Weikusat:
>> > > The unix_dgram_sendmsg routine use the following test
>> > > 
>> > > if (unlikely(unix_peer(other) != sk && unix_recvq_full(other))) {
>> 
>> [...]
>> 
>> > > This isn't correct as the> specified address could have been bound to
>> > > the sending socket itself
>> 
>> [...]
>> 
>> > After applying that patch at least my machine running the samba test no
>> > longer crashes.
>> 
>> There's a possible gotcha in there: Send-to-self used to be limited by
>> the queue limit. But the rationale for that (IIRC) was that someone
>> could keep using newly created sockets to queue ever more data to a
>> single, unrelated receiver. I don't think this should apply when
>> receiving and sending sockets are identical. But that's just my
>> opinion. The other option would be to avoid the unix_state_double_lock
>> for sk == other.
>
> Given that unix_state_double_lock() already handles sk == other, I'm
> not sure why you think it needs to be avoided.

Because the whole complication of restarting the operation after locking
both sk and other because other had to be unlocked before calling
unix_state_double_lock is useless for this case: As other == sk, there's
no reason to drop the lock on it which guarantees that the result of all
the earlier checks is still valid: If the -EAGAIN condition is not true,
execution can just continue.

>> I'd be willing to change this accordingly if someone
>> thinks the queue limit should apply to send-to-self.
>
> If we don't check the queue limit here, does anything else prevent the
> queue growing to the point it's a DoS?

The max_dgram_qlen limit exists specifically to prevent someone sending
'a lot' of messages to a socket unrelated to it by repeatedly creating a
socket, sending as many messages as the send buffer size will allow,
closing the socket, creating a new socket, ..., cf

http://netdev.vger.kernel.narkive.com/tcZIFJeC/get-rid-of-proc-sys-net-unix-max-dgram-qlen#post4
(first copy I found)

This 'attack' will obviously not work very well when sending and
receiving socket are identical.

[toc] | [prev] | [next] | [standalone]

#1333091 — Re: [PATCH net] af_unix: Guard against other == sk in unix_dgram_sendmsg

[Multipart message — attachments visible in raw view] — view raw

On Fri, 2016-02-12 at 20:17 +0000, Rainer Weikusat wrote:
> Ben Hutchings <ben@decadent.org.uk> writes:
> > On Fri, 2016-02-12 at 13:25 +0000, Rainer Weikusat wrote:
> > > Philipp Hahn <pmhahn@pmhahn.de> writes:
> > > > Hello Rainer,
> > > > 
> > > > Am 11.02.2016 um 20:37 schrieb Rainer Weikusat:
> > > > > The unix_dgram_sendmsg routine use the following test
> > > > > 
> > > > > if (unlikely(unix_peer(other) != sk && unix_recvq_full(other))) {
> > > 
> > > [...]
> > > 
> > > > > This isn't correct as the> specified address could have been bound to
> > > > > the sending socket itself
> > > 
> > > [...]
> > > 
> > > > After applying that patch at least my machine running the samba test no
> > > > longer crashes.
> > > 
> > > There's a possible gotcha in there: Send-to-self used to be limited by
> > > the queue limit. But the rationale for that (IIRC) was that someone
> > > could keep using newly created sockets to queue ever more data to a
> > > single, unrelated receiver. I don't think this should apply when
> > > receiving and sending sockets are identical. But that's just my
> > > opinion. The other option would be to avoid the unix_state_double_lock
> > > for sk == other.
> > 
> > Given that unix_state_double_lock() already handles sk == other, I'm
> > not sure why you think it needs to be avoided.
> 
> Because the whole complication of restarting the operation after locking
> both sk and other because other had to be unlocked before calling
> unix_state_double_lock is useless for this case: As other == sk, there's
> no reason to drop the lock on it which guarantees that the result of all
> the earlier checks is still valid: If the -EAGAIN condition is not true,
> execution can just continue.

Well of course it's useless, but it's also harmless.  If we really
wanted to optimise this we could also skip unlocking if other < sk.

> > > I'd be willing to change this accordingly if someone
> > > thinks the queue limit should apply to send-to-self.
> > 
> > If we don't check the queue limit here, does anything else prevent the
> > queue growing to the point it's a DoS?
> 
> The max_dgram_qlen limit exists specifically to prevent someone sending
> 'a lot' of messages to a socket unrelated to it by repeatedly creating a
> socket, sending as many messages as the send buffer size will allow,
> closing the socket, creating a new socket, ..., cf
> 
> http://netdev.vger.kernel.narkive.com/tcZIFJeC/get-rid-of-proc-sys-net-unix-max-dgram-qlen#post4
> (first copy I found)
> 
> This 'attack' will obviously not work very well when sending and
> receiving socket are identical.

It looked to me like the queue length was the only limit here, as I was
looking in vain for a charge to the receiving socket's memory.
However, to answer my own question, AF_UNIX skbs are always charged to
the sending socket (which is the same thing in this case, but still
affects where the buffer limit is applied).

Ben.

-- 
Ben Hutchings
I say we take off; nuke the site from orbit.  It's the only way to be sure.

[toc] | [prev] | [next] | [standalone]

#1333097 — Re: [PATCH net] af_unix: Guard against other == sk in unix_dgram_sendmsg

Ben Hutchings <ben@decadent.org.uk> writes:
> On Fri, 2016-02-12 at 20:17 +0000, Rainer Weikusat wrote:

[...]

>>>> I don't think this should apply when
>>>> receiving and sending sockets are identical. But that's just my
>>>> opinion. The other option would be to avoid the unix_state_double_lock
>>>> for sk == other.
>>> 
>>> Given that unix_state_double_lock() already handles sk == other, I'm
>>> not sure why you think it needs to be avoided.
>> 
>> Because the whole complication of restarting the operation after locking
>> both sk and other because other had to be unlocked before calling
>> unix_state_double_lock is useless for this case:

[...]

> Well of course it's useless, but it's also harmless.  

As is adding a

for (i = 0; i < 1000000; ++i);

between any two statements. And this isn't even entirely true as the
pointless double-lock will then require "did we pointlessly
doube-lock" checks elsewhere. I think it should be possible to do this
in a simpler way by not pointlessly double-locking (this may be
wrong but it's worth a try).

> If we really wanted to optimise this we could also skip unlocking if
> other < sk.

I wouldn't want to hardcode assumptions about the unix_state_double_lock
algorithm in functions using it. 

[toc] | [prev] | [next] | [standalone]

#1335688 — Re: [PATCH net] af_unix: Guard against other == sk in unix_dgram_sendmsg

From: Rainer Weikusat <rweikusat@mobileactivedefense.com>
Date: Thu, 11 Feb 2016 19:37:27 +0000

> The unix_dgram_sendmsg routine use the following test
> 
> if (unlikely(unix_peer(other) != sk && unix_recvq_full(other))) {
> 
> to determine if sk and other are in an n:1 association (either
> established via connect or by using sendto to send messages to an
> unrelated socket identified by address). This isn't correct as the
> specified address could have been bound to the sending socket itself or
> because this socket could have been connected to itself by the time of
> the unix_peer_get but disconnected before the unix_state_lock(other). In
> both cases, the if-block would be entered despite other == sk which
> might either block the sender unintentionally or lead to trying to unlock
> the same spin lock twice for a non-blocking send. Add a other != sk
> check to guard against this.
> 
> Fixes: 7d267278a9ec ("unix: avoid use-after-free in ep_remove_wait_queue")
> Reported-By: Philipp Hahn <pmhahn@pmhahn.de>
> Signed-off-by: Rainer Weikusat <rweikusat@mobileactivedefense.com>

Also applied and queued up for -stable, thanks.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web