Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1732303 > unrolled thread
| Started by | nixiaoming <nixiaoming@huawei.com> |
|---|---|
| First post | 2017-09-14 16:30 +0200 |
| Last post | 2017-09-15 19:50 +0200 |
| Articles | 5 — 3 participants |
Back to article view | Back to linux.kernel
[PATCH] net/packet: fix race condition between fanout_add and __unregister_prot_hook nixiaoming <nixiaoming@huawei.com> - 2017-09-14 16:30 +0200
Re: [PATCH] net/packet: fix race condition between fanout_add and __unregister_prot_hook Willem de Bruijn <willemdebruijn.kernel@gmail.com> - 2017-09-14 16:40 +0200
Re: [PATCH] net/packet: fix race condition between fanout_add and __unregister_prot_hook Willem de Bruijn <willemdebruijn.kernel@gmail.com> - 2017-09-15 19:50 +0200
Re: [PATCH] net/packet: fix race condition between fanout_add and __unregister_prot_hook Cong Wang <xiyou.wangcong@gmail.com> - 2017-09-15 20:20 +0200
Re: [PATCH] net/packet: fix race condition between fanout_add and __unregister_prot_hook Cong Wang <xiyou.wangcong@gmail.com> - 2017-09-15 19:50 +0200
| From | nixiaoming <nixiaoming@huawei.com> |
|---|---|
| Date | 2017-09-14 16:30 +0200 |
| Subject | [PATCH] net/packet: fix race condition between fanout_add and __unregister_prot_hook |
| Message-ID | <upDjr-6by-7@gated-at.bofh.it> |
From: l00219569 <lisimin@huawei.com>
If fanout_add is preempted after running po-> fanout = match
and before running __fanout_link,
it will cause BUG_ON when __unregister_prot_hook call __fanout_unlink
so, we need add mutex_lock(&fanout_mutex) to __unregister_prot_hook
or add spin_lock(&po->bind_lock) before po-> fanout = match
this is a patch for add po->bind_lock in fanout_add
test on linux 4.1.12:
./trinity -c setsockopt -C 2 -X &
BUG: failure at net/packet/af_packet.c:1414/__fanout_unlink()!
Kernel panic - not syncing: BUG!
CPU: 2 PID: 2271 Comm: trinity-c0 Tainted: G W O 4.1.12 #1
Hardware name: Hisilicon PhosphorHi1382 FPGA (DT)
Call trace:
[<ffffffc000209414>] dump_backtrace+0x0/0xf8
[<ffffffc00020952c>] show_stack+0x20/0x28
[<ffffffc000635574>] dump_stack+0xac/0xe4
[<ffffffc000633fb8>] panic+0xf8/0x268
[<ffffffc0005fa778>] __unregister_prot_hook+0xa0/0x144
[<ffffffc0005fba48>] packet_set_ring+0x280/0x5b4
[<ffffffc0005fc33c>] packet_setsockopt+0x320/0x950
[<ffffffc000554a04>] SyS_setsockopt+0xa4/0xd4
Signed-off-by: nixiaoming <nixiaoming@huawei.com>
Tested-by: wudesheng <dede.wu@huawei.com>
---
net/packet/af_packet.c | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index 54a18a8..7a52a3b 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -1446,12 +1446,16 @@ static int fanout_add(struct sock *sk, u16 id, u16 type_flags)
default:
return -EINVAL;
}
-
- if (!po->running)
+ spin_lock(&po->bind_lock);
+ if (!po->running) {
+ spin_unlock(&po->bind_lock);
return -EINVAL;
+ }
- if (po->fanout)
+ if (po->fanout) {
+ spin_unlock(&po->bind_lock);
return -EALREADY;
+ }
mutex_lock(&fanout_mutex);
match = NULL;
@@ -1501,6 +1505,7 @@ static int fanout_add(struct sock *sk, u16 id, u16 type_flags)
}
out:
mutex_unlock(&fanout_mutex);
+ spin_unlock(&po->bind_lock);
return err;
}
--
2.10.1
[toc] | [next] | [standalone]
| From | Willem de Bruijn <willemdebruijn.kernel@gmail.com> |
|---|---|
| Date | 2017-09-14 16:40 +0200 |
| Message-ID | <upDt8-6ey-21@gated-at.bofh.it> |
| In reply to | #1732303 |
On Thu, Sep 14, 2017 at 10:07 AM, nixiaoming <nixiaoming@huawei.com> wrote:
> From: l00219569 <lisimin@huawei.com>
>
> If fanout_add is preempted after running po-> fanout = match
> and before running __fanout_link,
> it will cause BUG_ON when __unregister_prot_hook call __fanout_unlink
>
> so, we need add mutex_lock(&fanout_mutex) to __unregister_prot_hook
The packet socket code has no shortage of locks, so there are many
ways to avoid the race condition between fanout_add and packet_set_ring.
Another option would be to lock the socket when calling fanout_add:
- return fanout_add(sk, val & 0xffff, val >> 16);
+ lock_sock(sk);
+ ret = fanout_add(sk, val & 0xffff, val >> 16);
+ release_sock(sk);
+ return ret;
But, for consistency, and to be able to continue to make sense of the
locking policy, we should use the most appropriate lock. This
is po->bind_lock, as it ensures atomicity between testing whether
a protocol hook is active through po->running and the actual existence
of that hook on the protocol hook list.
fanout_mutex protects the fanout object's list. Taking that on
__unregister_prot_hook even in the case where fanout is not
used (and __dev_remove_pack is called) complicates locking
in this already complicated code.
> or add spin_lock(&po->bind_lock) before po-> fanout = match
>
> this is a patch for add po->bind_lock in fanout_add
>
> test on linux 4.1.12:
> ./trinity -c setsockopt -C 2 -X &
Thanks for testing!
>
> BUG: failure at net/packet/af_packet.c:1414/__fanout_unlink()!
> Kernel panic - not syncing: BUG!
> CPU: 2 PID: 2271 Comm: trinity-c0 Tainted: G W O 4.1.12 #1
> Hardware name: Hisilicon PhosphorHi1382 FPGA (DT)
> Call trace:
> [<ffffffc000209414>] dump_backtrace+0x0/0xf8
> [<ffffffc00020952c>] show_stack+0x20/0x28
> [<ffffffc000635574>] dump_stack+0xac/0xe4
> [<ffffffc000633fb8>] panic+0xf8/0x268
> [<ffffffc0005fa778>] __unregister_prot_hook+0xa0/0x144
> [<ffffffc0005fba48>] packet_set_ring+0x280/0x5b4
> [<ffffffc0005fc33c>] packet_setsockopt+0x320/0x950
> [<ffffffc000554a04>] SyS_setsockopt+0xa4/0xd4
>
> Signed-off-by: nixiaoming <nixiaoming@huawei.com>
> Tested-by: wudesheng <dede.wu@huawei.com>
> ---
> net/packet/af_packet.c | 11 ++++++++---
> 1 file changed, 8 insertions(+), 3 deletions(-)
>
> diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
> index 54a18a8..7a52a3b 100644
> --- a/net/packet/af_packet.c
> +++ b/net/packet/af_packet.c
> @@ -1446,12 +1446,16 @@ static int fanout_add(struct sock *sk, u16 id, u16 type_flags)
> default:
> return -EINVAL;
> }
> -
> - if (!po->running)
> + spin_lock(&po->bind_lock);
> + if (!po->running) {
> + spin_unlock(&po->bind_lock);
> return -EINVAL;
> + }
>
> - if (po->fanout)
> + if (po->fanout) {
> + spin_unlock(&po->bind_lock);
> return -EALREADY;
> + }
>
> mutex_lock(&fanout_mutex);
> match = NULL;
> @@ -1501,6 +1505,7 @@ static int fanout_add(struct sock *sk, u16 id, u16 type_flags)
> }
> out:
> mutex_unlock(&fanout_mutex);
> + spin_unlock(&po->bind_lock);
This function can call kzalloc with GFP_KERNEL, which may sleep. It is
not correct to sleep while holding a spinlock. Which is why I take the lock
later and test po->running again.
I will clean up that patch and send it for review.
[toc] | [prev] | [next] | [standalone]
| From | Willem de Bruijn <willemdebruijn.kernel@gmail.com> |
|---|---|
| Date | 2017-09-15 19:50 +0200 |
| Message-ID | <uq2Ux-69v-11@gated-at.bofh.it> |
| In reply to | #1732317 |
On Fri, Sep 15, 2017 at 1:41 PM, Cong Wang <xiyou.wangcong@gmail.com> wrote:
> On Thu, Sep 14, 2017 at 7:35 AM, Willem de Bruijn
> <willemdebruijn.kernel@gmail.com> wrote:
>> On Thu, Sep 14, 2017 at 10:07 AM, nixiaoming <nixiaoming@huawei.com> wrote:
>>> From: l00219569 <lisimin@huawei.com>
>>>
>>> If fanout_add is preempted after running po-> fanout = match
>>> and before running __fanout_link,
>>> it will cause BUG_ON when __unregister_prot_hook call __fanout_unlink
>>>
>>> so, we need add mutex_lock(&fanout_mutex) to __unregister_prot_hook
>>
>> The packet socket code has no shortage of locks, so there are many
>> ways to avoid the race condition between fanout_add and packet_set_ring.
>>
>> Another option would be to lock the socket when calling fanout_add:
>>
>> - return fanout_add(sk, val & 0xffff, val >> 16);
>> + lock_sock(sk);
>> + ret = fanout_add(sk, val & 0xffff, val >> 16);
>> + release_sock(sk);
>> + return ret;
>>
>
> I don't think this is an option, because __unregister_prot_hook()
> can be called without lock_sock(), for example in packet_notifier().
>
>
>> But, for consistency, and to be able to continue to make sense of the
>> locking policy, we should use the most appropriate lock. This
>> is po->bind_lock, as it ensures atomicity between testing whether
>> a protocol hook is active through po->running and the actual existence
>> of that hook on the protocol hook list.
>
> Yeah, register_prot_hook() and unregister_prot_hook() already assume
> bind_lock.
>
> [...]
>
>>> out:
>>> mutex_unlock(&fanout_mutex);
>>> + spin_unlock(&po->bind_lock);
>>
>> This function can call kzalloc with GFP_KERNEL, which may sleep. It is
>> not correct to sleep while holding a spinlock. Which is why I take the lock
>> later and test po->running again.
>
>
> Right, no need to mention the mutex_unlock() before the spin_unlock()
> is clearly wrong.
>
>
>>
>> I will clean up that patch and send it for review.
>
> How about the following patch?
>
>
> diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
> index c26172995511..f5c696a548ed 100644
> --- a/net/packet/af_packet.c
> +++ b/net/packet/af_packet.c
> @@ -1754,10 +1754,14 @@ static int fanout_add(struct sock *sk, u16 id,
> u16 type_flags)
> match->prot_hook.dev == po->prot_hook.dev) {
> err = -ENOSPC;
> if (refcount_read(&match->sk_ref) < PACKET_FANOUT_MAX) {
> + spin_lock(&po->bind_lock);
> __dev_remove_pack(&po->prot_hook);
> - po->fanout = match;
> - refcount_set(&match->sk_ref,
> refcount_read(&match->sk_ref) + 1);
> - __fanout_link(sk, po);
> + if (po->running) {
> + refcount_set(&match->sk_ref,
> refcount_read(&match->sk_ref) + 1);
> + po->fanout = match;
> + __fanout_link(sk, po);
> + }
> + spin_unlock(&po->bind_lock);
> err = 0;
> }
> }
In case of failure we also need to unlink and free match. I
sent the following:
http://patchwork.ozlabs.org/patch/813945/
[toc] | [prev] | [next] | [standalone]
| From | Cong Wang <xiyou.wangcong@gmail.com> |
|---|---|
| Date | 2017-09-15 20:20 +0200 |
| Message-ID | <uq3nA-6EQ-11@gated-at.bofh.it> |
| In reply to | #1732984 |
On Fri, Sep 15, 2017 at 10:46 AM, Willem de Bruijn <willemdebruijn.kernel@gmail.com> wrote: > > In case of failure we also need to unlink and free match. I > sent the following: > > http://patchwork.ozlabs.org/patch/813945/ Ah, will take a look.
[toc] | [prev] | [next] | [standalone]
| From | Cong Wang <xiyou.wangcong@gmail.com> |
|---|---|
| Date | 2017-09-15 19:50 +0200 |
| Message-ID | <uq2Ux-69v-13@gated-at.bofh.it> |
| In reply to | #1732317 |
On Thu, Sep 14, 2017 at 7:35 AM, Willem de Bruijn
<willemdebruijn.kernel@gmail.com> wrote:
> On Thu, Sep 14, 2017 at 10:07 AM, nixiaoming <nixiaoming@huawei.com> wrote:
>> From: l00219569 <lisimin@huawei.com>
>>
>> If fanout_add is preempted after running po-> fanout = match
>> and before running __fanout_link,
>> it will cause BUG_ON when __unregister_prot_hook call __fanout_unlink
>>
>> so, we need add mutex_lock(&fanout_mutex) to __unregister_prot_hook
>
> The packet socket code has no shortage of locks, so there are many
> ways to avoid the race condition between fanout_add and packet_set_ring.
>
> Another option would be to lock the socket when calling fanout_add:
>
> - return fanout_add(sk, val & 0xffff, val >> 16);
> + lock_sock(sk);
> + ret = fanout_add(sk, val & 0xffff, val >> 16);
> + release_sock(sk);
> + return ret;
>
I don't think this is an option, because __unregister_prot_hook()
can be called without lock_sock(), for example in packet_notifier().
> But, for consistency, and to be able to continue to make sense of the
> locking policy, we should use the most appropriate lock. This
> is po->bind_lock, as it ensures atomicity between testing whether
> a protocol hook is active through po->running and the actual existence
> of that hook on the protocol hook list.
Yeah, register_prot_hook() and unregister_prot_hook() already assume
bind_lock.
[...]
>> out:
>> mutex_unlock(&fanout_mutex);
>> + spin_unlock(&po->bind_lock);
>
> This function can call kzalloc with GFP_KERNEL, which may sleep. It is
> not correct to sleep while holding a spinlock. Which is why I take the lock
> later and test po->running again.
Right, no need to mention the mutex_unlock() before the spin_unlock()
is clearly wrong.
>
> I will clean up that patch and send it for review.
How about the following patch?
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index c26172995511..f5c696a548ed 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -1754,10 +1754,14 @@ static int fanout_add(struct sock *sk, u16 id,
u16 type_flags)
match->prot_hook.dev == po->prot_hook.dev) {
err = -ENOSPC;
if (refcount_read(&match->sk_ref) < PACKET_FANOUT_MAX) {
+ spin_lock(&po->bind_lock);
__dev_remove_pack(&po->prot_hook);
- po->fanout = match;
- refcount_set(&match->sk_ref,
refcount_read(&match->sk_ref) + 1);
- __fanout_link(sk, po);
+ if (po->running) {
+ refcount_set(&match->sk_ref,
refcount_read(&match->sk_ref) + 1);
+ po->fanout = match;
+ __fanout_link(sk, po);
+ }
+ spin_unlock(&po->bind_lock);
err = 0;
}
}
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web