Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1233439 > unrolled thread

[PATCH 4.1 125/159] net: call rcu_read_lock early in process_backlog

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  [PATCH 4.1 125/159] net: call rcu_read_lock early in process_backlog Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2015-09-26 23:40 +0200
    Re: [PATCH 4.1 125/159] net: call rcu_read_lock early in  process_backlog "Andre Tomt (LKML)" <lkml@tomt.net> - 2015-09-29 04:20 +0200
      Re: [PATCH 4.1 125/159] net: call rcu_read_lock early in  process_backlog Julian Anastasov <ja@ssi.bg> - 2015-09-29 09:50 +0200
        Re: [PATCH 4.1 125/159] net: call rcu_read_lock early in  process_backlog Andre Tomt <andre@tomt.net> - 2015-09-29 12:50 +0200
          Re: [PATCH 4.1 125/159] net: call rcu_read_lock early in  process_backlog Zdenek Kaspar <zkaspar82@gmail.com> - 2015-09-30 02:30 +0200
          kernel 4.1.9: networking hangs with rcu_preempt self-detected stall, 4.1.8 works; was: Re: [PATCH 4.1 125/159] net: call rcu_read_lock early in process_backlog Wolfgang Walter <linux@stwm.de> - 2015-10-01 13:40 +0200
            Re: kernel 4.1.9: networking hangs with rcu_preempt self-detected  stall, 4.1.8 works; was: Re: [PATCH 4.1 125/159] net: call rcu_read_lock  early in process_backlog Holger Hoffstätte   <holger.hoffstaette@googlemail.com> - 2015-10-01 13:50 +0200
        Re: [PATCH 4.1 125/159] net: call rcu_read_lock early in  process_backlog Julian Anastasov <ja@ssi.bg> - 2015-09-29 21:30 +0200

#1233439 — [PATCH 4.1 125/159] net: call rcu_read_lock early in process_backlog

4.1-stable review patch.  If anyone has any objections, please let me know.

------------------

From: Julian Anastasov <ja@ssi.bg>

[ Upstream commit 2c17d27c36dcce2b6bf689f41a46b9e909877c21 ]

Incoming packet should be either in backlog queue or
in RCU read-side section. Otherwise, the final sequence of
flush_backlog() and synchronize_net() may miss packets
that can run without device reference:

CPU 1                  CPU 2
                       skb->dev: no reference
                       process_backlog:__skb_dequeue
                       process_backlog:local_irq_enable

on_each_cpu for
flush_backlog =>       IPI(hardirq): flush_backlog
                       - packet not found in backlog

                       CPU delayed ...
synchronize_net
- no ongoing RCU
read-side sections

netdev_run_todo,
rcu_barrier: no
ongoing callbacks
                       __netif_receive_skb_core:rcu_read_lock
                       - too late
free dev
                       process packet for freed dev

Fixes: 6e583ce5242f ("net: eliminate refcounting in backlog queue")
Cc: Eric W. Biederman <ebiederm@xmission.com>
Cc: Stephen Hemminger <stephen@networkplumber.org>
Signed-off-by: Julian Anastasov <ja@ssi.bg>
Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
 net/core/dev.c |   30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)

--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -3666,8 +3666,6 @@ static int __netif_receive_skb_core(stru
 
 	pt_prev = NULL;
 
-	rcu_read_lock();
-
 another_round:
 	skb->skb_iif = skb->dev->ifindex;
 
@@ -3677,7 +3675,7 @@ another_round:
 	    skb->protocol == cpu_to_be16(ETH_P_8021AD)) {
 		skb = skb_vlan_untag(skb);
 		if (unlikely(!skb))
-			goto unlock;
+			goto out;
 	}
 
 #ifdef CONFIG_NET_CLS_ACT
@@ -3707,7 +3705,7 @@ skip_taps:
 	if (static_key_false(&ingress_needed)) {
 		skb = handle_ing(skb, &pt_prev, &ret, orig_dev);
 		if (!skb)
-			goto unlock;
+			goto out;
 	}
 
 	skb->tc_verd = 0;
@@ -3724,7 +3722,7 @@ ncls:
 		if (vlan_do_receive(&skb))
 			goto another_round;
 		else if (unlikely(!skb))
-			goto unlock;
+			goto out;
 	}
 
 	rx_handler = rcu_dereference(skb->dev->rx_handler);
@@ -3736,7 +3734,7 @@ ncls:
 		switch (rx_handler(&skb)) {
 		case RX_HANDLER_CONSUMED:
 			ret = NET_RX_SUCCESS;
-			goto unlock;
+			goto out;
 		case RX_HANDLER_ANOTHER:
 			goto another_round;
 		case RX_HANDLER_EXACT:
@@ -3790,8 +3788,7 @@ drop:
 		ret = NET_RX_DROP;
 	}
 
-unlock:
-	rcu_read_unlock();
+out:
 	return ret;
 }
 
@@ -3822,29 +3819,30 @@ static int __netif_receive_skb(struct sk
 
 static int netif_receive_skb_internal(struct sk_buff *skb)
 {
+	int ret;
+
 	net_timestamp_check(netdev_tstamp_prequeue, skb);
 
 	if (skb_defer_rx_timestamp(skb))
 		return NET_RX_SUCCESS;
 
+	rcu_read_lock();
+
 #ifdef CONFIG_RPS
 	if (static_key_false(&rps_needed)) {
 		struct rps_dev_flow voidflow, *rflow = &voidflow;
-		int cpu, ret;
-
-		rcu_read_lock();
-
-		cpu = get_rps_cpu(skb->dev, skb, &rflow);
+		int cpu = get_rps_cpu(skb->dev, skb, &rflow);
 
 		if (cpu >= 0) {
 			ret = enqueue_to_backlog(skb, cpu, &rflow->last_qtail);
 			rcu_read_unlock();
 			return ret;
 		}
-		rcu_read_unlock();
 	}
 #endif
-	return __netif_receive_skb(skb);
+	ret = __netif_receive_skb(skb);
+	rcu_read_unlock();
+	return ret;
 }
 
 /**
@@ -4389,8 +4387,10 @@ static int process_backlog(struct napi_s
 		struct sk_buff *skb;
 
 		while ((skb = __skb_dequeue(&sd->process_queue))) {
+			rcu_read_lock();
 			local_irq_enable();
 			__netif_receive_skb(skb);
+			rcu_read_unlock();
 			local_irq_disable();
 			input_queue_head_incr(sd);
 			if (++work >= quota) {


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1234649 — Re: [PATCH 4.1 125/159] net: call rcu_read_lock early in process_backlog

On 26. sep. 2015 22:56, Greg Kroah-Hartman wrote:
> 4.1-stable review patch.  If anyone has any objections, please let me know.
>
> ------------------
>
> From: Julian Anastasov <ja@ssi.bg>
>
> [ Upstream commit 2c17d27c36dcce2b6bf689f41a46b9e909877c21 ]
>
> Incoming packet should be either in backlog queue or
> in RCU read-side section. Otherwise, the final sequence of
> flush_backlog() and synchronize_net() may miss packets
> that can run without device reference:
<snip>

Several of our 4.1.9-rc1 running systems are experiencing hangs 
requiring hardware/sysrq reset with this patch applied. Reverting it 
fixes the hangs completely.

4.2 includes this patch as well but I have no such problems there. 
4.2.2-rc1 works fine as well.

For now I think this patch should be reverted in 4.1.9.

The hangs have occured so far on Xen PV and KVM x86_64 virtual machines, 
they will hang completely within minutes or hours depending on the type 
of workload. The workloads are all fairly light, one running low traffic 
email/antispam, another running monitoring and metrics of ~5 hosts and 
one running a single terminal IRC client. All but the IRC one will hang 
within a few minutes of booting.

When they lock up they only respond to sysrq, with ttyS0/hvc0 not 
echoing anything typed in back, and are completely dead on the network. 
One system managed to report rcu stalls but no backtraces (I'll look 
over the debug config, if there is any interest).

My bare metal desktop has yet to be able to hit it, but it might be 
entirely down to a different type of workload.

Something missing in 4.1?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1234787 — Re: [PATCH 4.1 125/159] net: call rcu_read_lock early in process_backlog

	Hello,

On Tue, 29 Sep 2015, Andre Tomt (LKML) wrote:

> On 26. sep. 2015 22:56, Greg Kroah-Hartman wrote:
> > 4.1-stable review patch.  If anyone has any objections, please let me know.
> >
> > ------------------
> >
> > From: Julian Anastasov <ja@ssi.bg>
> >
> > [ Upstream commit 2c17d27c36dcce2b6bf689f41a46b9e909877c21 ]
> >
> > Incoming packet should be either in backlog queue or
> > in RCU read-side section. Otherwise, the final sequence of
> > flush_backlog() and synchronize_net() may miss packets
> > that can run without device reference:
> <snip>
> 
> Several of our 4.1.9-rc1 running systems are experiencing hangs requiring
> hardware/sysrq reset with this patch applied. Reverting it fixes the hangs
> completely.

	Bad to hear that...

> 4.2 includes this patch as well but I have no such problems there. 4.2.2-rc1
> works fine as well.

	No problem with 4.2+? Same setup/config?

> For now I think this patch should be reverted in 4.1.9.
> 
> The hangs have occured so far on Xen PV and KVM x86_64 virtual machines, they
> will hang completely within minutes or hours depending on the type of
> workload. The workloads are all fairly light, one running low traffic
> email/antispam, another running monitoring and metrics of ~5 hosts and one
> running a single terminal IRC client. All but the IRC one will hang within a
> few minutes of booting.
> 
> When they lock up they only respond to sysrq, with ttyS0/hvc0 not echoing
> anything typed in back, and are completely dead on the network. One system
> managed to report rcu stalls but no backtraces (I'll look over the debug
> config, if there is any interest).
> 
> My bare metal desktop has yet to be able to hit it, but it might be entirely
> down to a different type of workload.
> 
> Something missing in 4.1?

	They are 2 related patches, the first one is
[PATCH 4.1 124/159] net: do not process device backlog during unregistration

	But the problematic patch calls rcu_read_lock while
local IRQ is disabled (in process_backlog), this is something
that should be noted for the patch. I'll try to see what Xen does.
It would be useful to see .config and any kind of backtraces/stalls,
it will help also to other developers to catch the problem...

Regards

--
Julian Anastasov <ja@ssi.bg>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1234961 — Re: [PATCH 4.1 125/159] net: call rcu_read_lock early in process_backlog

On 29. sep. 2015 12:21, Andre Tomt wrote:
> Meanwhile I'll revert both the mentioned net patches and see how it goes.

So that blew up as well, meaning it's not any of these two patches:
[PATCH 4.1 124/159] net: do not process device backlog during unregistration
[PATCH 4.1 125/159] net: call rcu_read_lock early in process_backlog

I'll be offline for a half+ day, I'll look into bisecting when back if
nobody has figured it out by then.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1235656 — Re: [PATCH 4.1 125/159] net: call rcu_read_lock early in process_backlog

On 09/29/2015 12:48 PM, Andre Tomt wrote:
> On 29. sep. 2015 12:21, Andre Tomt wrote:
>> Meanwhile I'll revert both the mentioned net patches and see how it goes.
> 
> So that blew up as well, meaning it's not any of these two patches:
> [PATCH 4.1 124/159] net: do not process device backlog during unregistration
> [PATCH 4.1 125/159] net: call rcu_read_lock early in process_backlog
> 
> I'll be offline for a half+ day, I'll look into bisecting when back if
> nobody has figured it out by then.

After I noticed your crash I did ran some tests with 4.1.9 on physical
host (old core2duo) with kvm guest..

nfs client/server between guest/host + X on guest, next kvm guest to
force host to swap some memory to create memory pressure and all this
with 'perf top' on host machine without any problems..

HTH, Z.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1237296 — kernel 4.1.9: networking hangs with rcu_preempt self-detected stall, 4.1.8 works; was: Re: [PATCH 4.1 125/159] net: call rcu_read_lock early in process_backlog

Am Dienstag, 29. September 2015, 12:48:43 schrieb Andre Tomt:
> On 29. sep. 2015 12:21, Andre Tomt wrote:
> > Meanwhile I'll revert both the mentioned net patches and see how it goes.
> 
> So that blew up as well, meaning it's not any of these two patches:
> [PATCH 4.1 124/159] net: do not process device backlog during unregistration
> [PATCH 4.1 125/159] net: call rcu_read_lock early in process_backlog
> 
> I'll be offline for a half+ day, I'll look into bisecting when back if
> nobody has figured it out by then.
> --
> To unsubscribe from this list: send the line "unsubscribe stable" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

We see these rcu hangs with 4.1.9 on one of our routers, too. 4.1.8 runs fine.

The output I got the last time was:

[ 6488.174578] igb 000:06:00.1 eth3: Reset adapter
[ 6497.350183] INFO: rcu_preempt self-detected stall on CPU { 3}  (t=6301 jiffies g=383330 c=383329 q=1323)
[ 6497.350229] rcu_preempt kthread starved for 6007 jiffies!
[ 6560.311093] INFO: rcu_preempt self-detected stall on CPU { 3}  (t=25205 jiffies g=383330 c=383329 q=4479)
[ 6560.311140] rcu_preempt kthread starved for 24911 jiffies!
[ 6623.272005] INFO: rcu_preempt self-detected stall on CPU { 3}  (t=44109 jiffies g=383330 c=383329 q=7107)
[ 6623.272049] rcu_preempt kthread starved for 43815 jiffies!
[ 6633.053892] igb 000:06:00.0 eth2: Reset adapter
[ 6633.053892] rcu_preempt kthread starved for 62719 jiffies!
[ 6486.232914] INFO: rcu_preempt self-detected stall on CPU { 3}  (t=63013 jiffies g=383330 c=383329 q=8487)
[ 6486.233204] rcu_preempt kthread starved for 6007 jiffies!


All other hangs basically were the same, the cpu varies though.

After that the router completely hangs: networking stops working and we need to restart it.

Regards,
-- 
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1237311 — Re: kernel 4.1.9: networking hangs with rcu_preempt self-detected stall, 4.1.8 works; was: Re: [PATCH 4.1 125/159] net: call rcu_read_lock early in process_backlog

On 10/01/15 13:29, Wolfgang Walter wrote:
> Am Dienstag, 29. September 2015, 12:48:43 schrieb Andre Tomt:
>> On 29. sep. 2015 12:21, Andre Tomt wrote:
>>> Meanwhile I'll revert both the mentioned net patches and see how it goes.
>>
>> So that blew up as well, meaning it's not any of these two patches:
>> [PATCH 4.1 124/159] net: do not process device backlog during unregistration
>> [PATCH 4.1 125/159] net: call rcu_read_lock early in process_backlog
>>
>> I'll be offline for a half+ day, I'll look into bisecting when back if
>> nobody has figured it out by then.
>> --
>> To unsubscribe from this list: send the line "unsubscribe stable" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> We see these rcu hangs with 4.1.9 on one of our routers, too. 4.1.8 runs fine.
> 
> The output I got the last time was:
> 
> [ 6488.174578] igb 000:06:00.1 eth3: Reset adapter
> [ 6497.350183] INFO: rcu_preempt self-detected stall on CPU { 3}  (t=6301 jiffies g=383330 c=383329 q=1323)
> [ 6497.350229] rcu_preempt kthread starved for 6007 jiffies!
> [ 6560.311093] INFO: rcu_preempt self-detected stall on CPU { 3}  (t=25205 jiffies g=383330 c=383329 q=4479)
> [ 6560.311140] rcu_preempt kthread starved for 24911 jiffies!
> [ 6623.272005] INFO: rcu_preempt self-detected stall on CPU { 3}  (t=44109 jiffies g=383330 c=383329 q=7107)
> [ 6623.272049] rcu_preempt kthread starved for 43815 jiffies!
> [ 6633.053892] igb 000:06:00.0 eth2: Reset adapter
> [ 6633.053892] rcu_preempt kthread starved for 62719 jiffies!
> [ 6486.232914] INFO: rcu_preempt self-detected stall on CPU { 3}  (t=63013 jiffies g=383330 c=383329 q=8487)
> [ 6486.233204] rcu_preempt kthread starved for 6007 jiffies!
> 
> 
> All other hangs basically were the same, the cpu varies though.
> 
> After that the router completely hangs: networking stops working and we need to restart it.

Please see the following thread for a suggested solution:
http://www.spinics.net/lists/kernel/msg2087337.html

Please followup there as well.

thanks
Holger

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1235504 — Re: [PATCH 4.1 125/159] net: call rcu_read_lock early in process_backlog

	Hello,

On Tue, 29 Sep 2015, Andre Tomt wrote:

> On 29. sep. 2015 10:39, Andre Tomt (LKML) wrote:
> > I just had another hang with it reverted on two different guests..
> > However it took nearly 6 hours rather than the usual "few minutes" for
> > these two. So now I'm a little unsure about my initial conclusions.
> > 
> > On 29. sep. 2015 09:40, Julian Anastasov wrote:
> >> On Tue, 29 Sep 2015, Andre Tomt (LKML) wrote:
> <snip>
> >> 	They are 2 related patches, the first one is
> >> [PATCH 4.1 124/159] net: do not process device backlog during unregistration
> > 
> > Would reverting this change anything outside device unregistration at all?

	Its role is only during unregistration, so it
should not matter.

> I enabled CONFIG_RCU_CPU_STALL_INFO=y and disabled a bunch of non-virt
> drivers to speed up debugging. But no output this time either. Got any
> ideas on debugging options I've forgot? Useful sysrqs?

	Checking my .config for debugs... I'm not expert
on this but may be such settings can help:

CONFIG_PROVE_RCU=y
CONFIG_PROVE_RCU_REPEATEDLY=y
CONFIG_SPARSE_RCU_POINTER=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_PROVE_LOCKING=y
CONFIG_LOCKDEP=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_TRACE_IRQFLAGS=y
CONFIG_STACKTRACE=y - you have this one
CONFIG_DEBUG_BUGVERBOSE=y

Regards

--
Julian Anastasov <ja@ssi.bg>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web