Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1218294 > unrolled thread

[PATCH] net: tipc: fix stall during bclink wakeup procedure

Started byKolmakov Dmitriy <kolmakov.dmitriy@huawei.com>
First post2015-09-03 16:40 +0200
Last post2015-09-09 08:00 +0200
Articles 6 — 4 participants

Back to article view | Back to linux.kernel

Contents

  [PATCH] net: tipc: fix stall during bclink wakeup procedure Kolmakov Dmitriy <kolmakov.dmitriy@huawei.com> - 2015-09-03 16:40 +0200
    RE: [PATCH] net: tipc: fix stall during bclink wakeup procedure Jon Maloy <jon.maloy@ericsson.com> - 2015-09-03 18:10 +0200
      RE: [PATCH] net: tipc: fix stall during bclink wakeup procedure Jon Maloy <jon.maloy@ericsson.com> - 2015-09-04 15:00 +0200
    Re: [PATCH] net: tipc: fix stall during bclink wakeup procedure Ying Xue <ying.xue@windriver.com> - 2015-09-06 10:10 +0200
      RE: [PATCH] net: tipc: fix stall during bclink wakeup procedure Kolmakov Dmitriy <kolmakov.dmitriy@huawei.com> - 2015-09-07 11:10 +0200
        Re: [PATCH] net: tipc: fix stall during bclink wakeup procedure David Miller <davem@davemloft.net> - 2015-09-09 08:00 +0200

#1218294 — [PATCH] net: tipc: fix stall during bclink wakeup procedure

FromKolmakov Dmitriy <kolmakov.dmitriy@huawei.com>
Date2015-09-03 16:40 +0200
Subject[PATCH] net: tipc: fix stall during bclink wakeup procedure
Message-ID<q4DQe-6kK-25@gated-at.bofh.it>
From: Dmitry S Kolmakov <kolmakov.dmitriy@huawei.com>
 
If an attempt to wake up users of broadcast link is made when there is
no enough place in send queue than it may hang up inside the
tipc_sk_rcv() function since the loop breaks only after the wake up
queue becomes empty. This can lead to complete CPU stall with the
following message generated by RCU:
 
INFO: rcu_sched self-detected stall on CPU { 0}  (t=2101 jiffies g=54225 c=54224 q=11465)
Task dump for CPU 0:
tpch            R  running task        0 39949  39948 0x0000000a
 ffffffff818536c0 ffff88181fa037a0 ffffffff8106a4be 0000000000000000
 ffffffff818536c0 ffff88181fa037c0 ffffffff8106d8a8 ffff88181fa03800
 0000000000000001 ffff88181fa037f0 ffffffff81094a50 ffff88181fa15680
Call Trace:
 <IRQ>  [<ffffffff8106a4be>] sched_show_task+0xae/0x120
 [<ffffffff8106d8a8>] dump_cpu_task+0x38/0x40
 [<ffffffff81094a50>] rcu_dump_cpu_stacks+0x90/0xd0
 [<ffffffff81097c3b>] rcu_check_callbacks+0x3eb/0x6e0
 [<ffffffff8106e53f>] ? account_system_time+0x7f/0x170
 [<ffffffff81099e64>] update_process_times+0x34/0x60
 [<ffffffff810a84d1>] tick_sched_handle.isra.18+0x31/0x40
 [<ffffffff810a851c>] tick_sched_timer+0x3c/0x70
 [<ffffffff8109a43d>] __run_hrtimer.isra.34+0x3d/0xc0
 [<ffffffff8109aa95>] hrtimer_interrupt+0xc5/0x1e0
 [<ffffffff81030d52>] ? native_smp_send_reschedule+0x42/0x60
 [<ffffffff81032f04>] local_apic_timer_interrupt+0x34/0x60
 [<ffffffff810335bc>] smp_apic_timer_interrupt+0x3c/0x60
 [<ffffffff8165a3fb>] apic_timer_interrupt+0x6b/0x70
 [<ffffffff81659129>] ? _raw_spin_unlock_irqrestore+0x9/0x10
 [<ffffffff8107eb9f>] __wake_up_sync_key+0x4f/0x60
 [<ffffffffa313ddd1>] tipc_write_space+0x31/0x40 [tipc]
 [<ffffffffa313dadf>] filter_rcv+0x31f/0x520 [tipc]
 [<ffffffffa313d699>] ? tipc_sk_lookup+0xc9/0x110 [tipc]
 [<ffffffff81659259>] ? _raw_spin_lock_bh+0x19/0x30
 [<ffffffffa314122c>] tipc_sk_rcv+0x2dc/0x3e0 [tipc]
 [<ffffffffa312e7ff>] tipc_bclink_wakeup_users+0x2f/0x40 [tipc]
 [<ffffffffa313ce26>] tipc_node_unlock+0x186/0x190 [tipc]
 [<ffffffff81597c1c>] ? kfree_skb+0x2c/0x40
 [<ffffffffa313475c>] tipc_rcv+0x2ac/0x8c0 [tipc]
 [<ffffffffa312ff58>] tipc_l2_rcv_msg+0x38/0x50 [tipc]
 [<ffffffff815a76d3>] __netif_receive_skb_core+0x5a3/0x950
 [<ffffffff815a98d3>] __netif_receive_skb+0x13/0x60
 [<ffffffff815a993e>] netif_receive_skb_internal+0x1e/0x90
 [<ffffffff815aa138>] napi_gro_receive+0x78/0xa0
 [<ffffffffa07f93f4>] tg3_poll_work+0xc54/0xf40 [tg3]
 [<ffffffff81597c8c>] ? consume_skb+0x2c/0x40
 [<ffffffffa07f9721>] tg3_poll_msix+0x41/0x160 [tg3]
 [<ffffffff815ab0f2>] net_rx_action+0xe2/0x290
 [<ffffffff8104b92a>] __do_softirq+0xda/0x1f0
 [<ffffffff8104bc26>] irq_exit+0x76/0xa0
 [<ffffffff81004355>] do_IRQ+0x55/0xf0
 [<ffffffff8165a12b>] common_interrupt+0x6b/0x6b
 <EOI>

The issue occurs only when tipc_sk_rcv() is used to wake up postponed senders:

	tipc_bclink_wakeup_users()
		// wakeupq - is a queue which consists of special 
		// 		 messages with SOCK_WAKEUP type. 
		tipc_sk_rcv(wakeupq)
			...
			while (skb_queue_len(inputq)) {
				filter_rcv(skb)
					// Here the type of message is checked 
					// and if it is SOCK_WAKEUP than
					// it tries to wake up a sender.
					tipc_write_space(sk)
						wake_up_interruptible_sync_poll()
			}
				
After the sender thread is woke up it can gather control and perform an attempt to send a message. But if there is no enough place in send queue it will call link_schedule_user() function which puts a message of type SOCK_WAKEUP to the wakeup queue and put the sender to sleep. Thus the size of the queue actually is not changed and the while() loop never exits. 

The approach I proposed is to wake up only senders for which there is enough place in send queue so the described issue can't occur. Moreover the same approach is already used to wake up senders on unicast links.

I have got into the issue on our product code but to reproduce the issue I changed a benchmark test application (from tipcutils/demos/benchmark) to perform the following scenario:
	1. Run 64 instances of test application (nodes). It can be done on the one physical machine.
	2. Each application connects to all other using TIPC sockets in RDM mode.
	3. When setup is done all nodes start simultaneously send broadcast messages. 
	4. Everything hangs up.

The issue is reproducible only when a congestion on broadcast link occurs. For example, when there are only 8 nodes it works fine since congestion doesn't occur. Send queue limit is 40 in my case (I use a critical importance level) and when 64 nodes send a message at the same moment a congestion occurs every time.

Signed-off-by: Dmitry S Kolmakov <kolmakov.dmitriy@huawei.com>
---
diff --git a/net/tipc/bcast.c b/net/tipc/bcast.c
index c5cbdcb..997dd60 100644
--- a/net/tipc/bcast.c
+++ b/net/tipc/bcast.c
@@ -169,6 +169,30 @@ static void bclink_retransmit_pkt(struct tipc_net *tn, u32 after, u32 to)
 }
 
 /**
+ * bclink_prepare_wakeup - prepare users for wakeup after congestion
+ * @bcl: broadcast link
+ * @resultq: queue for users which can be woken up
+ * Move a number of waiting users, as permitted by available space in
+ * the send queue, from link wait queue to specified queue for wakeup
+ */
+static void bclink_prepare_wakeup(struct tipc_link *bcl, struct sk_buff_head *resultq)
+{
+	int pnd[TIPC_SYSTEM_IMPORTANCE + 1] = {0,};
+	int imp, lim;
+	struct sk_buff *skb, *tmp;
+
+	skb_queue_walk_safe(&bcl->wakeupq, skb, tmp) {
+		imp = TIPC_SKB_CB(skb)->chain_imp;
+		lim = bcl->window + bcl->backlog[imp].limit;
+		pnd[imp] += TIPC_SKB_CB(skb)->chain_sz;
+		if ((pnd[imp] + bcl->backlog[imp].len) >= lim)
+			continue;
+		skb_unlink(skb, &bcl->wakeupq);
+		skb_queue_tail(resultq, skb);
+	}
+}
+
+/**
  * tipc_bclink_wakeup_users - wake up pending users
  *
  * Called with no locks taken
@@ -176,8 +200,12 @@ static void bclink_retransmit_pkt(struct tipc_net *tn, u32 after, u32 to)
 void tipc_bclink_wakeup_users(struct net *net)
 {
 	struct tipc_net *tn = net_generic(net, tipc_net_id);
+	struct tipc_link *bcl = tn->bcl;
+	struct sk_buff_head resultq;
 
-	tipc_sk_rcv(net, &tn->bclink->link.wakeupq);
+	skb_queue_head_init(&resultq);
+	bclink_prepare_wakeup(bcl, &resultq);
+	tipc_sk_rcv(net, &resultq);
 }
 
 /**
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1218411

FromJon Maloy <jon.maloy@ericsson.com>
Date2015-09-03 18:10 +0200
Message-ID<q4Ffk-8tV-31@gated-at.bofh.it>
In reply to#1218294

> -----Original Message-----
> From: Kolmakov Dmitriy [mailto:kolmakov.dmitriy@huawei.com]
> Sent: Thursday, 03 September, 2015 10:39
> To: davem@davemloft.net
> Cc: Jon Maloy; Ying Xue; tipc-discussion@lists.sourceforge.net;
> netdev@vger.kernel.org; linux-kernel@vger.kernel.org
> Subject: [PATCH] net: tipc: fix stall during bclink wakeup procedure
> 
> From: Dmitry S Kolmakov <kolmakov.dmitriy@huawei.com>
> 
> If an attempt to wake up users of broadcast link is made when there is no
> enough place in send queue than it may hang up inside the
> tipc_sk_rcv() function since the loop breaks only after the wake up queue
> becomes empty. This can lead to complete CPU stall with the following
> message generated by RCU:
> 
> INFO: rcu_sched self-detected stall on CPU { 0}  (t=2101 jiffies g=54225
> c=54224 q=11465) Task dump for CPU 0:
> tpch            R  running task        0 39949  39948 0x0000000a
>  ffffffff818536c0 ffff88181fa037a0 ffffffff8106a4be 0000000000000000
>  ffffffff818536c0 ffff88181fa037c0 ffffffff8106d8a8 ffff88181fa03800
>  0000000000000001 ffff88181fa037f0 ffffffff81094a50 ffff88181fa15680 Call
> Trace:
>  <IRQ>  [<ffffffff8106a4be>] sched_show_task+0xae/0x120
> [<ffffffff8106d8a8>] dump_cpu_task+0x38/0x40  [<ffffffff81094a50>]
> rcu_dump_cpu_stacks+0x90/0xd0  [<ffffffff81097c3b>]
> rcu_check_callbacks+0x3eb/0x6e0  [<ffffffff8106e53f>] ?
> account_system_time+0x7f/0x170  [<ffffffff81099e64>]
> update_process_times+0x34/0x60  [<ffffffff810a84d1>]
> tick_sched_handle.isra.18+0x31/0x40
>  [<ffffffff810a851c>] tick_sched_timer+0x3c/0x70  [<ffffffff8109a43d>]
> __run_hrtimer.isra.34+0x3d/0xc0  [<ffffffff8109aa95>]
> hrtimer_interrupt+0xc5/0x1e0  [<ffffffff81030d52>] ?
> native_smp_send_reschedule+0x42/0x60
>  [<ffffffff81032f04>] local_apic_timer_interrupt+0x34/0x60
>  [<ffffffff810335bc>] smp_apic_timer_interrupt+0x3c/0x60
>  [<ffffffff8165a3fb>] apic_timer_interrupt+0x6b/0x70  [<ffffffff81659129>] ?
> _raw_spin_unlock_irqrestore+0x9/0x10
>  [<ffffffff8107eb9f>] __wake_up_sync_key+0x4f/0x60  [<ffffffffa313ddd1>]
> tipc_write_space+0x31/0x40 [tipc]  [<ffffffffa313dadf>]
> filter_rcv+0x31f/0x520 [tipc]  [<ffffffffa313d699>] ?
> tipc_sk_lookup+0xc9/0x110 [tipc]  [<ffffffff81659259>] ?
> _raw_spin_lock_bh+0x19/0x30  [<ffffffffa314122c>]
> tipc_sk_rcv+0x2dc/0x3e0 [tipc]  [<ffffffffa312e7ff>]
> tipc_bclink_wakeup_users+0x2f/0x40 [tipc]  [<ffffffffa313ce26>]
> tipc_node_unlock+0x186/0x190 [tipc]  [<ffffffff81597c1c>] ?
> kfree_skb+0x2c/0x40  [<ffffffffa313475c>] tipc_rcv+0x2ac/0x8c0 [tipc]
> [<ffffffffa312ff58>] tipc_l2_rcv_msg+0x38/0x50 [tipc]  [<ffffffff815a76d3>]
> __netif_receive_skb_core+0x5a3/0x950
>  [<ffffffff815a98d3>] __netif_receive_skb+0x13/0x60  [<ffffffff815a993e>]
> netif_receive_skb_internal+0x1e/0x90
>  [<ffffffff815aa138>] napi_gro_receive+0x78/0xa0  [<ffffffffa07f93f4>]
> tg3_poll_work+0xc54/0xf40 [tg3]  [<ffffffff81597c8c>] ?
> consume_skb+0x2c/0x40  [<ffffffffa07f9721>] tg3_poll_msix+0x41/0x160
> [tg3]  [<ffffffff815ab0f2>] net_rx_action+0xe2/0x290  [<ffffffff8104b92a>]
> __do_softirq+0xda/0x1f0  [<ffffffff8104bc26>] irq_exit+0x76/0xa0
> [<ffffffff81004355>] do_IRQ+0x55/0xf0  [<ffffffff8165a12b>]
> common_interrupt+0x6b/0x6b  <EOI>
> 
> The issue occurs only when tipc_sk_rcv() is used to wake up postponed
> senders:
> 
> 	tipc_bclink_wakeup_users()
> 		// wakeupq - is a queue which consists of special
> 		// 		 messages with SOCK_WAKEUP type.
> 		tipc_sk_rcv(wakeupq)
> 			...
> 			while (skb_queue_len(inputq)) {
> 				filter_rcv(skb)
> 					// Here the type of message is
> checked
> 					// and if it is SOCK_WAKEUP than
> 					// it tries to wake up a sender.
> 					tipc_write_space(sk)
> 
> 	wake_up_interruptible_sync_poll()
> 			}
> 
> After the sender thread is woke up it can gather control and perform an
> attempt to send a message. But if there is no enough place in send queue it
> will call link_schedule_user() function which puts a message of type
> SOCK_WAKEUP to the wakeup queue and put the sender to sleep. Thus the
> size of the queue actually is not changed and the while() loop never exits.
> 
> The approach I proposed is to wake up only senders for which there is
> enough place in send queue so the described issue can't occur. Moreover
> the same approach is already used to wake up senders on unicast links.

I looked closer at the code, and I don't see how you can enter into this loop.
SOCK_WAKEP is only issued if buffers actually have been released from the
transmit queue, so sooner or later there should be space in the queue for
any sender. I am starting to suspect that the root of this problem is elsewhere.

Maybe we should continue this thread at tipc-dicussion, so we don't pollute
the netdev list with our internal discussions?

///jon

> 
> I have got into the issue on our product code but to reproduce the issue I
> changed a benchmark test application (from tipcutils/demos/benchmark) to
> perform the following scenario:
> 	1. Run 64 instances of test application (nodes). It can be done on the
> one physical machine.
> 	2. Each application connects to all other using TIPC sockets in RDM
> mode.
> 	3. When setup is done all nodes start simultaneously send broadcast
> messages.
> 	4. Everything hangs up.
> 
> The issue is reproducible only when a congestion on broadcast link occurs.
> For example, when there are only 8 nodes it works fine since congestion
> doesn't occur. Send queue limit is 40 in my case (I use a critical importance
> level) and when 64 nodes send a message at the same moment a congestion
> occurs every time.
> 
> Signed-off-by: Dmitry S Kolmakov <kolmakov.dmitriy@huawei.com>
> ---
> diff --git a/net/tipc/bcast.c b/net/tipc/bcast.c index c5cbdcb..997dd60 100644
> --- a/net/tipc/bcast.c
> +++ b/net/tipc/bcast.c
> @@ -169,6 +169,30 @@ static void bclink_retransmit_pkt(struct tipc_net *tn,
> u32 after, u32 to)  }
> 
>  /**
> + * bclink_prepare_wakeup - prepare users for wakeup after congestion
> + * @bcl: broadcast link
> + * @resultq: queue for users which can be woken up
> + * Move a number of waiting users, as permitted by available space in
> + * the send queue, from link wait queue to specified queue for wakeup
> +*/ static void bclink_prepare_wakeup(struct tipc_link *bcl, struct
> +sk_buff_head *resultq) {
> +	int pnd[TIPC_SYSTEM_IMPORTANCE + 1] = {0,};
> +	int imp, lim;
> +	struct sk_buff *skb, *tmp;
> +
> +	skb_queue_walk_safe(&bcl->wakeupq, skb, tmp) {
> +		imp = TIPC_SKB_CB(skb)->chain_imp;
> +		lim = bcl->window + bcl->backlog[imp].limit;
> +		pnd[imp] += TIPC_SKB_CB(skb)->chain_sz;
> +		if ((pnd[imp] + bcl->backlog[imp].len) >= lim)
> +			continue;
> +		skb_unlink(skb, &bcl->wakeupq);
> +		skb_queue_tail(resultq, skb);
> +	}
> +}
> +
> +/**
>   * tipc_bclink_wakeup_users - wake up pending users
>   *
>   * Called with no locks taken
> @@ -176,8 +200,12 @@ static void bclink_retransmit_pkt(struct tipc_net *tn,
> u32 after, u32 to)  void tipc_bclink_wakeup_users(struct net *net)  {
>  	struct tipc_net *tn = net_generic(net, tipc_net_id);
> +	struct tipc_link *bcl = tn->bcl;
> +	struct sk_buff_head resultq;
> 
> -	tipc_sk_rcv(net, &tn->bclink->link.wakeupq);
> +	skb_queue_head_init(&resultq);
> +	bclink_prepare_wakeup(bcl, &resultq);
> +	tipc_sk_rcv(net, &resultq);
>  }
> 
>  /**
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1218902

FromJon Maloy <jon.maloy@ericsson.com>
Date2015-09-04 15:00 +0200
Message-ID<q4YL0-2vD-3@gated-at.bofh.it>
In reply to#1218411
Reviewed again, and finally understood. 
You are right;  I just didn't understand the problem description correctly.

Reviewed-by: Jon Maloy <jon.maloy@ericsson.com>

///jon

> -----Original Message-----
> From: netdev-owner@vger.kernel.org [mailto:netdev-
> owner@vger.kernel.org] On Behalf Of Jon Maloy
> Sent: Thursday, 03 September, 2015 12:07
> To: Kolmakov Dmitriy; davem@davemloft.net
> Cc: Ying Xue; tipc-discussion@lists.sourceforge.net; netdev@vger.kernel.org;
> linux-kernel@vger.kernel.org
> Subject: RE: [PATCH] net: tipc: fix stall during bclink wakeup procedure
> 
> 
> 
> > -----Original Message-----
> > From: Kolmakov Dmitriy [mailto:kolmakov.dmitriy@huawei.com]
> > Sent: Thursday, 03 September, 2015 10:39
> > To: davem@davemloft.net
> > Cc: Jon Maloy; Ying Xue; tipc-discussion@lists.sourceforge.net;
> > netdev@vger.kernel.org; linux-kernel@vger.kernel.org
> > Subject: [PATCH] net: tipc: fix stall during bclink wakeup procedure
> >
> > From: Dmitry S Kolmakov <kolmakov.dmitriy@huawei.com>
> >
> > If an attempt to wake up users of broadcast link is made when there is
> > no enough place in send queue than it may hang up inside the
> > tipc_sk_rcv() function since the loop breaks only after the wake up
> > queue becomes empty. This can lead to complete CPU stall with the
> > following message generated by RCU:
> >
> > INFO: rcu_sched self-detected stall on CPU { 0}  (t=2101 jiffies
> > g=54225
> > c=54224 q=11465) Task dump for CPU 0:
> > tpch            R  running task        0 39949  39948 0x0000000a
> >  ffffffff818536c0 ffff88181fa037a0 ffffffff8106a4be 0000000000000000
> >  ffffffff818536c0 ffff88181fa037c0 ffffffff8106d8a8 ffff88181fa03800
> >  0000000000000001 ffff88181fa037f0 ffffffff81094a50 ffff88181fa15680
> > Call
> > Trace:
> >  <IRQ>  [<ffffffff8106a4be>] sched_show_task+0xae/0x120
> > [<ffffffff8106d8a8>] dump_cpu_task+0x38/0x40  [<ffffffff81094a50>]
> > rcu_dump_cpu_stacks+0x90/0xd0  [<ffffffff81097c3b>]
> > rcu_check_callbacks+0x3eb/0x6e0  [<ffffffff8106e53f>] ?
> > account_system_time+0x7f/0x170  [<ffffffff81099e64>]
> > update_process_times+0x34/0x60  [<ffffffff810a84d1>]
> > tick_sched_handle.isra.18+0x31/0x40
> >  [<ffffffff810a851c>] tick_sched_timer+0x3c/0x70  [<ffffffff8109a43d>]
> > __run_hrtimer.isra.34+0x3d/0xc0  [<ffffffff8109aa95>]
> > hrtimer_interrupt+0xc5/0x1e0  [<ffffffff81030d52>] ?
> > native_smp_send_reschedule+0x42/0x60
> >  [<ffffffff81032f04>] local_apic_timer_interrupt+0x34/0x60
> >  [<ffffffff810335bc>] smp_apic_timer_interrupt+0x3c/0x60
> >  [<ffffffff8165a3fb>] apic_timer_interrupt+0x6b/0x70  [<ffffffff81659129>]
> ?
> > _raw_spin_unlock_irqrestore+0x9/0x10
> >  [<ffffffff8107eb9f>] __wake_up_sync_key+0x4f/0x60
> > [<ffffffffa313ddd1>]
> > tipc_write_space+0x31/0x40 [tipc]  [<ffffffffa313dadf>]
> > filter_rcv+0x31f/0x520 [tipc]  [<ffffffffa313d699>] ?
> > tipc_sk_lookup+0xc9/0x110 [tipc]  [<ffffffff81659259>] ?
> > _raw_spin_lock_bh+0x19/0x30  [<ffffffffa314122c>]
> > tipc_sk_rcv+0x2dc/0x3e0 [tipc]  [<ffffffffa312e7ff>]
> > tipc_bclink_wakeup_users+0x2f/0x40 [tipc]  [<ffffffffa313ce26>]
> > tipc_node_unlock+0x186/0x190 [tipc]  [<ffffffff81597c1c>] ?
> > kfree_skb+0x2c/0x40  [<ffffffffa313475c>] tipc_rcv+0x2ac/0x8c0 [tipc]
> > [<ffffffffa312ff58>] tipc_l2_rcv_msg+0x38/0x50 [tipc]
> > [<ffffffff815a76d3>]
> > __netif_receive_skb_core+0x5a3/0x950
> >  [<ffffffff815a98d3>] __netif_receive_skb+0x13/0x60
> > [<ffffffff815a993e>]
> > netif_receive_skb_internal+0x1e/0x90
> >  [<ffffffff815aa138>] napi_gro_receive+0x78/0xa0  [<ffffffffa07f93f4>]
> > tg3_poll_work+0xc54/0xf40 [tg3]  [<ffffffff81597c8c>] ?
> > consume_skb+0x2c/0x40  [<ffffffffa07f9721>] tg3_poll_msix+0x41/0x160
> > [tg3]  [<ffffffff815ab0f2>] net_rx_action+0xe2/0x290
> > [<ffffffff8104b92a>]
> > __do_softirq+0xda/0x1f0  [<ffffffff8104bc26>] irq_exit+0x76/0xa0
> > [<ffffffff81004355>] do_IRQ+0x55/0xf0  [<ffffffff8165a12b>]
> > common_interrupt+0x6b/0x6b  <EOI>
> >
> > The issue occurs only when tipc_sk_rcv() is used to wake up postponed
> > senders:
> >
> > 	tipc_bclink_wakeup_users()
> > 		// wakeupq - is a queue which consists of special
> > 		// 		 messages with SOCK_WAKEUP type.
> > 		tipc_sk_rcv(wakeupq)
> > 			...
> > 			while (skb_queue_len(inputq)) {
> > 				filter_rcv(skb)
> > 					// Here the type of message is
> > checked
> > 					// and if it is SOCK_WAKEUP than
> > 					// it tries to wake up a sender.
> > 					tipc_write_space(sk)
> >
> > 	wake_up_interruptible_sync_poll()
> > 			}
> >
> > After the sender thread is woke up it can gather control and perform
> > an attempt to send a message. But if there is no enough place in send
> > queue it will call link_schedule_user() function which puts a message
> > of type SOCK_WAKEUP to the wakeup queue and put the sender to sleep.
> > Thus the size of the queue actually is not changed and the while() loop
> never exits.
> >
> > The approach I proposed is to wake up only senders for which there is
> > enough place in send queue so the described issue can't occur.
> > Moreover the same approach is already used to wake up senders on
> unicast links.
> 
> I looked closer at the code, and I don't see how you can enter into this loop.
> SOCK_WAKEP is only issued if buffers actually have been released from the
> transmit queue, so sooner or later there should be space in the queue for
> any sender. I am starting to suspect that the root of this problem is
> elsewhere.
> 
> Maybe we should continue this thread at tipc-dicussion, so we don't pollute
> the netdev list with our internal discussions?
> 
> ///jon
> 
> >
> > I have got into the issue on our product code but to reproduce the
> > issue I changed a benchmark test application (from
> > tipcutils/demos/benchmark) to perform the following scenario:
> > 	1. Run 64 instances of test application (nodes). It can be done on
> > the one physical machine.
> > 	2. Each application connects to all other using TIPC sockets in RDM
> > mode.
> > 	3. When setup is done all nodes start simultaneously send broadcast
> > messages.
> > 	4. Everything hangs up.
> >
> > The issue is reproducible only when a congestion on broadcast link occurs.
> > For example, when there are only 8 nodes it works fine since
> > congestion doesn't occur. Send queue limit is 40 in my case (I use a
> > critical importance
> > level) and when 64 nodes send a message at the same moment a
> > congestion occurs every time.
> >
> > Signed-off-by: Dmitry S Kolmakov <kolmakov.dmitriy@huawei.com>
> > ---
> > diff --git a/net/tipc/bcast.c b/net/tipc/bcast.c index
> > c5cbdcb..997dd60 100644
> > --- a/net/tipc/bcast.c
> > +++ b/net/tipc/bcast.c
> > @@ -169,6 +169,30 @@ static void bclink_retransmit_pkt(struct tipc_net
> > *tn,
> > u32 after, u32 to)  }
> >
> >  /**
> > + * bclink_prepare_wakeup - prepare users for wakeup after congestion
> > + * @bcl: broadcast link
> > + * @resultq: queue for users which can be woken up
> > + * Move a number of waiting users, as permitted by available space in
> > + * the send queue, from link wait queue to specified queue for wakeup
> > +*/ static void bclink_prepare_wakeup(struct tipc_link *bcl, struct
> > +sk_buff_head *resultq) {
> > +	int pnd[TIPC_SYSTEM_IMPORTANCE + 1] = {0,};
> > +	int imp, lim;
> > +	struct sk_buff *skb, *tmp;
> > +
> > +	skb_queue_walk_safe(&bcl->wakeupq, skb, tmp) {
> > +		imp = TIPC_SKB_CB(skb)->chain_imp;
> > +		lim = bcl->window + bcl->backlog[imp].limit;
> > +		pnd[imp] += TIPC_SKB_CB(skb)->chain_sz;
> > +		if ((pnd[imp] + bcl->backlog[imp].len) >= lim)
> > +			continue;
> > +		skb_unlink(skb, &bcl->wakeupq);
> > +		skb_queue_tail(resultq, skb);
> > +	}
> > +}
> > +
> > +/**
> >   * tipc_bclink_wakeup_users - wake up pending users
> >   *
> >   * Called with no locks taken
> > @@ -176,8 +200,12 @@ static void bclink_retransmit_pkt(struct tipc_net
> > *tn,
> > u32 after, u32 to)  void tipc_bclink_wakeup_users(struct net *net)  {
> >  	struct tipc_net *tn = net_generic(net, tipc_net_id);
> > +	struct tipc_link *bcl = tn->bcl;
> > +	struct sk_buff_head resultq;
> >
> > -	tipc_sk_rcv(net, &tn->bclink->link.wakeupq);
> > +	skb_queue_head_init(&resultq);
> > +	bclink_prepare_wakeup(bcl, &resultq);
> > +	tipc_sk_rcv(net, &resultq);
> >  }
> >
> >  /**
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in the body
> of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1219773

FromYing Xue <ying.xue@windriver.com>
Date2015-09-06 10:10 +0200
Message-ID<q5Dbr-1p9-9@gated-at.bofh.it>
In reply to#1218294
On 09/03/2015 10:39 PM, Kolmakov Dmitriy wrote:
> From: Dmitry S Kolmakov <kolmakov.dmitriy@huawei.com>
>  
> If an attempt to wake up users of broadcast link is made when there is
> no enough place in send queue than it may hang up inside the
> tipc_sk_rcv() function since the loop breaks only after the wake up
> queue becomes empty. This can lead to complete CPU stall with the
> following message generated by RCU:
>  
> INFO: rcu_sched self-detected stall on CPU { 0}  (t=2101 jiffies g=54225 c=54224 q=11465)
> Task dump for CPU 0:
> tpch            R  running task        0 39949  39948 0x0000000a
>  ffffffff818536c0 ffff88181fa037a0 ffffffff8106a4be 0000000000000000
>  ffffffff818536c0 ffff88181fa037c0 ffffffff8106d8a8 ffff88181fa03800
>  0000000000000001 ffff88181fa037f0 ffffffff81094a50 ffff88181fa15680
> Call Trace:
>  <IRQ>  [<ffffffff8106a4be>] sched_show_task+0xae/0x120
>  [<ffffffff8106d8a8>] dump_cpu_task+0x38/0x40
>  [<ffffffff81094a50>] rcu_dump_cpu_stacks+0x90/0xd0
>  [<ffffffff81097c3b>] rcu_check_callbacks+0x3eb/0x6e0
>  [<ffffffff8106e53f>] ? account_system_time+0x7f/0x170
>  [<ffffffff81099e64>] update_process_times+0x34/0x60
>  [<ffffffff810a84d1>] tick_sched_handle.isra.18+0x31/0x40
>  [<ffffffff810a851c>] tick_sched_timer+0x3c/0x70
>  [<ffffffff8109a43d>] __run_hrtimer.isra.34+0x3d/0xc0
>  [<ffffffff8109aa95>] hrtimer_interrupt+0xc5/0x1e0
>  [<ffffffff81030d52>] ? native_smp_send_reschedule+0x42/0x60
>  [<ffffffff81032f04>] local_apic_timer_interrupt+0x34/0x60
>  [<ffffffff810335bc>] smp_apic_timer_interrupt+0x3c/0x60
>  [<ffffffff8165a3fb>] apic_timer_interrupt+0x6b/0x70
>  [<ffffffff81659129>] ? _raw_spin_unlock_irqrestore+0x9/0x10
>  [<ffffffff8107eb9f>] __wake_up_sync_key+0x4f/0x60
>  [<ffffffffa313ddd1>] tipc_write_space+0x31/0x40 [tipc]
>  [<ffffffffa313dadf>] filter_rcv+0x31f/0x520 [tipc]
>  [<ffffffffa313d699>] ? tipc_sk_lookup+0xc9/0x110 [tipc]
>  [<ffffffff81659259>] ? _raw_spin_lock_bh+0x19/0x30
>  [<ffffffffa314122c>] tipc_sk_rcv+0x2dc/0x3e0 [tipc]
>  [<ffffffffa312e7ff>] tipc_bclink_wakeup_users+0x2f/0x40 [tipc]
>  [<ffffffffa313ce26>] tipc_node_unlock+0x186/0x190 [tipc]
>  [<ffffffff81597c1c>] ? kfree_skb+0x2c/0x40
>  [<ffffffffa313475c>] tipc_rcv+0x2ac/0x8c0 [tipc]
>  [<ffffffffa312ff58>] tipc_l2_rcv_msg+0x38/0x50 [tipc]
>  [<ffffffff815a76d3>] __netif_receive_skb_core+0x5a3/0x950
>  [<ffffffff815a98d3>] __netif_receive_skb+0x13/0x60
>  [<ffffffff815a993e>] netif_receive_skb_internal+0x1e/0x90
>  [<ffffffff815aa138>] napi_gro_receive+0x78/0xa0
>  [<ffffffffa07f93f4>] tg3_poll_work+0xc54/0xf40 [tg3]
>  [<ffffffff81597c8c>] ? consume_skb+0x2c/0x40
>  [<ffffffffa07f9721>] tg3_poll_msix+0x41/0x160 [tg3]
>  [<ffffffff815ab0f2>] net_rx_action+0xe2/0x290
>  [<ffffffff8104b92a>] __do_softirq+0xda/0x1f0
>  [<ffffffff8104bc26>] irq_exit+0x76/0xa0
>  [<ffffffff81004355>] do_IRQ+0x55/0xf0
>  [<ffffffff8165a12b>] common_interrupt+0x6b/0x6b
>  <EOI>
> 
> The issue occurs only when tipc_sk_rcv() is used to wake up postponed senders:
> 
> 	tipc_bclink_wakeup_users()
> 		// wakeupq - is a queue which consists of special 
> 		// 		 messages with SOCK_WAKEUP type. 
> 		tipc_sk_rcv(wakeupq)
> 			...
> 			while (skb_queue_len(inputq)) {
> 				filter_rcv(skb)
> 					// Here the type of message is checked 
> 					// and if it is SOCK_WAKEUP than

s/than/then

> 					// it tries to wake up a sender.
> 					tipc_write_space(sk)
> 						wake_up_interruptible_sync_poll()
> 			}
> 				
> After the sender thread is woke up it can gather control and perform an attempt to send a message. But if there is no enough place in send queue it will call link_schedule_user() function which puts a message of type SOCK_WAKEUP to the wakeup queue and put the sender to sleep. Thus the size of the queue actually is not changed and the while() loop never exits. 
> 
> The approach I proposed is to wake up only senders for which there is enough place in send queue so the described issue can't occur. Moreover the same approach is already used to wake up senders on unicast links.
> 
> I have got into the issue on our product code but to reproduce the issue I changed a benchmark test application (from tipcutils/demos/benchmark) to perform the following scenario:
> 	1. Run 64 instances of test application (nodes). It can be done on the one physical machine.
> 	2. Each application connects to all other using TIPC sockets in RDM mode.
> 	3. When setup is done all nodes start simultaneously send broadcast messages. 
> 	4. Everything hangs up.
> 
> The issue is reproducible only when a congestion on broadcast link occurs. For example, when there are only 8 nodes it works fine since congestion doesn't occur. Send queue limit is 40 in my case (I use a critical importance level) and when 64 nodes send a message at the same moment a congestion occurs every time.
> 

Please don't let above maximum line length exceed 70 chars.

Acked-by: Ying Xue <ying.xue@windriver.com>

> Signed-off-by: Dmitry S Kolmakov <kolmakov.dmitriy@huawei.com>
> ---
> diff --git a/net/tipc/bcast.c b/net/tipc/bcast.c
> index c5cbdcb..997dd60 100644
> --- a/net/tipc/bcast.c
> +++ b/net/tipc/bcast.c
> @@ -169,6 +169,30 @@ static void bclink_retransmit_pkt(struct tipc_net *tn, u32 after, u32 to)
>  }
>  
>  /**
> + * bclink_prepare_wakeup - prepare users for wakeup after congestion
> + * @bcl: broadcast link
> + * @resultq: queue for users which can be woken up
> + * Move a number of waiting users, as permitted by available space in
> + * the send queue, from link wait queue to specified queue for wakeup
> + */
> +static void bclink_prepare_wakeup(struct tipc_link *bcl, struct sk_buff_head *resultq)
> +{
> +	int pnd[TIPC_SYSTEM_IMPORTANCE + 1] = {0,};
> +	int imp, lim;
> +	struct sk_buff *skb, *tmp;
> +
> +	skb_queue_walk_safe(&bcl->wakeupq, skb, tmp) {
> +		imp = TIPC_SKB_CB(skb)->chain_imp;
> +		lim = bcl->window + bcl->backlog[imp].limit;
> +		pnd[imp] += TIPC_SKB_CB(skb)->chain_sz;
> +		if ((pnd[imp] + bcl->backlog[imp].len) >= lim)
> +			continue;
> +		skb_unlink(skb, &bcl->wakeupq);
> +		skb_queue_tail(resultq, skb);
> +	}
> +}
> +
> +/**
>   * tipc_bclink_wakeup_users - wake up pending users
>   *
>   * Called with no locks taken
> @@ -176,8 +200,12 @@ static void bclink_retransmit_pkt(struct tipc_net *tn, u32 after, u32 to)
>  void tipc_bclink_wakeup_users(struct net *net)
>  {
>  	struct tipc_net *tn = net_generic(net, tipc_net_id);
> +	struct tipc_link *bcl = tn->bcl;
> +	struct sk_buff_head resultq;
>  
> -	tipc_sk_rcv(net, &tn->bclink->link.wakeupq);
> +	skb_queue_head_init(&resultq);
> +	bclink_prepare_wakeup(bcl, &resultq);
> +	tipc_sk_rcv(net, &resultq);
>  }
>  
>  /**
> 
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1220008

FromKolmakov Dmitriy <kolmakov.dmitriy@huawei.com>
Date2015-09-07 11:10 +0200
Message-ID<q60B3-15o-1@gated-at.bofh.it>
In reply to#1219773
SWYgYW4gYXR0ZW1wdCB0byB3YWtlIHVwIHVzZXJzIG9mIGJyb2FkY2FzdCBsaW5rIGlzIG1hZGUg
d2hlbiB0aGVyZSBpcw0Kbm8gZW5vdWdoIHBsYWNlIGluIHNlbmQgcXVldWUgdGhhbiBpdCBtYXkg
aGFuZyB1cCBpbnNpZGUgdGhlDQp0aXBjX3NrX3JjdigpIGZ1bmN0aW9uIHNpbmNlIHRoZSBsb29w
IGJyZWFrcyBvbmx5IGFmdGVyIHRoZSB3YWtlIHVwDQpxdWV1ZSBiZWNvbWVzIGVtcHR5LiBUaGlz
IGNhbiBsZWFkIHRvIGNvbXBsZXRlIENQVSBzdGFsbCB3aXRoIHRoZQ0KZm9sbG93aW5nIG1lc3Nh
Z2UgZ2VuZXJhdGVkIGJ5IFJDVToNCg0KSU5GTzogcmN1X3NjaGVkIHNlbGYtZGV0ZWN0ZWQgc3Rh
bGwgb24gQ1BVIHsgMH0gICh0PTIxMDEgamlmZmllcw0KCQkJCQlnPTU0MjI1IGM9NTQyMjQgcT0x
MTQ2NSkNClRhc2sgZHVtcCBmb3IgQ1BVIDA6DQp0cGNoICAgICAgICAgICAgUiAgcnVubmluZyB0
YXNrICAgICAgICAwIDM5OTQ5ICAzOTk0OCAweDAwMDAwMDBhDQogZmZmZmZmZmY4MTg1MzZjMCBm
ZmZmODgxODFmYTAzN2EwIGZmZmZmZmZmODEwNmE0YmUgMDAwMDAwMDAwMDAwMDAwMA0KIGZmZmZm
ZmZmODE4NTM2YzAgZmZmZjg4MTgxZmEwMzdjMCBmZmZmZmZmZjgxMDZkOGE4IGZmZmY4ODE4MWZh
MDM4MDANCiAwMDAwMDAwMDAwMDAwMDAxIGZmZmY4ODE4MWZhMDM3ZjAgZmZmZmZmZmY4MTA5NGE1
MCBmZmZmODgxODFmYTE1NjgwDQpDYWxsIFRyYWNlOg0KIDxJUlE+ICBbPGZmZmZmZmZmODEwNmE0
YmU+XSBzY2hlZF9zaG93X3Rhc2srMHhhZS8weDEyMA0KIFs8ZmZmZmZmZmY4MTA2ZDhhOD5dIGR1
bXBfY3B1X3Rhc2srMHgzOC8weDQwDQogWzxmZmZmZmZmZjgxMDk0YTUwPl0gcmN1X2R1bXBfY3B1
X3N0YWNrcysweDkwLzB4ZDANCiBbPGZmZmZmZmZmODEwOTdjM2I+XSByY3VfY2hlY2tfY2FsbGJh
Y2tzKzB4M2ViLzB4NmUwDQogWzxmZmZmZmZmZjgxMDZlNTNmPl0gPyBhY2NvdW50X3N5c3RlbV90
aW1lKzB4N2YvMHgxNzANCiBbPGZmZmZmZmZmODEwOTllNjQ+XSB1cGRhdGVfcHJvY2Vzc190aW1l
cysweDM0LzB4NjANCiBbPGZmZmZmZmZmODEwYTg0ZDE+XSB0aWNrX3NjaGVkX2hhbmRsZS5pc3Jh
LjE4KzB4MzEvMHg0MA0KIFs8ZmZmZmZmZmY4MTBhODUxYz5dIHRpY2tfc2NoZWRfdGltZXIrMHgz
Yy8weDcwDQogWzxmZmZmZmZmZjgxMDlhNDNkPl0gX19ydW5faHJ0aW1lci5pc3JhLjM0KzB4M2Qv
MHhjMA0KIFs8ZmZmZmZmZmY4MTA5YWE5NT5dIGhydGltZXJfaW50ZXJydXB0KzB4YzUvMHgxZTAN
CiBbPGZmZmZmZmZmODEwMzBkNTI+XSA/IG5hdGl2ZV9zbXBfc2VuZF9yZXNjaGVkdWxlKzB4NDIv
MHg2MA0KIFs8ZmZmZmZmZmY4MTAzMmYwND5dIGxvY2FsX2FwaWNfdGltZXJfaW50ZXJydXB0KzB4
MzQvMHg2MA0KIFs8ZmZmZmZmZmY4MTAzMzViYz5dIHNtcF9hcGljX3RpbWVyX2ludGVycnVwdCsw
eDNjLzB4NjANCiBbPGZmZmZmZmZmODE2NWEzZmI+XSBhcGljX3RpbWVyX2ludGVycnVwdCsweDZi
LzB4NzANCiBbPGZmZmZmZmZmODE2NTkxMjk+XSA/IF9yYXdfc3Bpbl91bmxvY2tfaXJxcmVzdG9y
ZSsweDkvMHgxMA0KIFs8ZmZmZmZmZmY4MTA3ZWI5Zj5dIF9fd2FrZV91cF9zeW5jX2tleSsweDRm
LzB4NjANCiBbPGZmZmZmZmZmYTMxM2RkZDE+XSB0aXBjX3dyaXRlX3NwYWNlKzB4MzEvMHg0MCBb
dGlwY10NCiBbPGZmZmZmZmZmYTMxM2RhZGY+XSBmaWx0ZXJfcmN2KzB4MzFmLzB4NTIwIFt0aXBj
XQ0KIFs8ZmZmZmZmZmZhMzEzZDY5OT5dID8gdGlwY19za19sb29rdXArMHhjOS8weDExMCBbdGlw
Y10NCiBbPGZmZmZmZmZmODE2NTkyNTk+XSA/IF9yYXdfc3Bpbl9sb2NrX2JoKzB4MTkvMHgzMA0K
IFs8ZmZmZmZmZmZhMzE0MTIyYz5dIHRpcGNfc2tfcmN2KzB4MmRjLzB4M2UwIFt0aXBjXQ0KIFs8
ZmZmZmZmZmZhMzEyZTdmZj5dIHRpcGNfYmNsaW5rX3dha2V1cF91c2VycysweDJmLzB4NDAgW3Rp
cGNdDQogWzxmZmZmZmZmZmEzMTNjZTI2Pl0gdGlwY19ub2RlX3VubG9jaysweDE4Ni8weDE5MCBb
dGlwY10NCiBbPGZmZmZmZmZmODE1OTdjMWM+XSA/IGtmcmVlX3NrYisweDJjLzB4NDANCiBbPGZm
ZmZmZmZmYTMxMzQ3NWM+XSB0aXBjX3JjdisweDJhYy8weDhjMCBbdGlwY10NCiBbPGZmZmZmZmZm
YTMxMmZmNTg+XSB0aXBjX2wyX3Jjdl9tc2crMHgzOC8weDUwIFt0aXBjXQ0KIFs8ZmZmZmZmZmY4
MTVhNzZkMz5dIF9fbmV0aWZfcmVjZWl2ZV9za2JfY29yZSsweDVhMy8weDk1MA0KIFs8ZmZmZmZm
ZmY4MTVhOThkMz5dIF9fbmV0aWZfcmVjZWl2ZV9za2IrMHgxMy8weDYwDQogWzxmZmZmZmZmZjgx
NWE5OTNlPl0gbmV0aWZfcmVjZWl2ZV9za2JfaW50ZXJuYWwrMHgxZS8weDkwDQogWzxmZmZmZmZm
ZjgxNWFhMTM4Pl0gbmFwaV9ncm9fcmVjZWl2ZSsweDc4LzB4YTANCiBbPGZmZmZmZmZmYTA3Zjkz
ZjQ+XSB0ZzNfcG9sbF93b3JrKzB4YzU0LzB4ZjQwIFt0ZzNdDQogWzxmZmZmZmZmZjgxNTk3Yzhj
Pl0gPyBjb25zdW1lX3NrYisweDJjLzB4NDANCiBbPGZmZmZmZmZmYTA3Zjk3MjE+XSB0ZzNfcG9s
bF9tc2l4KzB4NDEvMHgxNjAgW3RnM10NCiBbPGZmZmZmZmZmODE1YWIwZjI+XSBuZXRfcnhfYWN0
aW9uKzB4ZTIvMHgyOTANCiBbPGZmZmZmZmZmODEwNGI5MmE+XSBfX2RvX3NvZnRpcnErMHhkYS8w
eDFmMA0KIFs8ZmZmZmZmZmY4MTA0YmMyNj5dIGlycV9leGl0KzB4NzYvMHhhMA0KIFs8ZmZmZmZm
ZmY4MTAwNDM1NT5dIGRvX0lSUSsweDU1LzB4ZjANCiBbPGZmZmZmZmZmODE2NWExMmI+XSBjb21t
b25faW50ZXJydXB0KzB4NmIvMHg2Yg0KIDxFT0k+DQoNClRoZSBpc3N1ZSBvY2N1cnMgb25seSB3
aGVuIHRpcGNfc2tfcmN2KCkgaXMgdXNlZCB0byB3YWtlIHVwIHBvc3Rwb25lZA0Kc2VuZGVyczoN
Cg0KCXRpcGNfYmNsaW5rX3dha2V1cF91c2VycygpDQoJCS8vIHdha2V1cHEgLSBpcyBhIHF1ZXVl
IHdoaWNoIGNvbnNpc3RzIG9mIHNwZWNpYWwNCgkJLy8gCQkgbWVzc2FnZXMgd2l0aCBTT0NLX1dB
S0VVUCB0eXBlLg0KCQl0aXBjX3NrX3Jjdih3YWtldXBxKQ0KCQkJLi4uDQoJCQl3aGlsZSAoc2ti
X3F1ZXVlX2xlbihpbnB1dHEpKSB7DQoJCQkJZmlsdGVyX3Jjdihza2IpDQoJCQkJCS8vIEhlcmUg
dGhlIHR5cGUgb2YgbWVzc2FnZSBpcyBjaGVja2VkDQoJCQkJCS8vIGFuZCBpZiBpdCBpcyBTT0NL
X1dBS0VVUCB0aGVuDQoJCQkJCS8vIGl0IHRyaWVzIHRvIHdha2UgdXAgYSBzZW5kZXIuDQoJCQkJ
CXRpcGNfd3JpdGVfc3BhY2Uoc2spDQoJCQkJCQl3YWtlX3VwX2ludGVycnVwdGlibGVfc3luY19w
b2xsKCkNCgkJCX0NCg0KQWZ0ZXIgdGhlIHNlbmRlciB0aHJlYWQgaXMgd29rZSB1cCBpdCBjYW4g
Z2F0aGVyIGNvbnRyb2wgYW5kIHBlcmZvcm0NCmFuIGF0dGVtcHQgdG8gc2VuZCBhIG1lc3NhZ2Uu
IEJ1dCBpZiB0aGVyZSBpcyBubyBlbm91Z2ggcGxhY2UgaW4gc2VuZA0KcXVldWUgaXQgd2lsbCBj
YWxsIGxpbmtfc2NoZWR1bGVfdXNlcigpIGZ1bmN0aW9uIHdoaWNoIHB1dHMgYSBtZXNzYWdlDQpv
ZiB0eXBlIFNPQ0tfV0FLRVVQIHRvIHRoZSB3YWtldXAgcXVldWUgYW5kIHB1dCB0aGUgc2VuZGVy
IHRvIHNsZWVwLg0KVGh1cyB0aGUgc2l6ZSBvZiB0aGUgcXVldWUgYWN0dWFsbHkgaXMgbm90IGNo
YW5nZWQgYW5kIHRoZSB3aGlsZSgpDQpsb29wIG5ldmVyIGV4aXRzLg0KDQpUaGUgYXBwcm9hY2gg
SSBwcm9wb3NlZCBpcyB0byB3YWtlIHVwIG9ubHkgc2VuZGVycyBmb3Igd2hpY2ggdGhlcmUgaXMN
CmVub3VnaCBwbGFjZSBpbiBzZW5kIHF1ZXVlIHNvIHRoZSBkZXNjcmliZWQgaXNzdWUgY2FuJ3Qg
b2NjdXIuDQpNb3Jlb3ZlciB0aGUgc2FtZSBhcHByb2FjaCBpcyBhbHJlYWR5IHVzZWQgdG8gd2Fr
ZSB1cCBzZW5kZXJzIG9uDQp1bmljYXN0IGxpbmtzLg0KDQpJIGhhdmUgZ290IGludG8gdGhlIGlz
c3VlIG9uIG91ciBwcm9kdWN0IGNvZGUgYnV0IHRvIHJlcHJvZHVjZSB0aGUNCmlzc3VlIEkgY2hh
bmdlZCBhIGJlbmNobWFyayB0ZXN0IGFwcGxpY2F0aW9uIChmcm9tDQp0aXBjdXRpbHMvZGVtb3Mv
YmVuY2htYXJrKSB0byBwZXJmb3JtIHRoZSBmb2xsb3dpbmcgc2NlbmFyaW86DQoJMS4gUnVuIDY0
IGluc3RhbmNlcyBvZiB0ZXN0IGFwcGxpY2F0aW9uIChub2RlcykuIEl0IGNhbiBiZSBkb25lDQoJ
ICAgb24gdGhlIG9uZSBwaHlzaWNhbCBtYWNoaW5lLg0KCTIuIEVhY2ggYXBwbGljYXRpb24gY29u
bmVjdHMgdG8gYWxsIG90aGVyIHVzaW5nIFRJUEMgc29ja2V0cyBpbg0KCSAgIFJETSBtb2RlLg0K
CTMuIFdoZW4gc2V0dXAgaXMgZG9uZSBhbGwgbm9kZXMgc3RhcnQgc2ltdWx0YW5lb3VzbHkgc2Vu
ZA0KCSAgIGJyb2FkY2FzdCBtZXNzYWdlcy4NCgk0LiBFdmVyeXRoaW5nIGhhbmdzIHVwLg0KDQpU
aGUgaXNzdWUgaXMgcmVwcm9kdWNpYmxlIG9ubHkgd2hlbiBhIGNvbmdlc3Rpb24gb24gYnJvYWRj
YXN0IGxpbmsNCm9jY3Vycy4gRm9yIGV4YW1wbGUsIHdoZW4gdGhlcmUgYXJlIG9ubHkgOCBub2Rl
cyBpdCB3b3JrcyBmaW5lIHNpbmNlDQpjb25nZXN0aW9uIGRvZXNuJ3Qgb2NjdXIuIFNlbmQgcXVl
dWUgbGltaXQgaXMgNDAgaW4gbXkgY2FzZSAoSSB1c2UgYQ0KY3JpdGljYWwgaW1wb3J0YW5jZSBs
ZXZlbCkgYW5kIHdoZW4gNjQgbm9kZXMgc2VuZCBhIG1lc3NhZ2UgYXQgdGhlDQpzYW1lIG1vbWVu
dCBhIGNvbmdlc3Rpb24gb2NjdXJzIGV2ZXJ5IHRpbWUuDQoNClNpZ25lZC1vZmYtYnk6IERtaXRy
eSBTIEtvbG1ha292IDxrb2xtYWtvdi5kbWl0cml5QGh1YXdlaS5jb20+DQpSZXZpZXdlZC1ieTog
Sm9uIE1hbG95IDxqb24ubWFsb3lAZXJpY3Nzb24uY29tPg0KQWNrZWQtYnk6IFlpbmcgWHVlIDx5
aW5nLnh1ZUB3aW5kcml2ZXIuY29tPg0KLS0tDQp2MjogVXBkYXRlZCBhZnRlciBjb21tZW50cyBm
cm9tIEpvbiBhbmQgWWluZy4NCg0KZGlmZiAtLWdpdCBhL25ldC90aXBjL2JjYXN0LmMgYi9uZXQv
dGlwYy9iY2FzdC5jDQppbmRleCBjNWNiZGNiLi45OTdkZDYwIDEwMDY0NA0KLS0tIGEvbmV0L3Rp
cGMvYmNhc3QuYw0KKysrIGIvbmV0L3RpcGMvYmNhc3QuYw0KQEAgLTE2OSw2ICsxNjksMzAgQEAg
c3RhdGljIHZvaWQgYmNsaW5rX3JldHJhbnNtaXRfcGt0KHN0cnVjdCB0aXBjX25ldCAqdG4sIHUz
MiBhZnRlciwgdTMyIHRvKQ0KIH0NCg0KIC8qKg0KKyAqIGJjbGlua19wcmVwYXJlX3dha2V1cCAt
IHByZXBhcmUgdXNlcnMgZm9yIHdha2V1cCBhZnRlciBjb25nZXN0aW9uDQorICogQGJjbDogYnJv
YWRjYXN0IGxpbmsNCisgKiBAcmVzdWx0cTogcXVldWUgZm9yIHVzZXJzIHdoaWNoIGNhbiBiZSB3
b2tlbiB1cA0KKyAqIE1vdmUgYSBudW1iZXIgb2Ygd2FpdGluZyB1c2VycywgYXMgcGVybWl0dGVk
IGJ5IGF2YWlsYWJsZSBzcGFjZSBpbg0KKyAqIHRoZSBzZW5kIHF1ZXVlLCBmcm9tIGxpbmsgd2Fp
dCBxdWV1ZSB0byBzcGVjaWZpZWQgcXVldWUgZm9yIHdha2V1cA0KKyAqLw0KK3N0YXRpYyB2b2lk
IGJjbGlua19wcmVwYXJlX3dha2V1cChzdHJ1Y3QgdGlwY19saW5rICpiY2wsIHN0cnVjdCBza19i
dWZmX2hlYWQgKnJlc3VsdHEpDQorew0KKwlpbnQgcG5kW1RJUENfU1lTVEVNX0lNUE9SVEFOQ0Ug
KyAxXSA9IHswLH07DQorCWludCBpbXAsIGxpbTsNCisJc3RydWN0IHNrX2J1ZmYgKnNrYiwgKnRt
cDsNCisNCisJc2tiX3F1ZXVlX3dhbGtfc2FmZSgmYmNsLT53YWtldXBxLCBza2IsIHRtcCkgew0K
KwkJaW1wID0gVElQQ19TS0JfQ0Ioc2tiKS0+Y2hhaW5faW1wOw0KKwkJbGltID0gYmNsLT53aW5k
b3cgKyBiY2wtPmJhY2tsb2dbaW1wXS5saW1pdDsNCisJCXBuZFtpbXBdICs9IFRJUENfU0tCX0NC
KHNrYiktPmNoYWluX3N6Ow0KKwkJaWYgKChwbmRbaW1wXSArIGJjbC0+YmFja2xvZ1tpbXBdLmxl
bikgPj0gbGltKQ0KKwkJCWNvbnRpbnVlOw0KKwkJc2tiX3VubGluayhza2IsICZiY2wtPndha2V1
cHEpOw0KKwkJc2tiX3F1ZXVlX3RhaWwocmVzdWx0cSwgc2tiKTsNCisJfQ0KK30NCisNCisvKioN
CiAgKiB0aXBjX2JjbGlua193YWtldXBfdXNlcnMgLSB3YWtlIHVwIHBlbmRpbmcgdXNlcnMNCiAg
Kg0KICAqIENhbGxlZCB3aXRoIG5vIGxvY2tzIHRha2VuDQpAQCAtMTc2LDggKzIwMCwxMiBAQCBz
dGF0aWMgdm9pZCBiY2xpbmtfcmV0cmFuc21pdF9wa3Qoc3RydWN0IHRpcGNfbmV0ICp0biwgdTMy
IGFmdGVyLCB1MzIgdG8pDQogdm9pZCB0aXBjX2JjbGlua193YWtldXBfdXNlcnMoc3RydWN0IG5l
dCAqbmV0KQ0KIHsNCiAJc3RydWN0IHRpcGNfbmV0ICp0biA9IG5ldF9nZW5lcmljKG5ldCwgdGlw
Y19uZXRfaWQpOw0KKwlzdHJ1Y3QgdGlwY19saW5rICpiY2wgPSB0bi0+YmNsOw0KKwlzdHJ1Y3Qg
c2tfYnVmZl9oZWFkIHJlc3VsdHE7DQoNCi0JdGlwY19za19yY3YobmV0LCAmdG4tPmJjbGluay0+
bGluay53YWtldXBxKTsNCisJc2tiX3F1ZXVlX2hlYWRfaW5pdCgmcmVzdWx0cSk7DQorCWJjbGlu
a19wcmVwYXJlX3dha2V1cChiY2wsICZyZXN1bHRxKTsNCisJdGlwY19za19yY3YobmV0LCAmcmVz
dWx0cSk7DQogfQ0KDQogLyoqDQo=
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1221249

FromDavid Miller <davem@davemloft.net>
Date2015-09-09 08:00 +0200
Message-ID<q6GAi-2qZ-9@gated-at.bofh.it>
In reply to#1220008
From: Kolmakov Dmitriy <kolmakov.dmitriy@huawei.com>
Date: Mon, 7 Sep 2015 09:05:48 +0000

> If an attempt to wake up users of broadcast link is made when there is
> no enough place in send queue than it may hang up inside the
> tipc_sk_rcv() function since the loop breaks only after the wake up
> queue becomes empty. This can lead to complete CPU stall with the
> following message generated by RCU:
 ...
> The issue occurs only when tipc_sk_rcv() is used to wake up postponed
> senders:
 ...
> After the sender thread is woke up it can gather control and perform
> an attempt to send a message. But if there is no enough place in send
> queue it will call link_schedule_user() function which puts a message
> of type SOCK_WAKEUP to the wakeup queue and put the sender to sleep.
> Thus the size of the queue actually is not changed and the while()
> loop never exits.
> 
> The approach I proposed is to wake up only senders for which there is
> enough place in send queue so the described issue can't occur.
> Moreover the same approach is already used to wake up senders on
> unicast links.
> 
> I have got into the issue on our product code but to reproduce the
> issue I changed a benchmark test application (from
> tipcutils/demos/benchmark) to perform the following scenario:
> 	1. Run 64 instances of test application (nodes). It can be done
> 	   on the one physical machine.
> 	2. Each application connects to all other using TIPC sockets in
> 	   RDM mode.
> 	3. When setup is done all nodes start simultaneously send
> 	   broadcast messages.
> 	4. Everything hangs up.
> 
> The issue is reproducible only when a congestion on broadcast link
> occurs. For example, when there are only 8 nodes it works fine since
> congestion doesn't occur. Send queue limit is 40 in my case (I use a
> critical importance level) and when 64 nodes send a message at the
> same moment a congestion occurs every time.
> 
> Signed-off-by: Dmitry S Kolmakov <kolmakov.dmitriy@huawei.com>
> Reviewed-by: Jon Maloy <jon.maloy@ericsson.com>
> Acked-by: Ying Xue <ying.xue@windriver.com>
> ---
> v2: Updated after comments from Jon and Ying.

Applied, thanks.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web