Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.networking > #2070 > unrolled thread

linux networking buffer

Back to article view | Back to comp.os.linux.networking

Contents

  linux networking buffer Surinder <Surinder.Singh@example.net> - 2013-03-30 21:29 +0530
    Re: linux networking buffer Rick Jones <rick.jones2@hp.com> - 2013-04-05 21:39 +0000
      Re: linux networking buffer Tauno Voipio <tauno.voipio@notused.fi.invalid> - 2013-04-06 19:37 +0300
        Re: linux networking buffer J G Miller <miller@yoyo.ORG> - 2013-04-06 19:36 +0000
          Re: linux networking buffer Rick Jones <rick.jones2@hp.com> - 2013-04-08 20:23 +0000
          Re: linux networking buffer Surinder <Surinder.Singh@example.net> - 2013-04-09 10:33 +0530
            Re: linux networking buffer J G Miller <miller@yoyo.ORG> - 2013-04-09 13:10 +0000
        Re: linux networking buffer Surinder <Surinder.Singh@example.net> - 2013-04-09 10:54 +0530
      Re: linux networking buffer Surinder <surinder@esurinder.net> - 2013-04-09 10:20 +0530
        Re: linux networking buffer Rick Jones <rick.jones2@hp.com> - 2013-04-09 17:19 +0000
          Re: linux networking buffer Surinder <Surinder.Singh@example.net> - 2013-04-11 22:15 +0530
          Re: linux networking buffer Surinder <Surinder.Singh@example.net> - 2013-04-13 15:07 +0530

#2070 — linux networking buffer

Hi,

A) RX count as seen and set by ethtool.
B) sysctl -w net.core.netdev_max_backlog=3000
C) sysctl -w net.core.rmem_max=256000 along with setsockopt(SO_RCVBUF)

on transmit side
D) TX count as seen and set by ethtool
E) txqueuelen as seen and set by ifconfig
F) sysctl -w net.core.wmem_max and setsockopt(SO_SNDBUF)

I guess
A,D are set in NIC eprom and meant for NIC memory
	or NIC driver (host non-pagable memory) ?
B,E are kept in kernel outside any device driver
C,F are kept in kernel networking stack (not part of NIC driver)

A,D changes are for that very NIC
B,E are about system wide queue size
C,F allows per socket buffer size tuning.

I wrote a ping program in C/Linux to send icmp echo to 1000 hosts, I got 
    around 170 drops in single iteration i.e. without retry.
These are my buffer values
RX(200), backlog(1000), net.core.rmem_max = 131071, SO_RCVBUF :262142

packet sniffer got all 1000 reply packets.
packet size is about 100byte on wire and 84byte with IP header. so 
262142 should easily hold 2600 packets without drop.
changing RX didn't make difference.
on increasing rmem_max, all 1000 are received though.
There was no other traffic to the collector (as verified by packet 
capture tool).

TIA
- Surinder

[toc] | [next] | [standalone]

#2095

Surinder <Surinder.Singh@example.net> wrote:
> These are my buffer values
> RX(200), backlog(1000), net.core.rmem_max = 131071, SO_RCVBUF :262142

> packet sniffer got all 1000 reply packets.
> packet size is about 100byte on wire and 84byte with IP header. so
> 262142 should easily hold 2600 packets without drop.

Except that Linux socket buffer limits are applied to not just the
packet headers/data but also the size of the buffer(s) used to hold
the data.  So, if the NIC driver happens to post 2048 byte buffers to
the card, that 262141 bytes would be able to hold no more than 127
packets.  So, my guess would be that your receiver wasn't keeping-up
with all the replies coming-in and lost the race a few times.

> changing RX didn't make difference.

Suggesting that the "queue" which overflows is not the NIC's RX
queue(s).

> on increasing rmem_max, all 1000 are received though.
> There was no other traffic to the collector (as verified by packet 
> capture tool).

That would be consistent then with the increased rmem_max size being
sufficient to cover those cases where your receiver wasn't able to
keep-up for a short length of time.

If your application is simply sending all 1000 requests and *then*
looking to receive/process replies, you should alter the logic of your
application to be able to also look for replies as you are sending
requests.

rick jones
-- 
firebug n, the idiot who tosses a lit cigarette out his car window
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

[toc] | [prev] | [next] | [standalone]

#2096

On 6.4.13 12:39 , Rick Jones wrote:
> Surinder <Surinder.Singh@example.net> wrote:
>> These are my buffer values
>> RX(200), backlog(1000), net.core.rmem_max = 131071, SO_RCVBUF :262142
>
>> packet sniffer got all 1000 reply packets.
>> packet size is about 100byte on wire and 84byte with IP header. so
>> 262142 should easily hold 2600 packets without drop.
>
> Except that Linux socket buffer limits are applied to not just the
> packet headers/data but also the size of the buffer(s) used to hold
> the data.  So, if the NIC driver happens to post 2048 byte buffers to
> the card, that 262141 bytes would be able to hold no more than 127
> packets.  So, my guess would be that your receiver wasn't keeping-up
> with all the replies coming-in and lost the race a few times.
>
>> changing RX didn't make difference.
>
> Suggesting that the "queue" which overflows is not the NIC's RX
> queue(s).
>
>> on increasing rmem_max, all 1000 are received though.
>> There was no other traffic to the collector (as verified by packet
>> capture tool).
>
> That would be consistent then with the increased rmem_max size being
> sufficient to cover those cases where your receiver wasn't able to
> keep-up for a short length of time.
>
> If your application is simply sending all 1000 requests and *then*
> looking to receive/process replies, you should alter the logic of your
> application to be able to also look for replies as you are sending
> requests.
>
> rick jones


I'm not sure if it is wise to continue handholding the OP. The task
at hand smells a bit too much of first stage of a spam-o-bot.

Luckily it seems that he's not too much up to the task.

-- 

Tauno Voipio

[toc] | [prev] | [next] | [standalone]

#2097

On Saturday, April 6th, 2013, at 19:37:26h +0300, Tauno Voipio observed:

> Luckily it seems that he's not too much up to the task.

It does seems a little strange that Thunderbird is up to version 17.0.4
but the original posting in this thread was made with a six year old
version from 2006

        User-Agent: Thunderbird 1.5.0.9 (X11/20061206)

So one does wonder which version of the kernel is in use? 
2.6.20 or earlier?

[toc] | [prev] | [next] | [standalone]

#2101

J G Miller <miller@yoyo.org> wrote:
> On Saturday, April 6th, 2013, at 19:37:26h +0300, Tauno Voipio observed:

> > Luckily it seems that he's not too much up to the task.

> It does seems a little strange that Thunderbird is up to version 17.0.4
> but the original posting in this thread was made with a six year old
> version from 2006

>         User-Agent: Thunderbird 1.5.0.9 (X11/20061206)

> So one does wonder which version of the kernel is in use? 
> 2.6.20 or earlier?

Oh, back when my primary workstation was an HP-UX 11.11-running HP9000
J5600 I used the same version of browser/email client for years at a
time.  I wasn't being forced-upgraded to a new version every couple
months.

rick jones
-- 
denial, anger, bargaining, depression, acceptance, rebirth...
                                     where do you want to be today?
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com  but NOT BOTH...

[toc] | [prev] | [next] | [standalone]

#2103

J G Miller wrote:
> On Saturday, April 6th, 2013, at 19:37:26h +0300, Tauno Voipio observed:
> 
>> Luckily it seems that he's not too much up to the task.
> 
> It does seems a little strange that Thunderbird is up to version 17.0.4
> but the original posting in this thread was made with a six year old
> version from 2006
> 
>         User-Agent: Thunderbird 1.5.0.9 (X11/20061206)
> 
> So one does wonder which version of the kernel is in use? 
> 2.6.20 or earlier?

Miller,

This is my Linux info though I intended to ask for Linux in general.
2.6.13-15.18-default #1 Tue Oct 2 17:36:20 UTC 2007 i686 i386 GNU/Linux

surely a lot would have changed in linux kernel since.

How would Thunderbird make or version relate to my question :)

- Surinder

[toc] | [prev] | [next] | [standalone]

#2105

On Tuesday, April 9th, 2013, at 10:33:16h +0530, Surinder wrote:

> This is my Linux info though I intended to ask for Linux in general.
>
> 2.6.13-15.18-default #1 Tue Oct 2 17:36:20 UTC 2007 i686 i386 GNU/Linux
> 
> surely a lot would have changed in linux kernel since.

Your question was about networking and there were very important
changes in Linux kernel networking features in going from 2.4 to 2.6,
so it is important to know which version of the kernel you are using.

In going from 2.6.13 which is about 6 years old to today's kernel
version of 3.{4,5} depending on distribution or latest 3.7 there
have been some important changes as well.

[toc] | [prev] | [next] | [standalone]

#2104

Tauno Voipio wrote:
> On 6.4.13 12:39 , Rick Jones wrote:
>> Surinder <Surinder.Singh@example.net> wrote:
>>> These are my buffer values
>>> RX(200), backlog(1000), net.core.rmem_max = 131071, SO_RCVBUF :262142
>>
>>> packet sniffer got all 1000 reply packets.
>>> packet size is about 100byte on wire and 84byte with IP header. so
>>> 262142 should easily hold 2600 packets without drop.
>>
>> Except that Linux socket buffer limits are applied to not just the
>> packet headers/data but also the size of the buffer(s) used to hold
>> the data.  So, if the NIC driver happens to post 2048 byte buffers to
>> the card, that 262141 bytes would be able to hold no more than 127
>> packets.  So, my guess would be that your receiver wasn't keeping-up
>> with all the replies coming-in and lost the race a few times.
>>
>>> changing RX didn't make difference.
>>
>> Suggesting that the "queue" which overflows is not the NIC's RX
>> queue(s).
>>
>>> on increasing rmem_max, all 1000 are received though.
>>> There was no other traffic to the collector (as verified by packet
>>> capture tool).
>>
>> That would be consistent then with the increased rmem_max size being
>> sufficient to cover those cases where your receiver wasn't able to
>> keep-up for a short length of time.
>>
>> If your application is simply sending all 1000 requests and *then*
>> looking to receive/process replies, you should alter the logic of your
>> application to be able to also look for replies as you are sending
>> requests.
>>
>> rick jones
> 
> 
> I'm not sure if it is wise to continue handholding the OP. The task
> at hand smells a bit too much of first stage of a spam-o-bot.
> 
> Luckily it seems that he's not too much up to the task.
> 

Tauno,

A tool has no intent else than doing its job.
Its the person who has intents.
And it is different thing to make guess on intents of others.
And to guess their capacity to follow their intents.
And that on forum that is meant for understand making/working of tool.
And to act as wet blanket.

I hate spam more than I hate mosquitoes.
spam has (despite of all anti spam tools) make lot of non reversible 
losses to Internet. Worst affected being NNTP newsgroups.
sometimes a valid email ends up in spam box.

The concerned application is ping check for a set of devices just for 
reachability.

- Surinder

[toc] | [prev] | [next] | [standalone]

#2102

Rick Jones wrote:
> Surinder <Surinder.Singh@example.net> wrote:
>> These are my buffer values
>> RX(200), backlog(1000), net.core.rmem_max = 131071, SO_RCVBUF :262142
> 
>> packet sniffer got all 1000 reply packets.
>> packet size is about 100byte on wire and 84byte with IP header. so
>> 262142 should easily hold 2600 packets without drop.
> 
> Except that Linux socket buffer limits are applied to not just the
> packet headers/data but also the size of the buffer(s) used to hold
> the data.  So, if the NIC driver happens to post 2048 byte buffers to
> the card, that 262141 bytes would be able to hold no more than 127
> packets.  So, my guess would be that your receiver wasn't keeping-up
> with all the replies coming-in and lost the race a few times.

Rick,
Thanks for reply.
SO_RCVBUF of 262141 may be coming into picture after kernel got the 
packet and putting it into socket specific queues. I understand that
you are suggesting that total packet capacity is calculated based on
some constant size for the packet. But that would lead to wastage of
memory when the packet size are small. as tcp or udp is 65K.
buffer of 262141 could hold 4 packets of 65K each.

> 
>> changing RX didn't make difference.
> 
> Suggesting that the "queue" which overflows is not the NIC's RX
> queue(s).
Agree.

> 
>> on increasing rmem_max, all 1000 are received though.
>> There was no other traffic to the collector (as verified by packet 
>> capture tool).
> 
> That would be consistent then with the increased rmem_max size being
> sufficient to cover those cases where your receiver wasn't able to
> keep-up for a short length of time.

I got my program working properly with bigger buffer size, just that I 
want to optimise on buffer size. huge size of rmem_max could mean more
non-pagable memory reserved for kernel itself. not that I am running low
on RAM but just want to be conservative.

> 
> If your application is simply sending all 1000 requests and *then*
> looking to receive/process replies, you should alter the logic of your
> application to be able to also look for replies as you are sending
> requests.
> 
> rick jones

I could get the required reliability and performance with having 
all-sends/all-recvs in simpler and readable code.
If it hit the wall, then I would be having sender-thread/recvr-thread to 
go in parallel, which would ease load on the queues even though that 
would add complexity of synchronizing.

And also I wanted to understand que sizing mechanism between linux 
socket sender/rcvr.

This is my Linux info though I intended to ask for Linux in general.
Linux khyber 2.6.13-15.18-default #1 Tue Oct 2 17:36:20 UTC 2007 i686 
i686 i386 GNU/Linux


Thanks Again.
- Surinder

--- news://freenews.netfront.net/ - complaints: news@netfront.net ---

[toc] | [prev] | [next] | [standalone]

#2106

Socket buffers in general, and things like net.core.[rw]mem_max
specifically are not preallocations.  They are limits.  So, an
SO_RCVBUF of 256KB is not consuming 256 KB of memory unless there are
actually data/packets waiting therein.

For the inbound data path, the allocations are actually made by the
NIC driver when it allocates buffers to post to the NIC for inbound
DMA.  The strategies employed for buffer sizes there will vary from
NIC to NIC, and will depend on the NIC's programming model.

rick jones
-- 
Process shall set you free from the need for rational thought. 
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

[toc] | [prev] | [next] | [standalone]

#2107

Rick Jones wrote:
> Socket buffers in general, and things like net.core.[rw]mem_max
> specifically are not preallocations.  They are limits.  So, an
> SO_RCVBUF of 256KB is not consuming 256 KB of memory unless there are
> actually data/packets waiting therein.
> 
> For the inbound data path, the allocations are actually made by the
> NIC driver when it allocates buffers to post to the NIC for inbound
> DMA.  The strategies employed for buffer sizes there will vary from
> NIC to NIC, and will depend on the NIC's programming model.
> 
> rick jones

Good to hear that.

- Surinder

[toc] | [prev] | [next] | [standalone]

#2108

Rick Jones wrote:
> Socket buffers in general, and things like net.core.[rw]mem_max
> specifically are not preallocations.  They are limits.  So, an
> SO_RCVBUF of 256KB is not consuming 256 KB of memory unless there are
> actually data/packets waiting therein.
> 
> For the inbound data path, the allocations are actually made by the
> NIC driver when it allocates buffers to post to the NIC for inbound
> DMA.  The strategies employed for buffer sizes there will vary from
> NIC to NIC, and will depend on the NIC's programming model.
> 
> rick jones


Following code answers few of my questions.
(inline with what Rick told)
http://lxr.linux.no/#linux+v3.8.7/net/core/sock.c#L689
  708set_rcvbuf:
  709             sk->sk_userlocks |= SOCK_RCVBUF_LOCK;
  710             /*
  711              * We double it on the way in to account for
  712              * "struct sk_buff" etc. overhead.   Applications
  713              * assume that the SO_RCVBUF setting they make will
  714              * allow that much actual data to be received on that
  715              * socket.
  716              *
  717              * Applications are unaware that "struct sk_buff" and
  718              * other overheads allocate from the receive buffer
  719              * during socket buffer allocation.
  720              *
  721              * And after considering the possible alternatives,
  722              * returning the value we actually used in getsockopt
  723              * is the most desirable behavior.
  724              */
  725              sk->sk_rcvbuf = max_t(u32, val * 2, SOCK_MIN_RCVBUF);
  726              break;

As my packets are below 100 bytes data, the sk_buff overhead would be 
proportionally quite high. Visually looking at sk_buff, for a 32-bit 
system, the size of sk_buff appears to be around 200 bytes.
For 100 byte on wire packet, that means only one-third of rcvbuf is 
holding packet data. 2/3 is sk_buff overhead.

So for SO_RCVBUF :262142,
262142/3 = 87380 bytes are available for packet data.
87380/100 = 873 is number of packets that can be held in it.

1000 - 873 = 127 drops out of 1000 are expected.
What I got was 170, which is not far from expectation.

Following Chapters/Books were really helpfull.
[1] Understanding Linux Network Internals By Christian Benvenuti
Part III: Transmission and Reception

[2] Essential Linux Device Drivers by Sreekrishnan Venkateswaran
Chapter 15. Network Interface Cards

Thanks all folks here.

- Surinder

[toc] | [prev] | [standalone]

Back to top | Article view | comp.os.linux.networking

csiph-web