Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #28831 > unrolled thread

High packet rate ethernet with LwIP

Back to article view | Back to comp.arch.embedded

Contents

  High packet rate ethernet with LwIP Stef <stef33d@yahooI-N-V-A-L-I-D.com.invalid> - 2019-09-20 18:27 +0200
    Re: High packet rate ethernet with LwIP upsidedown@downunder.com - 2019-09-20 21:20 +0300
      Re: High packet rate ethernet with LwIP Stef <stef33d@yahooI-N-V-A-L-I-D.com.invalid> - 2019-09-23 10:37 +0200
        Re: High packet rate ethernet with LwIP Tauno Voipio <tauno.voipio@notused.fi.invalid> - 2019-09-23 14:08 +0300
        Re: High packet rate ethernet with LwIP Richard Damon <Richard@Damon-Family.org> - 2019-09-23 07:13 -0400
          Re: High packet rate ethernet with LwIP Tauno Voipio <tauno.voipio@notused.fi.invalid> - 2019-09-23 18:55 +0300
        Re: High packet rate ethernet with LwIP David Brown <david.brown@hesbynett.no> - 2019-09-23 14:04 +0200
          Re: High packet rate ethernet with LwIP Dimiter_Popoff <dp@tgi-sci.com> - 2019-09-23 17:03 +0300
            Re: High packet rate ethernet with LwIP David Brown <david.brown@hesbynett.no> - 2019-09-23 18:37 +0200
              Re: High packet rate ethernet with LwIP Stef <stef33d@yahooI-N-V-A-L-I-D.com.invalid> - 2019-09-24 14:22 +0200
                Re: High packet rate ethernet with LwIP upsidedown@downunder.com - 2019-09-25 07:55 +0300
                  Re: High packet rate ethernet with LwIP David Brown <david.brown@hesbynett.no> - 2019-09-25 09:51 +0200
                    Re: High packet rate ethernet with LwIP Stef <stef33d@yahooI-N-V-A-L-I-D.com.invalid> - 2019-09-25 16:03 +0200
                  Re: High packet rate ethernet with LwIP Stef <stef33d@yahooI-N-V-A-L-I-D.com.invalid> - 2019-09-25 11:59 +0200
                    Re: High packet rate ethernet with LwIP Stef <stef33d@yahooI-N-V-A-L-I-D.com.invalid> - 2019-09-25 13:16 +0200
                Re: High packet rate ethernet with LwIP David Brown <david.brown@hesbynett.no> - 2019-09-25 09:42 +0200
                  Re: High packet rate ethernet with LwIP Stef <stef33d@yahooI-N-V-A-L-I-D.com.invalid> - 2019-09-25 13:41 +0200
                    Re: High packet rate ethernet with LwIP David Brown <david.brown@hesbynett.no> - 2019-09-25 14:10 +0200
                      Re: High packet rate ethernet with LwIP Stef <stef33d@yahooI-N-V-A-L-I-D.com.invalid> - 2019-09-25 15:22 +0200
        Re: High packet rate ethernet with LwIP upsidedown@downunder.com - 2019-09-23 22:26 +0300
          Re: High packet rate ethernet with LwIP Stef <stef33d@yahooI-N-V-A-L-I-D.com.invalid> - 2019-09-23 22:16 +0200
            Re: High packet rate ethernet with LwIP David Brown <david.brown@hesbynett.no> - 2019-09-23 23:08 +0200
    Re: High packet rate ethernet with LwIP Joe Chisolm <jchisolm6@earthlink.net> - 2019-09-20 13:44 -0500
      Re: High packet rate ethernet with LwIP Stef <stef33d@yahooI-N-V-A-L-I-D.com.invalid> - 2019-09-23 10:46 +0200
    Re: High packet rate ethernet with LwIP David Brown <david.brown@hesbynett.no> - 2019-09-21 19:00 +0200
      Re: High packet rate ethernet with LwIP upsidedown@downunder.com - 2019-09-22 00:01 +0300
        Re: High packet rate ethernet with LwIP David Brown <david.brown@hesbynett.no> - 2019-09-22 09:06 +0200
          Re: High packet rate ethernet with LwIP upsidedown@downunder.com - 2019-09-22 11:29 +0300
            Re: High packet rate ethernet with LwIP David Brown <david.brown@hesbynett.no> - 2019-09-22 23:24 +0200
          Re: High packet rate ethernet with LwIP Stef <stef33d@yahooI-N-V-A-L-I-D.com.invalid> - 2019-09-23 11:10 +0200
      Re: High packet rate ethernet with LwIP Stef <stef33d@yahooI-N-V-A-L-I-D.com.invalid> - 2019-09-23 11:03 +0200

Page 1 of 2  [1] 2  Next page →

#28831 — High packet rate ethernet with LwIP

For transportation of several high speed serial ports, we want to use a
serial over ethernet approach. At a minimum I need to transfer four ports
with 512 byte packets that come in every 40 ms at 2 Mbps.

Search for a ready to use (for test at least) device has turned up nothing.
Standard devices usually go up to 230 kbps, some to 1 Mbps. Most faster
ones are single port only. And it should be a small device, embeddable,
logic level is all that is needed. If anyone knows of a usable device...

Since the board that this is on incorparates an ethernet capable controller
(LPC4088) that has not that much work to do richt now, I thought it should
be possible to add ethernet to it.

For testing I got an LPC4088 demo board with ethernet interface and got
LwIP running on it (stand alone, no OS).

Got it working OK to simulate 4 ports sending slow data to 4 TCP ports
that can be opened with a PC terminal program.

But when I turn up the data volume and packet rate, it stops working.
In some cases stuck in the hard fault handler, in other cases in a
timed waiting loop in the LwIP stack.

At this moment it is running fine with 2 ports open and a 512 byte packet
every 100 ms on each. When I increase the speed to 50 ms, it stops after
a while.

I have no previous experience with LwIP so where to start looking?
- First of all does it sound feasable over a 100 Mbit ethernet
  connection using TCP?
- What can be tuned in LwIP to improve this?
  I have already increased PBUF_POOL_SIZE from default 7 to 64 and
  MEM_SIZE from 12k to 32k


-- 
Stef    (remove caps, dashes and .invalid from e-mail address to reply by mail)

A man who keeps stealing mopeds is an obvious cycle-path.

[toc] | [next] | [standalone]

#28832

On Fri, 20 Sep 2019 18:27:52 +0200, Stef
<stef33d@yahooI-N-V-A-L-I-D.com.invalid> wrote:

>For transportation of several high speed serial ports, we want to use a
>serial over ethernet approach. At a minimum I need to transfer four ports
>with 512 byte packets that come in every 40 ms at 2 Mbps.
>
>Search for a ready to use (for test at least) device has turned up nothing.
>Standard devices usually go up to 230 kbps, some to 1 Mbps. Most faster
>ones are single port only. And it should be a small device, embeddable,
>logic level is all that is needed. If anyone knows of a usable device...
>
>Since the board that this is on incorparates an ethernet capable controller
>(LPC4088) that has not that much work to do richt now, I thought it should
>be possible to add ethernet to it.
>
>For testing I got an LPC4088 demo board with ethernet interface and got
>LwIP running on it (stand alone, no OS).
>
>Got it working OK to simulate 4 ports sending slow data to 4 TCP ports
>that can be opened with a PC terminal program.
>
>But when I turn up the data volume and packet rate, it stops working.
>In some cases stuck in the hard fault handler, in other cases in a
>timed waiting loop in the LwIP stack.
>
>At this moment it is running fine with 2 ports open and a 512 byte packet
>every 100 ms on each. When I increase the speed to 50 ms, it stops after
>a while.
>
>I have no previous experience with LwIP so where to start looking?
>- First of all does it sound feasable over a 100 Mbit ethernet
>  connection using TCP?
>- What can be tuned in LwIP to improve this?
>  I have already increased PBUF_POOL_SIZE from default 7 to 64 and
>  MEM_SIZE from 12k to 32k

Have you looked at the Ethernet traffic with e.g. Wireshark ? Look for
TCP retransmissions. If each serial line pumps data at 2 Mbit/s
without any flow control, a TCP retransmission will upset the Ethernet
traffic for a while, possibly overflowing buffers. Anyway, if retrials
are expected, the gross (ethernet) speed must be several times larger
than the net data rate.

Is it possible to use UDP and accept that some packets may be lost ?
Include a serial number in each frame and the receiver can detect
missing packets (that have failed CRC test) and mark the missing data
accordingly. This way the net speed can be much closer to the raw data
rate.

[toc] | [prev] | [next] | [standalone]

#28846

On 2019-09-20 upsidedown@downunder.com wrote in comp.arch.embedded:
> On Fri, 20 Sep 2019 18:27:52 +0200, Stef
><stef33d@yahooI-N-V-A-L-I-D.com.invalid> wrote:
>
>>For transportation of several high speed serial ports, we want to use a
>>serial over ethernet approach. At a minimum I need to transfer four ports
>>with 512 byte packets that come in every 40 ms at 2 Mbps.
>>
>>Search for a ready to use (for test at least) device has turned up nothing.
>>Standard devices usually go up to 230 kbps, some to 1 Mbps. Most faster
>>ones are single port only. And it should be a small device, embeddable,
>>logic level is all that is needed. If anyone knows of a usable device...
>>
>>Since the board that this is on incorparates an ethernet capable controller
>>(LPC4088) that has not that much work to do richt now, I thought it should
>>be possible to add ethernet to it.
>>
>>For testing I got an LPC4088 demo board with ethernet interface and got
>>LwIP running on it (stand alone, no OS).
>>
>>Got it working OK to simulate 4 ports sending slow data to 4 TCP ports
>>that can be opened with a PC terminal program.
>>
>>But when I turn up the data volume and packet rate, it stops working.
>>In some cases stuck in the hard fault handler, in other cases in a
>>timed waiting loop in the LwIP stack.
>>
>>At this moment it is running fine with 2 ports open and a 512 byte packet
>>every 100 ms on each. When I increase the speed to 50 ms, it stops after
>>a while.
>>
>>I have no previous experience with LwIP so where to start looking?
>>- First of all does it sound feasable over a 100 Mbit ethernet
>>  connection using TCP?
>>- What can be tuned in LwIP to improve this?
>>  I have already increased PBUF_POOL_SIZE from default 7 to 64 and
>>  MEM_SIZE from 12k to 32k
>
> Have you looked at the Ethernet traffic with e.g. Wireshark ? Look for
> TCP retransmissions. If each serial line pumps data at 2 Mbit/s
> without any flow control, a TCP retransmission will upset the Ethernet
> traffic for a while, possibly overflowing buffers. Anyway, if retrials
> are expected, the gross (ethernet) speed must be several times larger
> than the net data rate.

No, not done that yet, but indeed something I need to do.

> Is it possible to use UDP and accept that some packets may be lost ?
> Include a serial number in each frame and the receiver can detect
> missing packets (that have failed CRC test) and mark the missing data
> accordingly. This way the net speed can be much closer to the raw data
> rate.

That is something that came to my mind just after posting the question.
Would UDP be a better match for this application? The whole TCP
retransmission and timers may be the mechanisms causing the problem.
I think the standard TCP timeouts ar far too slow for this kind of
thing? Think I saw a 250 ms timer tick in the LwIP docs?

Missing packets may not be a problem. It is measurement data and if
any re-transmit takes over 40 ms there is no point anyway, there is
already new data. On a much slower rate, some control connection does
have to be reliable. So that could use TCP (if is does not disturb
the parallel UDP flow in LwIP) or use UDP as well and implement the
retries in the application.


-- 
Stef    (remove caps, dashes and .invalid from e-mail address to reply by mail)

Your temporary financial embarrassment will be relieved in a surprising manner.

[toc] | [prev] | [next] | [standalone]

#28850

On 23.9.19 11:37, Stef wrote:
> On 2019-09-20 upsidedown@downunder.com wrote in comp.arch.embedded:
>> On Fri, 20 Sep 2019 18:27:52 +0200, Stef
>> <stef33d@yahooI-N-V-A-L-I-D.com.invalid> wrote:
>>
>>> For transportation of several high speed serial ports, we want to use a
>>> serial over ethernet approach. At a minimum I need to transfer four ports
>>> with 512 byte packets that come in every 40 ms at 2 Mbps.
>>>
>>> Search for a ready to use (for test at least) device has turned up nothing.
>>> Standard devices usually go up to 230 kbps, some to 1 Mbps. Most faster
>>> ones are single port only. And it should be a small device, embeddable,
>>> logic level is all that is needed. If anyone knows of a usable device...
>>>
>>> Since the board that this is on incorparates an ethernet capable controller
>>> (LPC4088) that has not that much work to do richt now, I thought it should
>>> be possible to add ethernet to it.
>>>
>>> For testing I got an LPC4088 demo board with ethernet interface and got
>>> LwIP running on it (stand alone, no OS).
>>>
>>> Got it working OK to simulate 4 ports sending slow data to 4 TCP ports
>>> that can be opened with a PC terminal program.
>>>
>>> But when I turn up the data volume and packet rate, it stops working.
>>> In some cases stuck in the hard fault handler, in other cases in a
>>> timed waiting loop in the LwIP stack.
>>>
>>> At this moment it is running fine with 2 ports open and a 512 byte packet
>>> every 100 ms on each. When I increase the speed to 50 ms, it stops after
>>> a while.
>>>
>>> I have no previous experience with LwIP so where to start looking?
>>> - First of all does it sound feasable over a 100 Mbit ethernet
>>>   connection using TCP?
>>> - What can be tuned in LwIP to improve this?
>>>   I have already increased PBUF_POOL_SIZE from default 7 to 64 and
>>>   MEM_SIZE from 12k to 32k
>>
>> Have you looked at the Ethernet traffic with e.g. Wireshark ? Look for
>> TCP retransmissions. If each serial line pumps data at 2 Mbit/s
>> without any flow control, a TCP retransmission will upset the Ethernet
>> traffic for a while, possibly overflowing buffers. Anyway, if retrials
>> are expected, the gross (ethernet) speed must be several times larger
>> than the net data rate.
> 
> No, not done that yet, but indeed something I need to do.
> 
>> Is it possible to use UDP and accept that some packets may be lost ?
>> Include a serial number in each frame and the receiver can detect
>> missing packets (that have failed CRC test) and mark the missing data
>> accordingly. This way the net speed can be much closer to the raw data
>> rate.
> 
> That is something that came to my mind just after posting the question.
> Would UDP be a better match for this application? The whole TCP
> retransmission and timers may be the mechanisms causing the problem.
> I think the standard TCP timeouts ar far too slow for this kind of
> thing? Think I saw a 250 ms timer tick in the LwIP docs?
> 
> Missing packets may not be a problem. It is measurement data and if
> any re-transmit takes over 40 ms there is no point anyway, there is
> already new data. On a much slower rate, some control connection does
> have to be reliable. So that could use TCP (if is does not disturb
> the parallel UDP flow in LwIP) or use UDP as well and implement the
> retries in the application.


UDP may be the better option here. TCP does not guarantee to transfer
the packet boundaries, but UDP is just a lightweight cover for raw
IP packets.

If your raw data comes in with asynchronous serial lines, the non-
reliability of datagram traffic is already there, and it does not
matter much what UDP adds there.

Your processor may be overwhelmed with serial line character interrupt
load.

-- 

-TV

[toc] | [prev] | [next] | [standalone]

#28851

On 9/23/19 4:37 AM, Stef wrote:
> On 2019-09-20 upsidedown@downunder.com wrote in comp.arch.embedded:
>> Have you looked at the Ethernet traffic with e.g. Wireshark ? Look for
>> TCP retransmissions. If each serial line pumps data at 2 Mbit/s
>> without any flow control, a TCP retransmission will upset the Ethernet
>> traffic for a while, possibly overflowing buffers. Anyway, if retrials
>> are expected, the gross (ethernet) speed must be several times larger
>> than the net data rate.
> 
> No, not done that yet, but indeed something I need to do.
> 
>> Is it possible to use UDP and accept that some packets may be lost ?
>> Include a serial number in each frame and the receiver can detect
>> missing packets (that have failed CRC test) and mark the missing data
>> accordingly. This way the net speed can be much closer to the raw data
>> rate.
> 
> That is something that came to my mind just after posting the question.
> Would UDP be a better match for this application? The whole TCP
> retransmission and timers may be the mechanisms causing the problem.
> I think the standard TCP timeouts ar far too slow for this kind of
> thing? Think I saw a 250 ms timer tick in the LwIP docs?
> 
> Missing packets may not be a problem. It is measurement data and if
> any re-transmit takes over 40 ms there is no point anyway, there is
> already new data. On a much slower rate, some control connection does
> have to be reliable. So that could use TCP (if is does not disturb
> the parallel UDP flow in LwIP) or use UDP as well and implement the
> retries in the application.
> 
> 

If occasional missing packets aren't an issue, but timely delivery is,
then TCP isn't really the right choice. TCP is designed to get packets
there eventually (if possible), at the expense of possible delays. It is
also generally tuned for a long path (and thus the long delays for
re-transmission).

[toc] | [prev] | [next] | [standalone]

#28854

On 23.9.19 14:13, Richard Damon wrote:
> On 9/23/19 4:37 AM, Stef wrote:
>> On 2019-09-20 upsidedown@downunder.com wrote in comp.arch.embedded:
>>> Have you looked at the Ethernet traffic with e.g. Wireshark ? Look for
>>> TCP retransmissions. If each serial line pumps data at 2 Mbit/s
>>> without any flow control, a TCP retransmission will upset the Ethernet
>>> traffic for a while, possibly overflowing buffers. Anyway, if retrials
>>> are expected, the gross (ethernet) speed must be several times larger
>>> than the net data rate.
>>
>> No, not done that yet, but indeed something I need to do.
>>
>>> Is it possible to use UDP and accept that some packets may be lost ?
>>> Include a serial number in each frame and the receiver can detect
>>> missing packets (that have failed CRC test) and mark the missing data
>>> accordingly. This way the net speed can be much closer to the raw data
>>> rate.
>>
>> That is something that came to my mind just after posting the question.
>> Would UDP be a better match for this application? The whole TCP
>> retransmission and timers may be the mechanisms causing the problem.
>> I think the standard TCP timeouts ar far too slow for this kind of
>> thing? Think I saw a 250 ms timer tick in the LwIP docs?
>>
>> Missing packets may not be a problem. It is measurement data and if
>> any re-transmit takes over 40 ms there is no point anyway, there is
>> already new data. On a much slower rate, some control connection does
>> have to be reliable. So that could use TCP (if is does not disturb
>> the parallel UDP flow in LwIP) or use UDP as well and implement the
>> retries in the application.
>>
>>
> 
> If occasional missing packets aren't an issue, but timely delivery is,
> then TCP isn't really the right choice. TCP is designed to get packets
> there eventually (if possible), at the expense of possible delays. It is
> also generally tuned for a long path (and thus the long delays for
> re-transmission).


TCP will get the octets (bytes) there in the correct order, but
it is explicitly specified that it does not transfer the packet
boundaries. Many TCP stacks transfer the data in the same size
chunks the user gives them for transport, but it is allowed to
re-package the octet stream in any way the routers under way feel
fit.

UDP does not need any more checksumming, the Ethernet, IP and UDP
layers have already taken care of that, discarding bad datagrams.
There are good reasons to add packet sequence numbering to the
datagrams sent, to track possibly discarded datagrams.

-- 

-TV

[toc] | [prev] | [next] | [standalone]

#28852

On 23/09/2019 10:37, Stef wrote:
> On 2019-09-20 upsidedown@downunder.com wrote in comp.arch.embedded:
>> On Fri, 20 Sep 2019 18:27:52 +0200, Stef
>> <stef33d@yahooI-N-V-A-L-I-D.com.invalid> wrote:
>>
>>> For transportation of several high speed serial ports, we want to use a
>>> serial over ethernet approach. At a minimum I need to transfer four ports
>>> with 512 byte packets that come in every 40 ms at 2 Mbps.
>>>
>>> Search for a ready to use (for test at least) device has turned up nothing.
>>> Standard devices usually go up to 230 kbps, some to 1 Mbps. Most faster
>>> ones are single port only. And it should be a small device, embeddable,
>>> logic level is all that is needed. If anyone knows of a usable device...
>>>
>>> Since the board that this is on incorparates an ethernet capable controller
>>> (LPC4088) that has not that much work to do richt now, I thought it should
>>> be possible to add ethernet to it.
>>>
>>> For testing I got an LPC4088 demo board with ethernet interface and got
>>> LwIP running on it (stand alone, no OS).
>>>
>>> Got it working OK to simulate 4 ports sending slow data to 4 TCP ports
>>> that can be opened with a PC terminal program.
>>>
>>> But when I turn up the data volume and packet rate, it stops working.
>>> In some cases stuck in the hard fault handler, in other cases in a
>>> timed waiting loop in the LwIP stack.
>>>
>>> At this moment it is running fine with 2 ports open and a 512 byte packet
>>> every 100 ms on each. When I increase the speed to 50 ms, it stops after
>>> a while.
>>>
>>> I have no previous experience with LwIP so where to start looking?
>>> - First of all does it sound feasable over a 100 Mbit ethernet
>>>  connection using TCP?
>>> - What can be tuned in LwIP to improve this?
>>>  I have already increased PBUF_POOL_SIZE from default 7 to 64 and
>>>  MEM_SIZE from 12k to 32k
>>
>> Have you looked at the Ethernet traffic with e.g. Wireshark ? Look for
>> TCP retransmissions. If each serial line pumps data at 2 Mbit/s
>> without any flow control, a TCP retransmission will upset the Ethernet
>> traffic for a while, possibly overflowing buffers. Anyway, if retrials
>> are expected, the gross (ethernet) speed must be several times larger
>> than the net data rate.
> 
> No, not done that yet, but indeed something I need to do.
> 
>> Is it possible to use UDP and accept that some packets may be lost ?
>> Include a serial number in each frame and the receiver can detect
>> missing packets (that have failed CRC test) and mark the missing data
>> accordingly. This way the net speed can be much closer to the raw data
>> rate.
> 
> That is something that came to my mind just after posting the question.
> Would UDP be a better match for this application? The whole TCP
> retransmission and timers may be the mechanisms causing the problem.
> I think the standard TCP timeouts ar far too slow for this kind of
> thing? Think I saw a 250 ms timer tick in the LwIP docs?

TCP/IP timeouts are slow, yes - you need to consider if that will be an
issue.  But TCP/IP gives you better guarantees about delivery - either
the packets will all arrive, with correct contents, sorted in the right
order, or you will get a clear failure indication for the whole
connection.  With UDP, you need to handle this manually - you might need
more checksumming, re-ordering, re-tries, etc., depending on the
application.  But UDP has fewer overheads, and with manual coding comes
greater control and flexibility.

TCP/IP is also /much/ easier to handle over longer networks - through
NAT routers, VPN's, ssh tunnels, and the like.

Note that if you are getting retries or re-ordering of TCP/IP packets
while you are on local network, you have something badly wrong with the
software or the network - retries should be very rare.

> 
> Missing packets may not be a problem. It is measurement data and if
> any re-transmit takes over 40 ms there is no point anyway, there is
> already new data. On a much slower rate, some control connection does
> have to be reliable. So that could use TCP (if is does not disturb
> the parallel UDP flow in LwIP) or use UDP as well and implement the
> retries in the application.
> 
> 

[toc] | [prev] | [next] | [standalone]

#28853

On 9/23/2019 15:04, David Brown wrote:
> .....
> 
> Note that if you are getting retries or re-ordering of TCP/IP packets
> while you are on local network, you have something badly wrong with the
> software or the network - retries should be very rare.
> 

Generally true but you can be easily surprised - like I was a while ago.
A new windows 10 based laptop, about top of the line Acer had less than
a year ago, was communicating with my DPS system a lot slower than
my older windows XP based laptop (also Acer, about 10 years old).
This via ftp at 100 Mbps, all 3 devices connected to the same switch
(I let some huge file, 1G or more, transfer and watch how it goes).

I started to investigate and it turned out the windows 10 machine
was just losing too many packets and my (well, that of the DPS machine)
retries were not aggressive enough. Making them so took care of that,
but the surprise was there allright. Did not investigate seriously the
windows side as for me it was easier to do it on the DPS side.

Dimiter

======================================================
Dimiter Popoff, TGI             http://www.tgi-sci.com
======================================================
http://www.flickr.com/photos/didi_tgi/

[toc] | [prev] | [next] | [standalone]

#28855

On 23/09/2019 16:03, Dimiter_Popoff wrote:
> On 9/23/2019 15:04, David Brown wrote:
>> .....
>>
>> Note that if you are getting retries or re-ordering of TCP/IP packets
>> while you are on local network, you have something badly wrong with the
>> software or the network - retries should be very rare.
>>
> 
> Generally true but you can be easily surprised - like I was a while ago.
> A new windows 10 based laptop, about top of the line Acer had less than
> a year ago, was communicating with my DPS system a lot slower than
> my older windows XP based laptop (also Acer, about 10 years old).
> This via ftp at 100 Mbps, all 3 devices connected to the same switch
> (I let some huge file, 1G or more, transfer and watch how it goes).
> 
> I started to investigate and it turned out the windows 10 machine
> was just losing too many packets and my (well, that of the DPS machine)
> retries were not aggressive enough. Making them so took care of that,
> but the surprise was there allright. Did not investigate seriously the
> windows side as for me it was easier to do it on the DPS side.
> 

There you go - something was badly wrong with the network.  It had a 
Windows 10 machine on it :-)

[toc] | [prev] | [next] | [standalone]

#28859

>> On 9/23/2019 15:04, David Brown wrote:
...
> There you go - something was badly wrong with the network.  It had a 
> Windows 10 machine on it :-)

That might be my problem as well. ;-)

Turned on LwIP statistics and did a first test with a single simulated
serial port (data not actually coming in, just generated by processor
itself).

LwIP runs on an LPC4088 and 'serial port' is opened by Docklight running
on a W10 PC.

First, 512 byte packets a 1 second intervals and printing the stats every
5 seconds.
- TCP
        xmit: 44
        recv: 45
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 0
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

No memerr and xmit and recv at a constant 1 difference.


Then a test with one port, 10 ms intervals:
- (Aparently) Immediate hangup, stacks keeps going into msDelay().


Then one port, 20 ms intervals;
Runs for a while but ultimately ends up in repeated msDelay().
Last statistics output (run time not always equal):
- TCP
        xmit: 1088
        recv: 827
        fw: 0
        drop: 0
        chkerr: 0
        lenerr: 0
        memerr: 270
        rterr: 0
        proterr: 0
        opterr: 0
        err: 0
        cachehit: 0

In the previous outputs you can see the gap between xmit and recv
increasing as well as the memerr.

Looks like the PC does just not respond fast enough, forcing LwIP to
keep the packet until an ACK is received?


Last 6 lines from the Wireshark packet window when the above stops:

6600	78.330701	192.168.125.130	192.168.125.128	TCP	1078	10001 → 51826 [PSH, ACK] Seq=975873 Ack=1 Win=5840 Len=1024
6601	78.379841	192.168.125.128	192.168.125.130	TCP	54	51826 → 10001 [ACK] Seq=1 Ack=976897 Win=63216 Len=0
6602	78.379850	192.168.125.128	192.168.125.130	TCP	54	[TCP Dup ACK 6601#1] 51826 → 10001 [ACK] Seq=1 Ack=976897 Win=63216 Len=0
6607	78.413921	192.168.125.130	192.168.125.128	TCP	566	10001 → 51826 [PSH, ACK] Seq=976897 Ack=1 Win=5840 Len=512
6609	78.459611	192.168.125.128	192.168.125.130	TCP	54	51826 → 10001 [ACK] Seq=1 Ack=977409 Win=64240 Len=0
6610	78.459619	192.168.125.128	192.168.125.130	TCP	54	[TCP Dup ACK 6609#1] 51826 → 10001 [ACK] Seq=1 Ack=977409 Win=64240 Len=0

What seems weird to me is that all ACK are followed by Dup ACK. Is this normal? See that al lot in other TCP traffic as well, but not on all ACK.


-- 
Stef    (remove caps, dashes and .invalid from e-mail address to reply by mail)

There is no opinion so absurd that some philosopher will not express it.
		-- Marcus Tullius Cicero, "Ad familiares"

[toc] | [prev] | [next] | [standalone]

#28860

On Tue, 24 Sep 2019 14:22:51 +0200, Stef
<stef33d@yahooI-N-V-A-L-I-D.com.invalid> wrote:

>
>Then one port, 20 ms intervals;
>Runs for a while but ultimately ends up in repeated msDelay().
>Last statistics output (run time not always equal):
>- TCP
>        xmit: 1088
>        recv: 827
>        fw: 0
>        drop: 0
>        chkerr: 0
>        lenerr: 0
>        memerr: 270
>        rterr: 0
>        proterr: 0
>        opterr: 0
>        err: 0
>        cachehit: 0
>
>In the previous outputs you can see the gap between xmit and recv
>increasing as well as the memerr.
>
>Looks like the PC does just not respond fast enough, forcing LwIP to
>keep the packet until an ACK is received?

Make sure that you do not cause excessive loading on PC by causing all
kinds of activity on the PC.

If you use Telnet (or other display program) as the TCP client, any
screen updates will cause loading the PC. At least minimize the Telnet
window to reduce the PC loading. The same applies to Wireshark, just
capture to disk. If possible use a separate PC for Telnet and
Wireshark, but you need a hub, not a switch, so that Wireshark will
see the traffic between the embedded device and Telnet PC.

>
>
>Last 6 lines from the Wireshark packet window when the above stops:
>
>6600	78.330701	192.168.125.130	192.168.125.128	TCP	1078	10001 ? 51826 [PSH, ACK] Seq=975873 Ack=1 Win=5840 Len=1024
>6601	78.379841	192.168.125.128	192.168.125.130	TCP	54	51826 ? 10001 [ACK] Seq=1 Ack=976897 Win=63216 Len=0
>6602	78.379850	192.168.125.128	192.168.125.130	TCP	54	[TCP Dup ACK 6601#1] 51826 ? 10001 [ACK] Seq=1 Ack=976897 Win=63216 Len=0
>6607	78.413921	192.168.125.130	192.168.125.128	TCP	566	10001 ? 51826 [PSH, ACK] Seq=976897 Ack=1 Win=5840 Len=512
>6609	78.459611	192.168.125.128	192.168.125.130	TCP	54	51826 ? 10001 [ACK] Seq=1 Ack=977409 Win=64240 Len=0
>6610	78.459619	192.168.125.128	192.168.125.130	TCP	54	[TCP Dup ACK 6609#1] 51826 ? 10001 [ACK] Seq=1 Ack=977409 Win=64240 Len=0

Look at the WireShark sequence numbering (first column), why is there
a jump between 6602 and 6607 ? Is this due to display filtering? Use
capture filtering to reduce Wireshark loading.

Look at timestamps (second column), the last digit is in microseconds.
There seems to be a much larger gap than 20 ms between to embedded
sends. The time between data and Ack is much larger than 20 ms. 

The Ack sequence number has fallen seriously behind data messages.
Apparently the sender fails to buffer about 1000 data frames (500 KB)
before being acknowledged, so clearly some messages are loss.

This suggest that the Telnet screen update slows down the traffic, so
minimize Telnet screen.

>
>What seems weird to me is that all ACK are followed by Dup ACK. Is this normal? See that al lot in other TCP traffic as well, but not on all ACK.
 
The difference between Ack and duplicate Ack is only 8-9 us. Strange.

[toc] | [prev] | [next] | [standalone]

#28862

On 25/09/2019 06:55, upsidedown@downunder.com wrote:
> On Tue, 24 Sep 2019 14:22:51 +0200, Stef
> <stef33d@yahooI-N-V-A-L-I-D.com.invalid> wrote:
> 
>>
>> Then one port, 20 ms intervals;
>> Runs for a while but ultimately ends up in repeated msDelay().
>> Last statistics output (run time not always equal):
>> - TCP
>>        xmit: 1088
>>        recv: 827
>>        fw: 0
>>        drop: 0
>>        chkerr: 0
>>        lenerr: 0
>>        memerr: 270
>>        rterr: 0
>>        proterr: 0
>>        opterr: 0
>>        err: 0
>>        cachehit: 0
>>
>> In the previous outputs you can see the gap between xmit and recv
>> increasing as well as the memerr.
>>
>> Looks like the PC does just not respond fast enough, forcing LwIP to
>> keep the packet until an ACK is received?
> 
> Make sure that you do not cause excessive loading on PC by causing all
> kinds of activity on the PC.
> 
> If you use Telnet (or other display program) as the TCP client, any
> screen updates will cause loading the PC. At least minimize the Telnet
> window to reduce the PC loading. The same applies to Wireshark, just
> capture to disk. If possible use a separate PC for Telnet and
> Wireshark, but you need a hub, not a switch, so that Wireshark will
> see the traffic between the embedded device and Telnet PC.
> 

Nonsense.  A modern PC will handle this with barely a blip on its
processor usage graph.  You /might/ see some limitations if you are
using a 2 GB "Intel compute stick" with a tiny Celeron and Windows 10.
But assuming the Win 10 has halfway sane specs, it is going to be
absolutely fine.  Run Task Manager and watch the graphs of processor and
memory usage to confirm that.

>>
>>
>> Last 6 lines from the Wireshark packet window when the above stops:
>>
>> 6600	78.330701	192.168.125.130	192.168.125.128	TCP	1078	10001 ? 51826 [PSH, ACK] Seq=975873 Ack=1 Win=5840 Len=1024
>> 6601	78.379841	192.168.125.128	192.168.125.130	TCP	54	51826 ? 10001 [ACK] Seq=1 Ack=976897 Win=63216 Len=0
>> 6602	78.379850	192.168.125.128	192.168.125.130	TCP	54	[TCP Dup ACK 6601#1] 51826 ? 10001 [ACK] Seq=1 Ack=976897 Win=63216 Len=0
>> 6607	78.413921	192.168.125.130	192.168.125.128	TCP	566	10001 ? 51826 [PSH, ACK] Seq=976897 Ack=1 Win=5840 Len=512
>> 6609	78.459611	192.168.125.128	192.168.125.130	TCP	54	51826 ? 10001 [ACK] Seq=1 Ack=977409 Win=64240 Len=0
>> 6610	78.459619	192.168.125.128	192.168.125.130	TCP	54	[TCP Dup ACK 6609#1] 51826 ? 10001 [ACK] Seq=1 Ack=977409 Win=64240 Len=0
> 
> Look at the WireShark sequence numbering (first column), why is there
> a jump between 6602 and 6607 ? Is this due to display filtering? Use
> capture filtering to reduce Wireshark loading.
> 

If you are doing large captures, then capture filtering makes a
difference by reduce the quantity of data.  For short captures, it will
make no measurable difference.

> Look at timestamps (second column), the last digit is in microseconds.
> There seems to be a much larger gap than 20 ms between to embedded
> sends. The time between data and Ack is much larger than 20 ms. 
> 
> The Ack sequence number has fallen seriously behind data messages.
> Apparently the sender fails to buffer about 1000 data frames (500 KB)
> before being acknowledged, so clearly some messages are loss.
> 
> This suggest that the Telnet screen update slows down the traffic, so
> minimize Telnet screen.

No, it suggests things are failing at the LWIP side, especially when we
see the LWIP statistics full of memerr counts.  A memerr occurs when
LWIP can't find a free buffer of some sort (there are several types
used) to handle a packet coming in or going out - it has no choice but
to drop that packet.  This will lead to resends and delays.

> 
>>
>> What seems weird to me is that all ACK are followed by Dup ACK. Is this normal? See that al lot in other TCP traffic as well, but not on all ACK.
>  
> The difference between Ack and duplicate Ack is only 8-9 us. Strange.
> 

It is, I think, due to lost packets.  (Google "wireshark dup ack" for
suggestions.)

[toc] | [prev] | [next] | [standalone]

#28868

On 2019-09-25 David Brown wrote in comp.arch.embedded:
> On 25/09/2019 06:55, upsidedown@downunder.com wrote:
>> On Tue, 24 Sep 2019 14:22:51 +0200, Stef
>> <stef33d@yahooI-N-V-A-L-I-D.com.invalid> wrote:
>> 
>>>
>>> What seems weird to me is that all ACK are followed by Dup ACK. Is this normal? See that al lot in other TCP traffic as well, but not on all ACK.
>>  
>> The difference between Ack and duplicate Ack is only 8-9 us. Strange.
>> 
>
> It is, I think, due to lost packets.  (Google "wireshark dup ack" for
> suggestions.)

Even stranger, when I just ping the board:

229	5.081046	192.168.125.128	192.168.125.133	ICMP	74	Echo (ping) request  id=0x0001, seq=3492/41997, ttl=128 (no response found!)
230	5.081051	192.168.125.128	192.168.125.133	ICMP	74	Echo (ping) request  id=0x0001, seq=3492/41997, ttl=128 (reply in 234)
234	5.081615	192.168.125.133	192.168.125.128	ICMP	74	Echo (ping) reply    id=0x0001, seq=3492/41997, ttl=255 (request in 230)

A duplicate request goes out from W10 with only 5 us difference? The same
happens on the rest of the pings, one second apart.

Nothing to do with the eval kit with LwIP, if I ping my router:

380	7.010693	192.168.125.128	192.168.125.254	ICMP	74	Echo (ping) request  id=0x0001, seq=3539/54029, ttl=128 (no response found!)380	7.010693	192.168.125.128	192.168.125.254	ICMP	74	Echo (ping) request  id=0x0001, seq=3539/54029, ttl=128 (no response found!)
381	7.010700	192.168.125.128	192.168.125.254	ICMP	74	Echo (ping) request  id=0x0001, seq=3539/54029, ttl=128 (reply in 382)
382	7.011670	192.168.125.254	192.168.125.128	ICMP	74	Echo (ping) reply    id=0x0001, seq=3539/54029, ttl=255 (request in 381)

And after looking a little better, it seems to happen on almost all
packets transmitted from this PC, but not on TLSv1.2 packets.

Something weird in W10 or Wireshark? Or the PC network interface?


-- 
Stef    (remove caps, dashes and .invalid from e-mail address to reply by mail)

Intolerance is the last defense of the insecure.

[toc] | [prev] | [next] | [standalone]

#28863

On 2019-09-25 upsidedown@downunder.com wrote in comp.arch.embedded:
> On Tue, 24 Sep 2019 14:22:51 +0200, Stef
><stef33d@yahooI-N-V-A-L-I-D.com.invalid> wrote:
>
>>
>>Then one port, 20 ms intervals;
>>Runs for a while but ultimately ends up in repeated msDelay().
>>Last statistics output (run time not always equal):
>>- TCP
>>        xmit: 1088
>>        recv: 827
>>        fw: 0
>>        drop: 0
>>        chkerr: 0
>>        lenerr: 0
>>        memerr: 270
>>        rterr: 0
>>        proterr: 0
>>        opterr: 0
>>        err: 0
>>        cachehit: 0
>>
>>In the previous outputs you can see the gap between xmit and recv
>>increasing as well as the memerr.
>>
>>Looks like the PC does just not respond fast enough, forcing LwIP to
>>keep the packet until an ACK is received?
>
> Make sure that you do not cause excessive loading on PC by causing all
> kinds of activity on the PC.

PC is not heavily loaded, debugger, terminal window, wireshark, a
browser with internet radio, not much more.

> If you use Telnet (or other display program) as the TCP client, any
> screen updates will cause loading the PC. At least minimize the Telnet
> window to reduce the PC loading. The same applies to Wireshark, just
> capture to disk. If possible use a separate PC for Telnet and
> Wireshark, but you need a hub, not a switch, so that Wireshark will
> see the traffic between the embedded device and Telnet PC.
>
>>
>>
>>Last 6 lines from the Wireshark packet window when the above stops:
>>
>>6600	78.330701	192.168.125.130	192.168.125.128	TCP	1078	10001 ? 51826 [PSH, ACK] Seq=975873 Ack=1 Win=5840 Len=1024
>>6601	78.379841	192.168.125.128	192.168.125.130	TCP	54	51826 ? 10001 [ACK] Seq=1 Ack=976897 Win=63216 Len=0
>>6602	78.379850	192.168.125.128	192.168.125.130	TCP	54	[TCP Dup ACK 6601#1] 51826 ? 10001 [ACK] Seq=1 Ack=976897 Win=63216 Len=0
>>6607	78.413921	192.168.125.130	192.168.125.128	TCP	566	10001 ? 51826 [PSH, ACK] Seq=976897 Ack=1 Win=5840 Len=512
>>6609	78.459611	192.168.125.128	192.168.125.130	TCP	54	51826 ? 10001 [ACK] Seq=1 Ack=977409 Win=64240 Len=0
>>6610	78.459619	192.168.125.128	192.168.125.130	TCP	54	[TCP Dup ACK 6609#1] 51826 ? 10001 [ACK] Seq=1 Ack=977409 Win=64240 Len=0
>
> Look at the WireShark sequence numbering (first column), why is there
> a jump between 6602 and 6607 ? Is this due to display filtering? Use
> capture filtering to reduce Wireshark loading.

Yes, display filtering is used.

> Look at timestamps (second column), the last digit is in microseconds.
> There seems to be a much larger gap than 20 ms between to embedded
> sends. The time between data and Ack is much larger than 20 ms. 

Why the sends are more than 20 ms apart, I don't know. The sender loop
tries to send a packet every 20 ms. Must be due to the lagging ACKs
and limited buffer space at the sender?

> The Ack sequence number has fallen seriously behind data messages.
> Apparently the sender fails to buffer about 1000 data frames (500 KB)
> before being acknowledged, so clearly some messages are loss.

OK

> This suggest that the Telnet screen update slows down the traffic, so
> minimize Telnet screen.

I don't think the terminal (just a plain serial terminal, no telnet) is
slowing the PC down too much, see below.


>>
>>What seems weird to me is that all ACK are followed by Dup ACK. Is this normal? See that al lot in other TCP traffic as well, but not on all ACK.
>  
> The difference between Ack and duplicate Ack is only 8-9 us. Strange.

Yes.

As discussed in other parts of this thread, UDP might be a better match
for this application, so I tried that.

A test with 4 ports sending 512 byte packets at 10 ms intervals was
successful (mostly). During this test 4 terminal screens were updating
and wireshark was running with display filtering. So the PC seems to
have no problem keeping up.

7888	23.208722	192.168.125.132	192.168.125.128	UDP	554	10001 → 10001 Len=512
7889	23.209536	192.168.125.132	192.168.125.128	UDP	554	10002 → 10002 Len=512
7890	23.210885	192.168.125.132	192.168.125.128	UDP	554	10003 → 10003 Len=512
7891	23.211184	192.168.125.132	192.168.125.128	UDP	554	10004 → 10004 Len=512
7892	23.219123	192.168.125.132	192.168.125.128	UDP	554	10001 → 10001 Len=512
7893	23.219593	192.168.125.132	192.168.125.128	UDP	554	10002 → 10002 Len=512
7894	23.220354	192.168.125.132	192.168.125.128	UDP	554	10003 → 10003 Len=512
7895	23.221256	192.168.125.132	192.168.125.128	UDP	554	10004 → 10004 Len=512

So it looks like UDP is indeed a better match for this application.
Have to implement some stuff in the application that TCP took care of.
Like opening a port and handling packet loss. Packet loss was already
an issue that needed to be handled by the final application. This is
because (as an other poster already mentioned) the data is just serial
data with no loss detection to begin with.

There was still an issue that at all tested speeds (10/20/40/100 ms) the
embedded side goes in a hard fault handler after a while. Aprox 30
seconds for 10/20/40 ms and 2 minutes for 100 ms. This was probably due
to my sending the 4 packets for each interval back to back without calling
any of the ethrnet handlers in between. If I call the handlers after each
packet, the hangups disapper. So I may have to revistit the TCP version to
check if that was a problem there as well. :-(


-- 
Stef    (remove caps, dashes and .invalid from e-mail address to reply by mail)

I *____knew* I had some reason for not logging you off... If I could just
remember what it was.

[toc] | [prev] | [next] | [standalone]

#28864

On 2019-09-25 Stef wrote in comp.arch.embedded:
...
> There was still an issue that at all tested speeds (10/20/40/100 ms) the
> embedded side goes in a hard fault handler after a while. Aprox 30
> seconds for 10/20/40 ms and 2 minutes for 100 ms. This was probably due
> to my sending the 4 packets for each interval back to back without calling
> any of the ethrnet handlers in between. If I call the handlers after each
> packet, the hangups disapper. So I may have to revistit the TCP version to
> check if that was a problem there as well. :-(

For get that, the failing TCP test was running only one port at 20 ms, so
it did not have the back to back problem. The failure was also different,
a repeating wait loop, not a hard fault.


-- 
Stef    (remove caps, dashes and .invalid from e-mail address to reply by mail)

I drink to make other people interesting.
		-- George Jean Nathan

[toc] | [prev] | [next] | [standalone]

#28861

On 24/09/2019 14:22, Stef wrote:
>>> On 9/23/2019 15:04, David Brown wrote:
> ...
>> There you go - something was badly wrong with the network.  It had a 
>> Windows 10 machine on it :-)
> 
> That might be my problem as well. ;-)

I was half joking, but only half.  Modern systems are very chatty on the
network - with newer OS's being worse than old ones.  Even Linux desktop
systems produce a lot of traffic, though my experience is that Windows
is much worse.  (I have not looked at Mac's, and server systems are
usually a lot quieter.)  Printers and other devices also chatter
continuously.  Some of this is necessary low-level traffic, like ARP and
DHCP packets.  Some is from the several dozen different "automatic
configuration" and "name service" protocols that everyone has to
implement, but almost no one uses.  Some is from applications like
DropBox or Steam trying to find neighbours on the network.

This all means there is a ridiculous amount of traffic on typical
Ethernet networks, with 90% of it being basically useless, and a good
deal of it being broadcast to all nodes.

> 
> Turned on LwIP statistics and did a first test with a single simulated
> serial port (data not actually coming in, just generated by processor
> itself).
> 
> LwIP runs on an LPC4088 and 'serial port' is opened by Docklight running
> on a W10 PC.
> 
> First, 512 byte packets a 1 second intervals and printing the stats every
> 5 seconds.
> - TCP
>         xmit: 44
>         recv: 45
>         fw: 0
>         drop: 0
>         chkerr: 0
>         lenerr: 0
>         memerr: 0
>         rterr: 0
>         proterr: 0
>         opterr: 0
>         err: 0
>         cachehit: 0
> 
> No memerr and xmit and recv at a constant 1 difference.
> 
> 
> Then a test with one port, 10 ms intervals:
> - (Aparently) Immediate hangup, stacks keeps going into msDelay().
> 
> 
> Then one port, 20 ms intervals;
> Runs for a while but ultimately ends up in repeated msDelay().
> Last statistics output (run time not always equal):
> - TCP
>         xmit: 1088
>         recv: 827
>         fw: 0
>         drop: 0
>         chkerr: 0
>         lenerr: 0
>         memerr: 270
>         rterr: 0
>         proterr: 0
>         opterr: 0
>         err: 0
>         cachehit: 0
> 
> In the previous outputs you can see the gap between xmit and recv
> increasing as well as the memerr.

If you are getting any memerr results, you have not given LWIP enough
resources.  I don't know off-hand what buffer types or other resources
can lead to memerr counts, but you are low on something.  These
statistics show that a quarter of your communications are failing to get
the buffers they need - not good at all.

> 
> Looks like the PC does just not respond fast enough, forcing LwIP to
> keep the packet until an ACK is received?

PC's are ridiculously fast.  Despite the network chatter, and despite
the mess of silliness Windows 10 machines are always running with their
absurd "start menu" full of adverts and other junk, they should be
handling packets in a fraction of a millisecond.  When I did the testing
I mentioned before, the test programs on the PC side (Linux rather than
Windows) was in Python, and not even particularly efficient Python - it
even had manual "sleep" calls.  Handling was well under a millisecond
per transaction.

> 
> 
> Last 6 lines from the Wireshark packet window when the above stops:
> 
> 6600	78.330701	192.168.125.130	192.168.125.128	TCP	1078	10001 → 51826 [PSH, ACK] Seq=975873 Ack=1 Win=5840 Len=1024
> 6601	78.379841	192.168.125.128	192.168.125.130	TCP	54	51826 → 10001 [ACK] Seq=1 Ack=976897 Win=63216 Len=0
> 6602	78.379850	192.168.125.128	192.168.125.130	TCP	54	[TCP Dup ACK 6601#1] 51826 → 10001 [ACK] Seq=1 Ack=976897 Win=63216 Len=0
> 6607	78.413921	192.168.125.130	192.168.125.128	TCP	566	10001 → 51826 [PSH, ACK] Seq=976897 Ack=1 Win=5840 Len=512
> 6609	78.459611	192.168.125.128	192.168.125.130	TCP	54	51826 → 10001 [ACK] Seq=1 Ack=977409 Win=64240 Len=0
> 6610	78.459619	192.168.125.128	192.168.125.130	TCP	54	[TCP Dup ACK 6609#1] 51826 → 10001 [ACK] Seq=1 Ack=977409 Win=64240 Len=0
> 
> What seems weird to me is that all ACK are followed by Dup ACK. Is this normal? See that al lot in other TCP traffic as well, but not on all ACK.
> 
> 

I /think/ the dup ack's are the result of your board failing to handle
some packets, probably related to limited buffers (giving the memerr
counts).

[toc] | [prev] | [next] | [standalone]

#28865

On 2019-09-25 David Brown wrote in comp.arch.embedded:
> On 24/09/2019 14:22, Stef wrote:
>>>> On 9/23/2019 15:04, David Brown wrote:
... 
>> Then one port, 20 ms intervals;
>> Runs for a while but ultimately ends up in repeated msDelay().
>> Last statistics output (run time not always equal):
>> - TCP
>>         xmit: 1088
>>         recv: 827
>>         fw: 0
>>         drop: 0
>>         chkerr: 0
>>         lenerr: 0
>>         memerr: 270
>>         rterr: 0
>>         proterr: 0
>>         opterr: 0
>>         err: 0
>>         cachehit: 0
>> 
>> In the previous outputs you can see the gap between xmit and recv
>> increasing as well as the memerr.
>
> If you are getting any memerr results, you have not given LWIP enough
> resources.  I don't know off-hand what buffer types or other resources
> can lead to memerr counts, but you are low on something.  These
> statistics show that a quarter of your communications are failing to get
> the buffers they need - not good at all.

No, not good. But 64 PBUF_POOL_SIZE and 32k mem should be okay for the
few packets that could be in-flight if the PC responds fast? I see no
other obviously releated tunable items in my lwipopts.h

But even from the start of transmission, the packets come out too slow.
Sometimes 512 byte packets after a > 20ms interval, mostly 1024 byte
packets at > 40 ms (mostly 50 ms). So this would quickly cause the buffers
to fill up as the I keep feeding them at 20 ms intervals.

I'll proceed with the UDP approach for now. Looks a lot cleaner.


-- 
Stef    (remove caps, dashes and .invalid from e-mail address to reply by mail)

"You're very sure of your facts, " he said at last, "I 
couldn't trust the thinking of a man who takes the Universe 
- if there is one - for granted. "

[toc] | [prev] | [next] | [standalone]

#28866

On 25/09/2019 13:41, Stef wrote:
> On 2019-09-25 David Brown wrote in comp.arch.embedded:
>> On 24/09/2019 14:22, Stef wrote:
>>>>> On 9/23/2019 15:04, David Brown wrote:
> ... 
>>> Then one port, 20 ms intervals;
>>> Runs for a while but ultimately ends up in repeated msDelay().
>>> Last statistics output (run time not always equal):
>>> - TCP
>>>         xmit: 1088
>>>         recv: 827
>>>         fw: 0
>>>         drop: 0
>>>         chkerr: 0
>>>         lenerr: 0
>>>         memerr: 270
>>>         rterr: 0
>>>         proterr: 0
>>>         opterr: 0
>>>         err: 0
>>>         cachehit: 0
>>>
>>> In the previous outputs you can see the gap between xmit and recv
>>> increasing as well as the memerr.
>>
>> If you are getting any memerr results, you have not given LWIP enough
>> resources.  I don't know off-hand what buffer types or other resources
>> can lead to memerr counts, but you are low on something.  These
>> statistics show that a quarter of your communications are failing to get
>> the buffers they need - not good at all.
> 
> No, not good. But 64 PBUF_POOL_SIZE and 32k mem should be okay for the
> few packets that could be in-flight if the PC responds fast? I see no
> other obviously releated tunable items in my lwipopts.h
> 
> But even from the start of transmission, the packets come out too slow.
> Sometimes 512 byte packets after a > 20ms interval, mostly 1024 byte
> packets at > 40 ms (mostly 50 ms). So this would quickly cause the buffers
> to fill up as the I keep feeding them at 20 ms intervals.
> 
> I'll proceed with the UDP approach for now. Looks a lot cleaner.
> 
> 

What are your other lwip config numbers?  I have:

#define MEM_SIZE                (48*1024)
#define MEMP_NUM_PBUF           32
#define MEMP_NUM_UDP_PCB        6
#define MEMP_NUM_TCP_PCB        128
#define MEMP_NUM_TCP_PCB_LISTEN 8
#define MEMP_NUM_TCP_SEG        16
#define MEMP_NUM_SYS_TIMEOUT    10
#define PBUF_POOL_SIZE          32
#define PBUF_POOL_BUFSIZE       1518

I can't say I have studied the usage in detail, so I might have much
more than I need for some of these.  But if any of your values are
hugely lower than mine, check them to be sure.  /Something/ is giving
you memory allocation errors, and you need to find and fix that something.

It is also possible that the rest of program structure means you are
getting slow handling of the packets - I can't guess anything about
that, because I don't know your code at all.

[toc] | [prev] | [next] | [standalone]

#28867

On 2019-09-25 David Brown wrote in comp.arch.embedded:
...
> What are your other lwip config numbers?  I have:
>
> #define MEM_SIZE                (48*1024)
> #define MEMP_NUM_PBUF           32
> #define MEMP_NUM_UDP_PCB        6
> #define MEMP_NUM_TCP_PCB        128
> #define MEMP_NUM_TCP_PCB_LISTEN 8
> #define MEMP_NUM_TCP_SEG        16
> #define MEMP_NUM_SYS_TIMEOUT    10
> #define PBUF_POOL_SIZE          32
> #define PBUF_POOL_BUFSIZE       1518

In lwipopt.h:
#define MEM_SIZE                        (32 * 1024)
#define MEMP_NUM_SYS_TIMEOUT            300
#define PBUF_POOL_SIZE                  64

Most were not in my lwipopt.h, so at default from opt.h:
#define MEMP_NUM_PBUF                   16
#define MEMP_NUM_UDP_PCB                4
#define MEMP_NUM_TCP_PCB                5
#define MEMP_NUM_TCP_PCB_LISTEN         8
#define MEMP_NUM_TCP_SEG                16
#define PBUF_POOL_BUFSIZE               LWIP_MEM_ALIGN_SIZE(TCP_MSS+40+PBUF_LINK_HLEN)

The last one expands to: ((1460+40+14)+3) & ~3 = 1516

> I can't say I have studied the usage in detail, so I might have much
> more than I need for some of these.  But if any of your values are
> hugely lower than mine, check them to be sure.  /Something/ is giving
> you memory allocation errors, and you need to find and fix that something.

Tried changing MEMP_NUM_TCP_PCB, MEMP_NUM_PBUF and
MEMP_NUM_SYS_TIMEOUT to your values, but no change in behaviour.

> It is also possible that the rest of program structure means you are
> getting slow handling of the packets - I can't guess anything about
> that, because I don't know your code at all.

It's just a loop calling the lwip handlers (the standard stand alone
echo example) and a function that checks a timer for when to send a
(dummy) packet. So really lightweight, just to test if I can get packets
out at the required rate.

I'm giving up the TCP version for now and go with the UDP, which seems
to work fine (with the same settings). I'll give TCP another try some
time.

-- 
Stef    (remove caps, dashes and .invalid from e-mail address to reply by mail)

No hardware designer should be allowed to produce any piece of hardware
until three software guys have signed off for it.
		-- Andy Tanenbaum

[toc] | [prev] | [next] | [standalone]

#28856

On Mon, 23 Sep 2019 10:37:17 +0200, Stef
<stef33d@yahooI-N-V-A-L-I-D.com.invalid> wrote:

>On 2019-09-20 upsidedown@downunder.com wrote in comp.arch.embedded:
>> On Fri, 20 Sep 2019 18:27:52 +0200, Stef
>><stef33d@yahooI-N-V-A-L-I-D.com.invalid> wrote:
>>
>>>For transportation of several high speed serial ports, we want to use a
>>>serial over ethernet approach. At a minimum I need to transfer four ports
>>>with 512 byte packets that come in every 40 ms at 2 Mbps.
>>>
>>>Search for a ready to use (for test at least) device has turned up nothing.
>>>Standard devices usually go up to 230 kbps, some to 1 Mbps. Most faster
>>>ones are single port only. And it should be a small device, embeddable,
>>>logic level is all that is needed. If anyone knows of a usable device...
>>>
>>>Since the board that this is on incorparates an ethernet capable controller
>>>(LPC4088) that has not that much work to do richt now, I thought it should
>>>be possible to add ethernet to it.
>>>
>>>For testing I got an LPC4088 demo board with ethernet interface and got
>>>LwIP running on it (stand alone, no OS).
>>>
>>>Got it working OK to simulate 4 ports sending slow data to 4 TCP ports
>>>that can be opened with a PC terminal program.
>>>
>>>But when I turn up the data volume and packet rate, it stops working.
>>>In some cases stuck in the hard fault handler, in other cases in a
>>>timed waiting loop in the LwIP stack.
>>>
>>>At this moment it is running fine with 2 ports open and a 512 byte packet
>>>every 100 ms on each. When I increase the speed to 50 ms, it stops after
>>>a while.
>>>
>>>I have no previous experience with LwIP so where to start looking?
>>>- First of all does it sound feasable over a 100 Mbit ethernet
>>>  connection using TCP?
>>>- What can be tuned in LwIP to improve this?
>>>  I have already increased PBUF_POOL_SIZE from default 7 to 64 and
>>>  MEM_SIZE from 12k to 32k
>>
>> Have you looked at the Ethernet traffic with e.g. Wireshark ? Look for
>> TCP retransmissions. If each serial line pumps data at 2 Mbit/s
>> without any flow control, a TCP retransmission will upset the Ethernet
>> traffic for a while, possibly overflowing buffers. Anyway, if retrials
>> are expected, the gross (ethernet) speed must be several times larger
>> than the net data rate.
>
>No, not done that yet, but indeed something I need to do.

Looking more closely on your original question, the actual duty cycle
is less than 10 %, so you could do the same thing with four 230 kbit/s
streams, thus the speed requirement is no big deal. There are lots of
eth/serial converters supporting 4-8 serial lines at 230 kbit/s.


>> Is it possible to use UDP and accept that some packets may be lost ?
>> Include a serial number in each frame and the receiver can detect
>> missing packets (that have failed CRC test) and mark the missing data
>> accordingly. This way the net speed can be much closer to the raw data
>> rate.
>
>That is something that came to my mind just after posting the question.
>Would UDP be a better match for this application? 

Using MAC or UDP framing is just using Ethernet as some very high
speed serial connections. The advantage of UDP compared to MAC
addressing is that ARP and more user friendly IP addressed compared to
Ethernet MAC addresses can be used. In addition, there is a better UDP
support in Windows, compared to raw Ethernet MAC framing.


>The whole TCP
>retransmission and timers may be the mechanisms causing the problem.
>I think the standard TCP timeouts ar far too slow for this kind of
>thing? Think I saw a 250 ms timer tick in the LwIP docs?

You really have to do the transmission and possible retransmission in
your 40 ms time window.

>Missing packets may not be a problem. It is measurement data and if
>any re-transmit takes over 40 ms there is no point anyway, there is
>already new data. 

Just make sure you know if some frames are missing (e.g. serial
numbering).

>On a much slower rate, some control connection does
>have to be reliable. So that could use TCP (if is does not disturb
>the parallel UDP flow in LwIP) or use UDP as well and implement the
>retries in the application.

Why not, you can use both TCP and UDP as two separate links, even with
the same IP-address and port number.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.arch.embedded

csiph-web