Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #61624 > unrolled thread

[newbie] trying socket as a replacement for nc

Back to article view | Back to comp.lang.python

Contents

  [newbie] trying socket as a replacement for nc Jean Dubois <jeandubois314@gmail.com> - 2013-12-11 15:08 -0800
    Re: [newbie] trying socket as a replacement for nc Dan Stromberg <drsalists@gmail.com> - 2013-12-11 15:20 -0800
      Re: [newbie] trying socket as a replacement for nc Jean Dubois <jeandubois314@gmail.com> - 2013-12-12 00:28 -0800
        Re: [newbie] trying socket as a replacement for nc Dan Stromberg <drsalists@gmail.com> - 2013-12-12 13:23 -0800
          Re: [newbie] trying socket as a replacement for nc Jean Dubois <jeandubois314@gmail.com> - 2013-12-12 19:23 -0800
            Re: [newbie] trying socket as a replacement for nc Dan Stromberg <drsalists@gmail.com> - 2013-12-12 19:32 -0800
              Re: [newbie] trying socket as a replacement for nc Jean Dubois <jeandubois314@gmail.com> - 2013-12-13 03:03 -0800
              Re: [newbie] trying socket as a replacement for nc Jean Dubois <jeandubois314@gmail.com> - 2013-12-13 03:56 -0800
            Re: [newbie] trying socket as a replacement for nc Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-12-13 08:35 +0000
              Re: [newbie] trying socket as a replacement for nc Jean Dubois <jeandubois314@gmail.com> - 2013-12-13 04:20 -0800
                Re: [newbie] trying socket as a replacement for nc rusi <rustompmody@gmail.com> - 2013-12-13 09:04 -0800
                Re: [newbie] trying socket as a replacement for nc rusi <rustompmody@gmail.com> - 2013-12-13 09:09 -0800
                  Re: [newbie] trying socket as a replacement for nc Jean Dubois <jeandubois314@gmail.com> - 2013-12-14 05:11 -0800
              Re: [newbie] trying socket as a replacement for nc Jean Dubois <jeandubois314@gmail.com> - 2013-12-14 05:14 -0800
                Re: [newbie] trying socket as a replacement for nc Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-12-14 13:29 +0000
          Re: [newbie] trying socket as a replacement for nc Jean Dubois <jeandubois314@gmail.com> - 2013-12-16 01:40 -0800
            Re: [newbie] trying socket as a replacement for nc Ervin Hegedüs <airween@gmail.com> - 2013-12-16 11:04 +0100
      Re: [newbie] trying socket as a replacement for nc Grant Edwards <invalid@invalid.invalid> - 2013-12-12 14:16 +0000
        Re: [newbie] trying socket as a replacement for nc Chris Angelico <rosuav@gmail.com> - 2013-12-13 01:23 +1100
        Re: [newbie] trying socket as a replacement for nc Dan Stromberg <drsalists@gmail.com> - 2013-12-12 13:27 -0800
          Re: [newbie] trying socket as a replacement for nc Grant Edwards <invalid@invalid.invalid> - 2013-12-13 16:06 +0000
            Re: [newbie] trying socket as a replacement for nc Dan Stromberg <drsalists@gmail.com> - 2013-12-14 17:24 -0800
              Re: [newbie] trying socket as a replacement for nc Grant Edwards <invalid@invalid.invalid> - 2013-12-15 15:15 +0000
                Re: [newbie] trying socket as a replacement for nc Roy Smith <roy@panix.com> - 2013-12-15 10:51 -0500
                  Re: [newbie] trying socket as a replacement for nc Chris Angelico <rosuav@gmail.com> - 2013-12-16 03:04 +1100
                    Re: [newbie] trying socket as a replacement for nc Roy Smith <roy@panix.com> - 2013-12-15 12:44 -0500
                  Re: [newbie] trying socket as a replacement for nc Grant Edwards <invalid@invalid.invalid> - 2013-12-15 22:42 +0000
                    Re: [newbie] trying socket as a replacement for nc Chris Angelico <rosuav@gmail.com> - 2013-12-16 09:48 +1100
        Re: [newbie] trying socket as a replacement for nc Chris Angelico <rosuav@gmail.com> - 2013-12-13 08:58 +1100
          Re: [newbie] trying socket as a replacement for nc Grant Edwards <invalid@invalid.invalid> - 2013-12-13 16:10 +0000
            Re: [newbie] trying socket as a replacement for nc Chris Angelico <rosuav@gmail.com> - 2013-12-14 03:19 +1100
              Re: [newbie] trying socket as a replacement for nc Grant Edwards <invalid@invalid.invalid> - 2013-12-13 16:57 +0000
                Re: [newbie] trying socket as a replacement for nc Chris Angelico <rosuav@gmail.com> - 2013-12-14 04:05 +1100
          Re: [newbie] trying socket as a replacement for nc 88888 Dihedral <dihedral88888@gmail.com> - 2013-12-16 04:38 -0800
            Re: [newbie] trying socket as a replacement for nc Chris Angelico <rosuav@gmail.com> - 2013-12-16 23:59 +1100
            Re: [newbie] trying socket as a replacement for nc Roy Smith <roy@panix.com> - 2013-12-16 09:03 -0500
              Re: [newbie] trying socket as a replacement for nc 88888 Dihedral <dihedral88888@gmail.com> - 2013-12-18 23:20 -0800
        Re: [newbie] trying socket as a replacement for nc Dave Angel <davea@davea.name> - 2013-12-12 19:52 -0500
    Re: [newbie] trying socket as a replacement for nc Conor Hughes <conorh@conorh.net> - 2013-12-11 15:38 -0800
      Re: [newbie] trying socket as a replacement for nc Jean Dubois <jeandubois314@gmail.com> - 2013-12-12 00:08 -0800
        Re: [newbie] trying socket as a replacement for nc Chris Angelico <rosuav@gmail.com> - 2013-12-12 19:21 +1100
          Re: [newbie] trying socket as a replacement for nc Jean Dubois <jeandubois314@gmail.com> - 2013-12-12 01:21 -0800
            Re: [newbie] trying socket as a replacement for nc Alister <alister.ware@ntlworld.com> - 2013-12-12 14:05 +0000
            Re: [newbie] trying socket as a replacement for nc Alister <alister.ware@ntlworld.com> - 2013-12-12 14:05 +0000
              Re: [newbie] trying socket as a replacement for nc Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-12-12 14:34 +0000
    Re: [newbie] trying socket as a replacement for nc Christian Gollwitzer <auriocus@gmx.de> - 2013-12-13 00:43 +0100

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#61827

On 2013-12-12, Dan Stromberg <drsalists@gmail.com> wrote:
> On Thu, Dec 12, 2013 at 6:16 AM, Grant Edwards <invalid@invalid.invalid> wrote:
>
>>> Sockets reserve the right to split one socket.send() into multiple
>>> socket.recv()'s on the other end of the communication, or to aggregate
>>> multiple socket.send()'s into a single socket.recv() - pretty much any way
>>> the relevant IP stacks and communications equipment feel like for the sake
>>> of performance or reliability.
>>
>> Just to be pedantic: _TCP_ sockets reserver that right.  UDP sockets
>> do not, and do in fact guarantee that each message is discrete.  [It
>> appears that the OP is undoubtedly using TCP sockets.]
>
> I haven't done a lot of UDP, but are you pretty sure UDP can't at
> least fragment large packets?  What's a router or switch to do if the
> Path MTU isn't large enough for an original packet?
>
> http://www.gamedev.net/topic/343577-fragmented-udp-packets/

You're conflating IP datagrams and Ethernet packets.  The IP stack can
fragment an IP datagram into multiple Ethernet packets which are then
reassembled by the receiving IP stack into a single datagram before
being passed up to the next layer (in this case, UDP).

Did you read the thread you pointed to?  Your question was answerd by
posting #4 in the thread you cited:

   1) Yes, packets will be fragmented at the network layer (IP), but this
      is something you do not have to worry about since the network
      layer will reassemble the fragments before passing them back up
      to the transport layer (UDP). UDP garentees preserved message
      boundaries, so you never have to worry about only receiving a
      packet fragment :~).


A few other references:

http://tools.ietf.org/html/rfc791

 1.1. Motivation

  [...] The internet protocol provides for transmitting blocks of data
  called datagrams from sources to destinations, [...] The internet
  protocol also provides for fragmentation and reassembly of long
  datagrams, if necessary, for transmission through "small packet"
  networks.
  
  [...]  

 1.4 Operation

  [...]
 
  The internet modules use fields in the internet header to fragment
  and reassemble internet datagrams when necessary for transmission
  through "small packet" networks.
  
  [...]

From http://en.wikipedia.org/wiki/IP_fragmentation  

  If a receiving host receives a fragmented IP packet, it has to
  reassemble the datagram and pass it to the higher protocol layer.
  Reassembly is intended to happen in the receiving host but in
  practice it may be done by an intermediate router, for example,
  network address translation may need to re-assemble fragments in
  order to translate data streams, e.g. the FTP control protocol, as
  described in RFC 2993
  
-- 
Grant Edwards               grant.b.edwards        Yow! I'm continually AMAZED
                                  at               at th'breathtaking effects
                              gmail.com            of WIND EROSION!!

[toc] | [prev] | [next] | [standalone]

#61925

On Fri, Dec 13, 2013 at 8:06 AM, Grant Edwards <invalid@invalid.invalid> wrote:
> On 2013-12-12, Dan Stromberg <drsalists@gmail.com> wrote:
>> On Thu, Dec 12, 2013 at 6:16 AM, Grant Edwards <invalid@invalid.invalid> wrote:
>>
>>>> Sockets reserve the right to split one socket.send() into multiple
>>>> socket.recv()'s on the other end of the communication, or to aggregate
>>>> multiple socket.send()'s into a single socket.recv() - pretty much any way
>>>> the relevant IP stacks and communications equipment feel like for the sake
>>>> of performance or reliability.
>>>
>>> Just to be pedantic: _TCP_ sockets reserver that right.  UDP sockets
>>> do not, and do in fact guarantee that each message is discrete.  [It
>>> appears that the OP is undoubtedly using TCP sockets.]
>>
>> I haven't done a lot of UDP, but are you pretty sure UDP can't at
>> least fragment large packets?  What's a router or switch to do if the
>> Path MTU isn't large enough for an original packet?
>>
>> http://www.gamedev.net/topic/343577-fragmented-udp-packets/
>
> You're conflating IP datagrams and Ethernet packets.  The IP stack can
> fragment an IP datagram into multiple Ethernet packets which are then
> reassembled by the receiving IP stack into a single datagram before
> being passed up to the next layer (in this case, UDP).

As long as you're saying this of UDP, I have no problem with it.

I've seen TCP fragment and not be reassembled though, which suggests
to me that the reassembly's happening in UDP rather than IP.  If it's
done by IP the same way for UDP and TCP, I'd not trust it in UDP
either.

> Did you read the thread you pointed to?  Your question was answerd by
> posting #4 in the thread you cited:
>
>    1) Yes, packets will be fragmented at the network layer (IP), but this
>       is something you do not have to worry about since the network
>       layer will reassemble the fragments before passing them back up
>       to the transport layer (UDP). UDP garentees preserved message
>       boundaries, so you never have to worry about only receiving a
>       packet fragment :~).

Actually, I believe the link I sent (which I skimmed) had people
coming down on both sides of the matter.  Some said that UDP would be
fine for small datagrams, while others said it would be fine,
irrespective of size.

>
> A few other references:
>
> http://tools.ietf.org/html/rfc791
>
>  1.1. Motivation
>
>   [...] The internet protocol provides for transmitting blocks of data
>   called datagrams from sources to destinations, [...] The internet
>   protocol also provides for fragmentation and reassembly of long
>   datagrams, if necessary, for transmission through "small packet"
>   networks.

I've personally seen this fail to occur in TCP - EG, it can cause a
stream of bytes to be written to tape with inconsistent block sizes if
transferred over rsh or ssh.  Usually the block sizes are consistent,
but not always.  Both SunOS 4.1.x and Ultrix had this issue; Ultrix
did it less often than SunOS, but Ultrix did do it.  I don't know for
certain if later *ix have the same issue, because I've been diligently
working around it ever since, but I suspect they do.

I've seen old time socket programmers explain that it cannot be relied
upon in TCP; send() and recv() and (read() and write()) are system
calls that return a length so that you can loop on them until all
relevant data has been transferred.  They don't return that length
just so you can ignore it.

>From the Socket HOWTO
(http://docs.python.org/2/howto/sockets.html#socket-howto) :
Now we come to the major stumbling block of sockets - send and recv
operate on the network buffers. They do not necessarily handle all the
bytes you hand them (or expect from them), because their major focus
is handling the network buffers. In general, they return when the
associated network buffers have been filled (send) or emptied (recv).
They then tell you how many bytes they handled. It is your
responsibility to call them again until your message has been
completely dealt with.

HTH

[toc] | [prev] | [next] | [standalone]

#61946

On 2013-12-15, Dan Stromberg <drsalists@gmail.com> wrote:
> On Fri, Dec 13, 2013 at 8:06 AM, Grant Edwards <invalid@invalid.invalid> wrote:
>> On 2013-12-12, Dan Stromberg <drsalists@gmail.com> wrote:

>>>> Just to be pedantic: _TCP_ sockets reserve that right.  UDP sockets
>>>> do not, and do in fact guarantee that each message is discrete.  [It
>>>> appears that the OP is undoubtedly using TCP sockets.]
>>>
>>> I haven't done a lot of UDP, but are you pretty sure UDP can't at
>>> least fragment large packets?  What's a router or switch to do if the
>>> Path MTU isn't large enough for an original packet?
>>>
>>> http://www.gamedev.net/topic/343577-fragmented-udp-packets/
>>
>> You're conflating IP datagrams and Ethernet packets.  The IP stack can
>> fragment an IP datagram into multiple Ethernet packets which are then
>> reassembled by the receiving IP stack into a single datagram before
>> being passed up to the next layer (in this case, UDP).
>
> As long as you're saying this of UDP, I have no problem with it.

That is indeed what I'm saying.  I apoligize if that was not clear in
my original posting.

> I've seen TCP fragment and not be reassembled though, which suggests
> to me that the reassembly's happening in UDP rather than IP.

That's something different.  In TCP, there's no guarantee that
reads/writes correspond 1:1 to IP datagrams.  TCP is a _stream_
protocol and there is no semantic meaning attached to the boundaries
between successive read/write calls the way there is with UDP.

> If it's done by IP the same way for UDP and TCP,

The IP layer is supposed to reassemble receive datagrams for both --
but that's got nothing to do with atomicity of TCP writes/reads.  The
TCP stack can (and often does) turn one write() call into multiple IP
datagrams.  It can also turn multiple writes into a singel IP
datagram.  On the other end, it can split up a single datagram into
multiple read()s and/or combined multiple datagrams into a single
read().  TCP is a stream service, not a datagram service like UDP.

> I'd not trust it in UDP either.

The standards all require UDP datagrams to be preserved.  All of the
UDP applications I've ever written or seen depend utterly on that, and
it's always worked that way for me.  If you've seen it fail, then you
ought to file a bug report.

>> Did you read the thread you pointed to?  Your question was answerd by
>> posting #4 in the thread you cited:
>>
>>    1) Yes, packets will be fragmented at the network layer (IP), but
>>       this is something you do not have to worry about since the
>>       network layer will reassemble the fragments before passing them
>>       back up to the transport layer (UDP). UDP garentees preserved
>>       message boundaries, so you never have to worry about only
>>       receiving a packet fragment :~).
>
> Actually, I believe the link I sent (which I skimmed) had people
> coming down on both sides of the matter.  Some said that UDP would be
> fine for small datagrams, while others said it would be fine,
> irrespective of size.

The maximum size of an IP datagram is 64KB, so it's not "fine
irrespecive of size".  If your UDP implementation is working correctly
it will be fine below that limit.

>> A few other references:
>>
>> http://tools.ietf.org/html/rfc791
>>
>>  1.1. Motivation
>>
>>   [...] The internet protocol provides for transmitting blocks of data
>>   called datagrams from sources to destinations, [...] The internet
>>   protocol also provides for fragmentation and reassembly of long
>>   datagrams, if necessary, for transmission through "small packet"
>>   networks.
>
> I've personally seen this fail to occur in TCP

You can't say that, because there's no correspondance between IP
datgrams and TCP read/write block sizes the way there is in UDP. 

With TCP there is nothing to fail (with respect to read/write block
sizes). TCP only guarantees that bytes will get there and get there in
the right order. It doesn't make any promises about block sizes.

> I've seen old time socket programmers explain that it cannot be relied
> upon in TCP; send() and recv() and (read() and write()) are system
> calls that return a length so that you can loop on them until all
> relevant data has been transferred.  They don't return that length
> just so you can ignore it.

That's true, but that's because of the design of the TCP _stream_
protocol, not because the IP datagram layer doesn't work right.
>
>>From the Socket HOWTO
>
> (http://docs.python.org/2/howto/sockets.html#socket-howto) : Now we
> come to the major stumbling block of sockets - send and recv operate
> on the network buffers. They do not necessarily handle all the bytes
> you hand them (or expect from them), because their major focus is
> handling the network buffers. In general, they return when the
> associated network buffers have been filled (send) or emptied (recv).
> They then tell you how many bytes they handled. It is your
> responsibility to call them again until your message has been
> completely dealt with.

If that's true for UDP, then the Python UDP implementation is broken,
and somebody should file a bug.  UDP is a a _datagram_ service.
Either all the bytes in a write() should get sent or none of them.
Sending a paritial datagram is _not_ a valid option.

-- 
Grant

[toc] | [prev] | [next] | [standalone]

#61951

In article <l8kh1r$bj8$1@reader1.panix.com>,
 Grant Edwards <invalid@invalid.invalid> wrote:

> UDP is a a _datagram_ service. Either all the bytes in a write() 
> should get sent or none of them. Sending a paritial datagram is _not_ 
> a valid option.

I would agree with the above if you said send() instead of write().  
Python socket objects don't have write() methods, file objects do.  You 
can wrap a file around a socket with socket.makefile(), but I'm not sure 
I would expect the UDP record boundary semantics to be honored once you 
did that.

[toc] | [prev] | [next] | [standalone]

#61952

On Mon, Dec 16, 2013 at 2:51 AM, Roy Smith <roy@panix.com> wrote:
> In article <l8kh1r$bj8$1@reader1.panix.com>,
>  Grant Edwards <invalid@invalid.invalid> wrote:
>
>> UDP is a a _datagram_ service. Either all the bytes in a write()
>> should get sent or none of them. Sending a paritial datagram is _not_
>> a valid option.
>
> I would agree with the above if you said send() instead of write().
> Python socket objects don't have write() methods, file objects do.  You
> can wrap a file around a socket with socket.makefile(), but I'm not sure
> I would expect the UDP record boundary semantics to be honored once you
> did that.

The underlying C API allows you, on Unix-like systems at least, to use
the standard write() function to send UDP packets (as long as you
first connect() - otherwise you need sendto() to specify a
destination). I don't usually use that method, but I would expect that
one call to write() becomes one UDP packet.

How that works with socket.makefile() in Python I have no idea.

ChrisA

[toc] | [prev] | [next] | [standalone]

#61959

In article <mailman.4143.1387123508.18130.python-list@python.org>,
 Chris Angelico <rosuav@gmail.com> wrote:

> On Mon, Dec 16, 2013 at 2:51 AM, Roy Smith <roy@panix.com> wrote:
> > In article <l8kh1r$bj8$1@reader1.panix.com>,
> >  Grant Edwards <invalid@invalid.invalid> wrote:
> >
> >> UDP is a a _datagram_ service. Either all the bytes in a write()
> >> should get sent or none of them. Sending a paritial datagram is _not_
> >> a valid option.
> >
> > I would agree with the above if you said send() instead of write().
> > Python socket objects don't have write() methods, file objects do.  You
> > can wrap a file around a socket with socket.makefile(), but I'm not sure
> > I would expect the UDP record boundary semantics to be honored once you
> > did that.
> 
> The underlying C API allows you, on Unix-like systems at least, to use
> the standard write() function to send UDP packets (as long as you
> first connect() - otherwise you need sendto() to specify a
> destination). I don't usually use that method, but I would expect that
> one call to write() becomes one UDP packet.

At the Unix system call level, yes.  But, given that this is a Python 
newsgroup, I made the assumption we were talking about the Python API 
level.  Silly me :-)

[toc] | [prev] | [next] | [standalone]

#61969

On 2013-12-15, Roy Smith <roy@panix.com> wrote:
> In article <l8kh1r$bj8$1@reader1.panix.com>,
>  Grant Edwards <invalid@invalid.invalid> wrote:
>
>> UDP is a a _datagram_ service. Either all the bytes in a write() 
>> should get sent or none of them. Sending a paritial datagram is _not_ 
>> a valid option.
>
> I would agree with the above if you said send() instead of write().

Good point -- I meant send().  I keep forgetting that the libc socket
write() operation is missing in Python and only the send() call has
been made visible. In C write() and send() are effectively the same
thing (the parameters are arranged a little differently, but they
behave identically otherwise).

> Python socket objects don't have write() methods, file objects do.  You 
> can wrap a file around a socket with socket.makefile(), but I'm not sure 
> I would expect the UDP record boundary semantics to be honored once you 
> did that.

No, I wouldn't exect that.

-- 
Grant Edwards               grant.b.edwards        Yow! I wish I was on a
                                  at               Cincinnati street corner
                              gmail.com            holding a clean dog!

[toc] | [prev] | [next] | [standalone]

#61971

On Mon, Dec 16, 2013 at 9:42 AM, Grant Edwards <invalid@invalid.invalid> wrote:
> Good point -- I meant send().  I keep forgetting that the libc socket
> write() operation is missing in Python and only the send() call has
> been made visible. In C write() and send() are effectively the same
> thing (the parameters are arranged a little differently, but they
> behave identically otherwise).

Mostly - send() lets you set options, write() is equivalent to send()
with no options set. But other than that, they're identical, as stated
in man 2 send.

ChrisA

[toc] | [prev] | [next] | [standalone]

#61762

On Fri, Dec 13, 2013 at 8:27 AM, Dan Stromberg <drsalists@gmail.com> wrote:
> On Thu, Dec 12, 2013 at 6:16 AM, Grant Edwards <invalid@invalid.invalid> wrote:
>
>>> Sockets reserve the right to split one socket.send() into multiple
>>> socket.recv()'s on the other end of the communication, or to aggregate
>>> multiple socket.send()'s into a single socket.recv() - pretty much any way
>>> the relevant IP stacks and communications equipment feel like for the sake
>>> of performance or reliability.
>>
>> Just to be pedantic: _TCP_ sockets reserver that right.  UDP sockets
>> do not, and do in fact guarantee that each message is discrete.  [It
>> appears that the OP is undoubtedly using TCP sockets.]
>
> I haven't done a lot of UDP, but are you pretty sure UDP can't at
> least fragment large packets?  What's a router or switch to do if the
> Path MTU isn't large enough for an original packet?
>
> http://www.gamedev.net/topic/343577-fragmented-udp-packets/

I'm no expert on this (mostly I do TCP, or UDP with fairly small
packets), but the packet should be reassembled at the far end. When
your application comes to receive it, it'll receive the entire UDP
packet as a whole.

UDP fragmentation has several problems. First, if any fragment is
lost, it won't be retransmitted (as TCP will), so the whole datagram
is lost. And secondly, if you stream data across the network in a
series of packets just a little too large to fit, each one will get
split in two and you'll end up with twice as many packets going out,
ergo abysmal performance. With TCP, there's the chance that the sender
and receiver can between them figure out what packet size to use (cf
path MTU discovery), but that won't happen with UDP unless the
application consciously does it. So it's something to be cautious of
in terms of performance, but if you want to send large UDP packets
because they make sense, just go ahead and do it.

Now, if you want reliability AND datagrams, it's a lot easier to add
boundaries to a TCP stream (sentinel or length prefixes) than to add
reliability to UDP...

ChrisA

[toc] | [prev] | [next] | [standalone]

#61828

On 2013-12-12, Chris Angelico <rosuav@gmail.com> wrote:

> Now, if you want reliability AND datagrams, it's a lot easier to add
> boundaries to a TCP stream (sentinel or length prefixes) than to add
> reliability to UDP...

It's unfortunate that there's no standardized reliable
connection-oriented datagram protocol.  The linux kernel implements 
one for Unix domain sockets (SOCK_SEQPACKET), and its really, really
useful.

Adding boundaries to a TCP stream achieves the same goal (and isn't
that hard to do), but since there's no standard for it, people keep
having to reinvent it (often badly and always incompaibly).

-- 
Grant Edwards               grant.b.edwards        Yow! Hello...  IRON
                                  at               CURTAIN?  Send over a
                              gmail.com            SAUSAGE PIZZA!  World War
                                                   III?  No thanks!

[toc] | [prev] | [next] | [standalone]

#61831

On Sat, Dec 14, 2013 at 3:10 AM, Grant Edwards <invalid@invalid.invalid> wrote:
> Adding boundaries to a TCP stream achieves the same goal (and isn't
> that hard to do), but since there's no standard for it, people keep
> having to reinvent it (often badly and always incompaibly).

Nearest to a standard would be the way heaps of internet protocols are
line-based - SMTP, POP, IMAP, FTP, and to a lesser extent HTTP as
well. The end-of-line sequence \r\n delimits messages.

ChrisA

[toc] | [prev] | [next] | [standalone]

#61844

On 2013-12-13, Chris Angelico <rosuav@gmail.com> wrote:
> On Sat, Dec 14, 2013 at 3:10 AM, Grant Edwards <invalid@invalid.invalid> wrote:
>> Adding boundaries to a TCP stream achieves the same goal (and isn't
>> that hard to do), but since there's no standard for it, people keep
>> having to reinvent it (often badly and always incompaibly).
>
> Nearest to a standard would be the way heaps of internet protocols are
> line-based - SMTP, POP, IMAP, FTP, and to a lesser extent HTTP as
> well. The end-of-line sequence \r\n delimits messages.

And that works very nicely for things that transport text.  It's easy
to implement, easy to debug, easy to test.  But, when you need to
transport binary data, it gets ugly and compatibility problems start
to arise pretty quickly.

One could also borrow standards from the old-school serial world and
us the SYN/STX/ETX framing with byte stuffing used by HDLC et al.

-- 
Grant Edwards               grant.b.edwards        Yow! My vaseline is
                                  at               RUNNING...
                              gmail.com            

[toc] | [prev] | [next] | [standalone]

#61847

On Sat, Dec 14, 2013 at 3:57 AM, Grant Edwards <invalid@invalid.invalid> wrote:
> On 2013-12-13, Chris Angelico <rosuav@gmail.com> wrote:
>> On Sat, Dec 14, 2013 at 3:10 AM, Grant Edwards <invalid@invalid.invalid> wrote:
>>> Adding boundaries to a TCP stream achieves the same goal (and isn't
>>> that hard to do), but since there's no standard for it, people keep
>>> having to reinvent it (often badly and always incompaibly).
>>
>> Nearest to a standard would be the way heaps of internet protocols are
>> line-based - SMTP, POP, IMAP, FTP, and to a lesser extent HTTP as
>> well. The end-of-line sequence \r\n delimits messages.
>
> And that works very nicely for things that transport text.  It's easy
> to implement, easy to debug, easy to test.  But, when you need to
> transport binary data, it gets ugly and compatibility problems start
> to arise pretty quickly.
>
> One could also borrow standards from the old-school serial world and
> us the SYN/STX/ETX framing with byte stuffing used by HDLC et al.

Yeah, or if it's a tight binary protocol with occasional bits of
bigger payload, either MIDI or TELNET could offer ideas. MIDI's SysEx
message can carry whatever is needed of it, and TELNET has
subnegotiation that can be used for the odd thingy or so. But for a
generic binary stream-of-messages pipe (as opposed to the
stream-of-bytes pipe that TCP normally offers), probably the easiest
is to precede each write with an N-byte length... and somehow
negotiate what N should be. (And endianness, but hopefully that's just
network byte order.)

Standards are awesome, there are so many to choose from! http://xkcd.com/927/

ChrisA

[toc] | [prev] | [next] | [standalone]

#62054

On Friday, December 13, 2013 5:58:49 AM UTC+8, Chris Angelico wrote:
> On Fri, Dec 13, 2013 at 8:27 AM, Dan Stromberg <drsalists@gmail.com> wrote:
> 
> > On Thu, Dec 12, 2013 at 6:16 AM, Grant Edwards <invalid@invalid.invalid> wrote:
> 
> >
> 
> >>> Sockets reserve the right to split one socket.send() into multiple
> 
> >>> socket.recv()'s on the other end of the communication, or to aggregate
> 
> >>> multiple socket.send()'s into a single socket.recv() - pretty much any way
> 
> >>> the relevant IP stacks and communications equipment feel like for the sake
> 
> >>> of performance or reliability.
> 
> >>
> 
> >> Just to be pedantic: _TCP_ sockets reserver that right.  UDP sockets
> 
> >> do not, and do in fact guarantee that each message is discrete.  [It
> 
> >> appears that the OP is undoubtedly using TCP sockets.]
> 
> >
> 
> > I haven't done a lot of UDP, but are you pretty sure UDP can't at
> 
> > least fragment large packets?  What's a router or switch to do if the
> 
> > Path MTU isn't large enough for an original packet?
> 
> >
> 
> > http://www.gamedev.net/topic/343577-fragmented-udp-packets/
> 
> 
> 
> I'm no expert on this (mostly I do TCP, or UDP with fairly small
> 
> packets), but the packet should be reassembled at the far end. When
> 
> your application comes to receive it, it'll receive the entire UDP
> 
> packet as a whole.
> 
> 
> 
> UDP fragmentation has several problems. First, if any fragment is
> 
> lost, it won't be retransmitted (as TCP will), so the whole datagram
> 
> is lost. And secondly, if you stream data across the network in a
> 
> series of packets just a little too large to fit, each one will get
> 
> split in two and you'll end up with twice as many packets going out,
> 
> ergo abysmal performance. With TCP, there's the chance that the sender
> 
> and receiver can between them figure out what packet size to use (cf
> 
> path MTU discovery), but that won't happen with UDP unless the
> 
> application consciously does it. So it's something to be cautious of
> 
> in terms of performance, but if you want to send large UDP packets
> 
> because they make sense, just go ahead and do it.
> 
> 
> 
> Now, if you want reliability AND datagrams, it's a lot easier to add
> 
> boundaries to a TCP stream (sentinel or length prefixes) than to add
> 
> reliability to UDP...
> 
> 
> 
> ChrisA
It is trivial to use UDP with 
forward error correction such as 
the CD in 1982.

[toc] | [prev] | [next] | [standalone]

#62057

On Mon, Dec 16, 2013 at 11:38 PM, 88888 Dihedral
<dihedral88888@gmail.com> wrote:
> It is trivial to use UDP with
> forward error correction such as
> the CD in 1982.

This is another reason for moving to IPv6. With IPv4, the size of a
datagram is limited to 64KB, but with IPv6, you could carry an entire
CD's contents in a single message. You could not carry a DVD, though,
so Dihedral is quite correct to cite the CD here.

ChrisA

[toc] | [prev] | [next] | [standalone]

#62063

On Friday, December 13, 2013 5:58:49 AM UTC+8, Chris Angelico wrote:
> > Now, if you want reliability AND datagrams, it's a lot easier to add
> > boundaries to a TCP stream (sentinel or length prefixes) than to add
> > reliability to UDP...

In article <11cb8cd3-7a12-46b2-abc6-53fbc2a54525@googlegroups.com>,
 88888 Dihedral <dihedral88888@gmail.com> wrote:

> It is trivial to use UDP with 
> forward error correction such as 
> the CD in 1982.

CD uses Reed-Solomon coding, which is great for correcting the types of 
errors expected on a CD.  Namely, bursts of bit errors caused by 
localized failure of the optical coating, scratches, dirt, etc.  It 
wouldn't be hard to build something like that on top of UDP, but those 
sorts of errors are not what you typically see in networks.

It's relatively rare for a bit to get corrupted in a network packet.  
And, when it does, it's almost certainly caught by lower-level 
mechanisms such as ethernet frame CRC.  Much more likely is for a packet 
to get dropped because of queue overflow, or for sequential packets to 
arrive out of order due to multiple transmission paths with different 
latencies.  Those are the sorts of things TCP protects against.

Sure, you could implement retransmit timers and packet reordering in 
user code, but it would be distinctly non-trivial and ultimately you 
would end up reinventing most of TCP.  Except that your implementation 
would suck compared to the kernel algorithms which have been 
continuously tested and fine-tuned for the past 30 years.

[toc] | [prev] | [next] | [standalone]

#62380

>  88888 Dihedral <dihedral88888@gmail.com> wrote:
> 
> 
> 
> > It is trivial to use UDP with 
> 
> > forward error correction such as 
> 
> > the CD in 1982.
> 
> 
> 
> CD uses Reed-Solomon coding, which is great for correcting the types of 
> 
> errors expected on a CD.  Namely, bursts of bit errors caused by 
> 
> localized failure of the optical coating, scratches, dirt, etc.  It 
> 

Don't you interleave your bytes of data
first before forming several UDP 
packets to send to the receiver?

If a whole packet is lost or 
timed out, then just mark missed bytes
in the missed UDP as erasures.

> wouldn't be hard to build something like that on top of UDP, but those 
> 
> sorts of errors are not what you typically see in networks.
> 
> 
> 
> It's relatively rare for a bit to get corrupted in a network packet.  
> 
> And, when it does, it's almost certainly caught by lower-level 
> 
> mechanisms such as ethernet frame CRC.  Much more likely is for a packet 
> 
> to get dropped because of queue overflow, or for sequential packets to 
> 
> arrive out of order due to multiple transmission paths with different 
> 
> latencies.  Those are the sorts of things TCP protects against.
> 
> 
> 
> Sure, you could implement retransmit timers and packet reordering in 
> 
> user code, but it would be distinctly non-trivial and ultimately you 
> 
> would end up reinventing most of TCP.  Except that your implementation 
> 
> would suck compared to the kernel algorithms which have been 
> 
> continuously tested and fine-tuned for the past 30 years.

[toc] | [prev] | [next] | [standalone]

#61778

On Thu, 12 Dec 2013 13:27:16 -0800, Dan Stromberg 
<drsalists@gmail.com> wrote:
> On Thu, Dec 12, 2013 at 6:16 AM, Grant Edwards 
<invalid@invalid.invalid> wrote:
> I haven't done a lot of UDP, but are you pretty sure UDP can't at
> least fragment large packets?  What's a router or switch to do if 
the
> Path MTU isn't large enough for an original packet?

A router doesn't talk udp or tcp, it deals with ip packets, which it 
IS permitted to fragment. 

Conversely the udp layer in the os doesn't care about MTU.

-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#61628

Jean Dubois <jeandubois314@gmail.com> writes:

> I have an ethernet-rs232 adapter which allows me to connect to a
> measurement instrument by means of netcat on a linux system.
> e.g. entering nc 10.128.59.63 7000
> allows me to enter e.g.
> *IDN?
> after which I get an identification string of the measurement
> instrument back.
> I thought I could accomplish the same using the python module "socket"
> and tried out the sample program below which doesn't work however:

In what way does it not work? Do you not get any data? Do you get the
wrong data? Does your program block at a point which you do not
understand?

Probably more to the point, are you sure you are sending exactly the
same data as you did with netcat? netcat running with stdin a terminal
sends data line-by-line, and includes the newline in the data that it
sends. You didn't send a newline in your example.

[toc] | [prev] | [next] | [standalone]

#61677

On Thursday, December 12, 2013 12:38:12 AM UTC+1, Conor Hughes wrote:
> Jean Dubois <jeandubois314@gmail.com> writes:
> 
> 
> 
> > I have an ethernet-rs232 adapter which allows me to connect to a
> 
> > measurement instrument by means of netcat on a linux system.
> 
> > e.g. entering nc 10.128.59.63 7000
> 
> > allows me to enter e.g.
> 
> > *IDN?
> 
> > after which I get an identification string of the measurement
> 
> > instrument back.
> 
> > I thought I could accomplish the same using the python module "socket"
> 
> > and tried out the sample program below which doesn't work however:
> 
> 
> 
> In what way does it not work? Do you not get any data? Do you get the
> 
> wrong data? Does your program block at a point which you do not
> 
> understand?
> 
> 
> 
> Probably more to the point, are you sure you are sending exactly the
> 
> same data as you did with netcat? netcat running with stdin a terminal
> 
> sends data line-by-line, and includes the newline in the data that it
> 
> sends. You didn't send a newline in your example.
Thanks for the reply, I changed the line you mentioned to
s.send('*IDN?\n')
Rerunning the program then shows me as response the first time
Received:
The measurement instrument gives a beep indicating it receives something
which is however not recognized als normal input

Running the script a  second time, the program hangs.. and I have to reboot the adapter

kind regards,
jean

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | comp.lang.python

csiph-web