Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #96915

Re: Lightwight socket IO wrapper

Show key headers only | View raw

On 21Sep2015 12:40, Chris Angelico <rosuav@gmail.com> wrote:
>On Mon, Sep 21, 2015 at 11:55 AM, Cameron Simpson <cs@zip.com.au> wrote:
>> On 21Sep2015 10:34, Chris Angelico <rosuav@gmail.com> wrote:
>>> If you're going to add sequencing and acknowledgements to UDP,
>>> wouldn't it be easier to use TCP and simply prefix every message with
>>> a two-byte length?
>>
>> Frankly, often yes. That's what I do. (different length encoding, but
>> otherwise...)
>
>Out of interest, what encoding?

NB: this is for binary protocols.

I don't like embedding arbitrary size limits in protocols or data formats if I 
can easily avoid it. So (for my home grown binary protocols) I encode unsigned 
integers as big endian octets with the top bit meaning "another octet follows" 
and the bottom 7 bits going to the value. So my packets look like:

  encoded(length)data

For sizes below 128, one byte of length. For sizes 128-16383, two bytes. And so 
on. Compact yet unbounded.

My new protocols ar probably going to derive from the scheme implemented  in 
the code cited below. "New" means as of some weeks ago, when I completely 
rewrote a painful ad hoc protocol of mine and pulled out the general features 
into what follows.

The actual packet format is implemented by the Packet class at the bottom of 
this:

  https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/serialise.py

Simple and flexible.

As for using that data format multiplexed with multiple channels, see the 
PacketConnection class here:

  https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/stream.py

Broadly, the packets are length[tag,flags[,channel#],payload] and one 
implements whatever semantics one needs on top of that.

You can see this exercised over UNIX pipes and TCP streams in the unit tests 
here:

  https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/stream_tests.py

On the subject of packet stuffing, my preferred loop for that is visible in the 
PacketConnection._send worker thread method, which goes:

    fp = self._send_fp
    Q = self._sendQ
    for P in Q:
      sig = (P.channel, P.tag, P.is_request)
      if sig in self.__sent:
        raise RuntimeError("second send of %s" % (P,))
      self.__sent.add(sig)
      write_Packet(fp, P)
      if Q.empty():
        fp.flush()
    fp.close()

In short: get packets from the queue and write them to the stream buffer. If 
the queue gets empty, _only then_ flush the buffer. This assures synchronicity 
in comms while giving the IO library a chance to fill a buffer with several 
packets.

Cheers,
Cameron Simpson <cs@zip.com.au>

ERROR 155 - You can't do that.  - Data General S200 Fortran error code list

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: Lightwight socket IO wrapper Cameron Simpson <cs@zip.com.au> - 2015-09-21 16:27 +1000

csiph-web