Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #80875 > unrolled thread

OTish: using short-term TCP connections to send to multiple slaves

Back to article view | Back to comp.lang.python

Contents

  OTish: using short-term TCP connections to send to multiple slaves jkn <jkn_gg@nicorp.f9.co.uk> - 2014-11-16 12:02 +0000
    Re: OTish: using short-term TCP connections to send to multiple slaves Chris Angelico <rosuav@gmail.com> - 2014-11-16 23:36 +1100
    Re: OTish: using short-term TCP connections to send to multiple slaves Marko Rauhamaa <marko@pacujo.net> - 2014-11-16 14:44 +0200
    Re: OTish: using short-term TCP connections to send to multiple slaves Alain Ketterlin <alain@dpt-info.u-strasbg.fr> - 2014-11-16 14:43 +0100
      Re: OTish: using short-term TCP connections to send to multiple slaves Chris Angelico <rosuav@gmail.com> - 2014-11-17 02:13 +1100
        Re: OTish: using short-term TCP connections to send to multiple slaves Grant Edwards <invalid@invalid.invalid> - 2014-11-16 16:53 +0000
        Re: OTish: using short-term TCP connections to send to multiple slaves Roy Smith <roy@panix.com> - 2014-11-16 12:21 -0500
          Re: OTish: using short-term TCP connections to send to multiple slaves Grant Edwards <invalid@invalid.invalid> - 2014-11-16 17:35 +0000
          Re: OTish: using short-term TCP connections to send to multiple slaves Chris Angelico <rosuav@gmail.com> - 2014-11-17 08:56 +1100
            Re: OTish: using short-term TCP connections to send to multiple slaves alister <alister.nospam.ware@ntlworld.com> - 2014-11-17 10:46 +0000
        Re: OTish: using short-term TCP connections to send to multiple slaves jkn <jkn_gg@nicorp.f9.co.uk> - 2014-11-16 21:45 +0000
          Re: OTish: using short-term TCP connections to send to multiple slaves Grant Edwards <invalid@invalid.invalid> - 2014-11-17 16:56 +0000
    Re: OTish: using short-term TCP connections to send to multiple slaves Grant Edwards <invalid@invalid.invalid> - 2014-11-16 16:42 +0000
    Re: OTish: using short-term TCP connections to send to multiple slaves Peter Otten <__peter__@web.de> - 2014-11-17 09:16 +0100

#80875 — OTish: using short-term TCP connections to send to multiple slaves

Hi all
    This is a little bit OT for this newsgroup, but I intend to use python 
for prototyping at least, and I know there are a lot of knowledgeable 
people using Python in a Network context here...

I have a use case of a single 'master' machine which will need to 
periodically 'push' data to a variety of 'slave' devices on a small local 
subnet, over Ethernet. We are talking perhaps a dozen devices in all with 
comms occurring perhaps once very few seconds, to much less often - once per 
half an hour, or less. There is probably an upper bound of 64KB or so of 
data that is likely to be sent on each occasion.

Previous similar systems have attempted to do this by maintaining multiple 
long-term TCP connections from the master to all the slave devices. The 
Master is the server and the slaves periodically check the master to see 
what has changed. Although this ... works ..., we have had trouble 
maintaining the connection, for reasons ... I am not yet fully aware of.

We are now considering an alternative approach where the master maintains a 
list of slave devices and acts as a client. Each device is a server from the 
point of view of data transfer. We would then use a short-lived TCP 
connection when data is available; the Master would connect to each slave 
which needed the data, send it, and close the connection.

I should also add that we desire our system to be 'robust' in the face of 
situations such as cable unplugging, device power cycles, etc.

Although this might get round some of our current problems, I can see that 
we might end up with new problems to deal with. I am wondering if this 
scenario rings bells with anyone, and seeking pointers to what has been done 
elsewhere. As I say, I am expecting to prototype it in Python so any 
specifics also welcome!

(suggestions as to a better forum to ask for previous experience also 
gratefully received)

Thanks al lot for any thoughts/suggestions

    Jon N

[toc] | [next] | [standalone]

#80876

On Sun, Nov 16, 2014 at 11:02 PM, jkn <jkn_gg@nicorp.f9.co.uk> wrote:
> I have a use case of a single 'master' machine which will need to
> periodically 'push' data to a variety of 'slave' devices on a small local
> subnet, over Ethernet. We are talking perhaps a dozen devices in all with
> comms occurring perhaps once very few seconds, to much less often - once per
> half an hour, or less. There is probably an upper bound of 64KB or so of
> data that is likely to be sent on each occasion.

This is very similar to what a MUD server has to do, with the added
advantage that you're working with ethernet (most MUDs are accessed
through the dangerous waters of the internet, and there are all sorts
of extra problems).

> Previous similar systems have attempted to do this by maintaining multiple
> long-term TCP connections from the master to all the slave devices. The
> Master is the server and the slaves periodically check the master to see
> what has changed. Although this ... works ..., we have had trouble
> maintaining the connection, for reasons ... I am not yet fully aware of.

The most likely case on the internet is that idle connections are
getting dropped, maybe after five minutes. On a LAN, that shouldn't be
happening, but can you run a test and see if maintaining the
connection (by sending dummy packets every minute, perhaps) stops the
dropouts? That might be all you need.

> We are now considering an alternative approach where the master maintains a
> list of slave devices and acts as a client. Each device is a server from the
> point of view of data transfer. We would then use a short-lived TCP
> connection when data is available; the Master would connect to each slave
> which needed the data, send it, and close the connection.

That's also possible. You may find you have a lot of overhead,
percentage-wise, but based on your figures above (a dozen devices, no
more frequent than every few seconds), I very much doubt it'll be a
problem.

> I should also add that we desire our system to be 'robust' in the face of
> situations such as cable unplugging, device power cycles, etc.

Then you want asynchronous connections. If the device at the other end
is down, don't wait for it.

> Although this might get round some of our current problems, I can see that
> we might end up with new problems to deal with. I am wondering if this
> scenario rings bells with anyone, and seeking pointers to what has been done
> elsewhere. As I say, I am expecting to prototype it in Python so any
> specifics also welcome!

Not a problem. I usually do this kind of work in Pike, rather than
Python, so I can't offer ready-to-go Python code for you, but it's
pretty similar. Is there any reason you plan just to _prototype_ this
in Python? Seems to me you could do the whole job in it, no reason to
shift to another language - unless there's a huge aspect to this
project that's computationally intensive or something. The part you've
described is going to be dominated by the network, not the programming
language.

> (suggestions as to a better forum to ask for previous experience also
> gratefully received)

I don't know of any place better. Let's keep it here, at least for the
moment (unless you switch to using Pike, in which case I'll help you
off list...). Here's how I'd do up something like you're describing,
assuming the active-server model:

class Client(object):
    def __init__(self, ip):
        self.ip = ip
        self.sock = None
        self.unsent = ""
    def send_message(self, message):
        self.unsent += message
        self.send_queued()
    def send_queued(self):
        if not self.sock:
            # attempt asynchronous establishment of socket
        else:
            # attempt to send text to the socket, asynchronously

If you're using Python 3.4, you could make use of asyncio to do most
of the work. I've not actually put it to use yet, but it looks
excellent. But the above structure is most of what I'd be looking at.
Establish connections only when you need them, and only if you don't
have them already. (Note that this assumes the other end can separate
messages out, so it's happy to combine them. You may need to make sure
you have some kind of boundary - with text, end-of-line makes good
sense.) You would then create a bunch of Clients, one for each IP you
need to connect to, and send_message() to all appropriate ones. It
should be possible to separate your async comms code from your message
generation code, and keep both of them clean.

In short: What you're doing is certainly possible, and makes some
degree of sense. (Though maintaining connections would be both easier
and simpler, if you can resolve your issues.)

ChrisA

[toc] | [prev] | [next] | [standalone]

#80878

jkn <jkn_gg@nicorp.f9.co.uk>:

> Although this ... works ..., we have had trouble maintaining the
> connection, for reasons ... I am not yet fully aware of.

I can see your TCP connections are choppy. Your posting is breaking up.

Seriously, though, there shouldn't be any reason for TCP connections
dropping on their own. You need to have a chat with your IT people.

> We are now considering an alternative approach where the master
> maintains a list of slave devices and acts as a client. Each device is
> a server from the point of view of data transfer. We would then use a
> short-lived TCP connection when data is available; the Master would
> connect to each slave which needed the data, send it, and close the
> connection.
>
> I should also add that we desire our system to be 'robust' in the face
> of situations such as cable unplugging, device power cycles, etc.
>
> Although this might get round some of our current problems, I can see that 
> we might end up with new problems to deal with.

There are numerous ways to accomplish what you desire, and they can all
be made to work.

One simple idea is to use tunneling. IP-over-IP, GRE or OpenVPN
tunneling would not mind the underlying ethernet interfaces going down
occasionally. At worst, there's some IP fragmentation issues, but your
data rates are so low that you wouldn't feel a thing.

> As I say, I am expecting to prototype it in Python so any specifics
> also welcome!

If you set up tunneling, you don't have to change your original working
Python code, and you'd be safe against cable thieves and adventurous IT
administrators.

> (suggestions as to a better forum to ask for previous experience also 

Marko

[toc] | [prev] | [next] | [standalone]

#80879

jkn <jkn_gg@nicorp.f9.co.uk> writes:

> I have a use case of a single 'master' machine which will need to 
> periodically 'push' data to a variety of 'slave' devices on a small local 
> subnet, over Ethernet. We are talking perhaps a dozen devices in all with 
> comms occurring perhaps once very few seconds, to much less often - once per 
> half an hour, or less. There is probably an upper bound of 64KB or so of 
> data that is likely to be sent on each occasion.

OK, no big requirements, but 64K is still way too much to consider UDP.

> Previous similar systems have attempted to do this by maintaining multiple 
> long-term TCP connections from the master to all the slave devices. The 
> Master is the server and the slaves periodically check the master to see 
> what has changed. Although this ... works ..., we have had trouble 
> maintaining the connection, for reasons ... I am not yet fully aware
> of.

This doesn't make much sense to me. On a packet switching network
"maintaining the connexion" simply means keeping both ends alive. There
is nothing "in the middle" that can break the connection (unless you
use, e.g., ssh tunneling, but then it's a configuration problem, or NAT,
but I doubt it is your case on an LAN).

> We are now considering an alternative approach where the master maintains a 
> list of slave devices and acts as a client. Each device is a server from the 
> point of view of data transfer. We would then use a short-lived TCP 
> connection when data is available; the Master would connect to each slave 
> which needed the data, send it, and close the connection.

Yes but a TCP server is slightly more complex: it has to "accept()"
connexions, which means it blocks waiting for something to come in. Or
it has to periodically poll its passive socket, but then you have a
problem with timeouts. Your describing the slaves as "devices" makes me
think they are simple machines, maybe embedded micro-controllers. If it
is the case, you may not want to put the burden on slaves.

(BTW why not have the slaves periodically connect -- as clients -- to
the master? No need to keep the connection open, they can disconnect and
reconnect later. All you need is a way for the master to identify slaves
across several connections, but an IP address should be enough, no?)

> I should also add that we desire our system to be 'robust' in the face of 
> situations such as cable unplugging, device power cycles, etc.

Then avoid connected protocols like TCP. Maybe you should consider SCTP,
which is a message-oriented, reliable protocol. There is a pysctp module
on pypi. (Haven't tried it, I've just made a quick search on SCTP in
Python.)

Or roll your own protocol on top of UDP, but that's another story.

[...]

-- Alain.

[toc] | [prev] | [next] | [standalone]

#80881

On Mon, Nov 17, 2014 at 12:43 AM, Alain Ketterlin
<alain@dpt-info.u-strasbg.fr> wrote:
> jkn <jkn_gg@nicorp.f9.co.uk> writes:
>
>> I have a use case of a single 'master' machine which will need to
>> periodically 'push' data to a variety of 'slave' devices on a small local
>> subnet, over Ethernet. We are talking perhaps a dozen devices in all with
>> comms occurring perhaps once very few seconds, to much less often - once per
>> half an hour, or less. There is probably an upper bound of 64KB or so of
>> data that is likely to be sent on each occasion.
>
> OK, no big requirements, but 64K is still way too much to consider UDP.

I wouldn't say "way too much"; the packet limit for UDP is actually
64KB (minus a few bytes of headers). But UDP for anything more than
your network's MTU is inefficient, plus you'd need to roll your own
acknowledgement system so you know when the client got the data, at
which point you're basically recreating TCP.

That said, though: UDP would be a good option, if and only if you can
comply with those two restrictions - sender doesn't care if the odd
packet doesn't get through, and no packet is allowed to exceed 64KB,
with typical packet sizes wanting to be more like 1KB. (DNS servers
usually switch you to TCP if you go above 512 bytes of response, but
that's because DNS responses are usually tiny.) It'd be pretty easy to
knock together a simple UDP system - you have the clients listen on
some particular port, and you could even make use of IP broadcast to
send to everyone all at once, since this is a single LAN.

>> Previous similar systems have attempted to do this by maintaining multiple
>> long-term TCP connections from the master to all the slave devices. The
>> Master is the server and the slaves periodically check the master to see
>> what has changed. Although this ... works ..., we have had trouble
>> maintaining the connection, for reasons ... I am not yet fully aware
>> of.
>
> This doesn't make much sense to me. On a packet switching network
> "maintaining the connexion" simply means keeping both ends alive. There
> is nothing "in the middle" that can break the connection (unless you
> use, e.g., ssh tunneling, but then it's a configuration problem, or NAT,
> but I doubt it is your case on an LAN).

NAT is the most common cause of breakage, but any router does have the
power to monitor and drop connections on any basis it likes. (I can't
imagine any reason a non-NAT router would want to prevent connections
from going idle, but it could be done.) It's also possible the
connections are being dropped at a lower level; for instance, a
wireless network might drop a client and force it to reconnect. I've
seen this happen with a number of cheap home wireless routers, and
generally Linux boxes hang onto any application-level connections just
fine (assuming DHCP doesn't cause a change of IP address at the same
time), but Windows XP (haven't checked more recent Windowses) goes and
closes any sockets that were using that connection... without sending
RST packets to the server, of course. So the client knows it's lost
link, but the server doesn't, until the next time it tries to send (at
which point it *might* get a courteous RST, or it might have to time
out).

So, I wouldn't say it's impossible for the connections to be dying...
but I would say for certain that connection death is diagnosable, and
on a LAN, often quite easily diagnosable.

> Yes but a TCP server is slightly more complex: it has to "accept()"
> connexions, which means it blocks waiting for something to come in. Or
> it has to periodically poll its passive socket, but then you have a
> problem with timeouts. Your describing the slaves as "devices" makes me
> think they are simple machines, maybe embedded micro-controllers. If it
> is the case, you may not want to put the burden on slaves.

You should be able to do an asynchronous accept. I don't know the
details of doing that in Python, but I expect asyncio can do this for
you. In Pike, I quite frequently run a single-threaded program that
just drops to a back-end loop, listening for new connections, new text
on the current connections, or any other events. At the C level, this
would be done with select() and friends. So you shouldn't have to
block, nor poll, though it would require a bit of extra work.

I wouldn't assume that "devices" are microcontrollers, though. I talk
about full-scale computers that way, if they're the "smaller" end of
the connection. For example, I have a video-playing server which
invokes VLC, and it runs a little HTTP daemon to allow end users to
choose what video gets played. The HTTP clients are usually on various
people's desktop computers, which generally have a lot more grunt than
the little video server does; but I call them "devices" in terms of
the Yosemite Project, because all they are is a source of signals like
"play such-and-such file", "pause", "skip forward", "stop".

> (BTW why not have the slaves periodically connect -- as clients -- to
> the master? No need to keep the connection open, they can disconnect and
> reconnect later. All you need is a way for the master to identify slaves
> across several connections, but an IP address should be enough, no?)

That's an option that works nicely if the slaves know when they've
lost link. Without knowing more about these connection failures, it's
hard to say which end should resolve the issue. In the worst-case
scenario, all transmission is in one direction (server to clients),
and packets for "old" connections simply start getting dropped;
possible, unlikely, and very annoying. In that case, the clients will
have no idea that they need to reconnect - the server will,
eventually, because its data packets aren't getting acknowledged, but
the clients just assume all's well. On the other hand, the example of
Windows XP and wifi is one where the clients know virtually instantly
that they've been cut off, but the server doesn't; if that's what's
happening, then definitely they should be the ones to reconnect (and
the server should probably close off the previous connection from that
IP).

>> I should also add that we desire our system to be 'robust' in the face of
>> situations such as cable unplugging, device power cycles, etc.
>
> Then avoid connected protocols like TCP. Maybe you should consider SCTP,
> which is a message-oriented, reliable protocol. There is a pysctp module
> on pypi. (Haven't tried it, I've just made a quick search on SCTP in
> Python.)
>
> Or roll your own protocol on top of UDP, but that's another story.

Not sure why TCP shouldn't be used here. Depending on the definition
of 'robust', TCP might be anywhere from "overkill but doesn't hurt" to
"perfectly doing what you need". All you need is a little layer around
it to cope with power cycling the device (something like "program
starts on device boot, connects to server, server maybe kicks off the
other connection") and everything else will be handled with return
codes. You do need to make sure you do everything asynchronously,
though - either non-blocking I/O or threads - because otherwise you'll
be sitting there waiting for timeouts when some client disconnects.

OP, more information needed:
1) What should happen when a client is unavailable for a while? Should
messages be queued, or dropped?
2) If a message arbitrarily sometimes doesn't get through, what are
the consequences? Are subsequent messages meaningless, or can it just
move on?
3) What usually happens when a connection breaks? Which end is first
to find out?

ChrisA

[toc] | [prev] | [next] | [standalone]

#80887

On 2014-11-16, Chris Angelico <rosuav@gmail.com> wrote:
>
>> OK, no big requirements, but 64K is still way too much to consider UDP.
>
> I wouldn't say "way too much"; the packet limit for UDP is actually
> 64KB (minus a few bytes of headers). But UDP for anything more than
> your network's MTU is inefficient, plus you'd need to roll your own
> acknowledgement system so you know when the client got the data, at
> which point you're basically recreating TCP.
>
> That said, though: UDP would be a good option, if and only if you can
> comply with those two restrictions - sender doesn't care if the odd
> packet doesn't get through, and no packet is allowed to exceed 64KB,
> with typical packet sizes wanting to be more like 1KB. (DNS servers
> usually switch you to TCP if you go above 512 bytes of response, but
> that's because DNS responses are usually tiny.) It'd be pretty easy to
> knock together a simple UDP system - you have the clients listen on
> some particular port, and you could even make use of IP broadcast to
> send to everyone all at once, since this is a single LAN.

If, as the OP seemed to state, this is strictly within a single
network segement (no firewalls or routers involved), then UDP
multicast might be worth considering.  It's not reliable like TCP, but
it does make everything very simple: just decide how many different
logical data streams you are going to have, and allocate a multicast
address for each one.  Have the server multicast new data whenever it
becomes available, and the slaves all register for and and listen to
for whichever data streams they care about.  Note that the "register
for" (or join, or whatever it's called) isn't something the server
knows about, that's purely done by the slave app to inform the slave's
network stack and any intervening routers what multicast streams the
slave app wants to see.

In _theory_, if betwixt the server and client the routers and
firewalls are properly configured, multicast should still work.  But,
the smart money will be giving long odds that they aren't, and it
won't.

One possible advantage of UDP is that is packet oriented, so if your
data isn't message oriented rather than an undifferentiated byte
stream, UDP saves you the effort of having to packetize and
unpacketize the data as you have to via TCP.

-- 
Grant

[toc] | [prev] | [next] | [standalone]

#80889

In article <mailman.15887.1416150791.18130.python-list@python.org>,
 Chris Angelico <rosuav@gmail.com> wrote:

> UDP for anything more than your network's MTU is inefficient

Why do you say it's inefficient?  Sure, the UDP datagram will get 
fragmented and re-assembled at the other end, but it's not like TCP 
would do any better.  One way or another, your data is going to be 
transmitted in packet that fit into the MTU.

> plus you'd need to roll your own acknowledgement system so you know 
> when the client got the data, at which point you're basically 
> recreating TCP.

That I certainly agree with.  UDP should be used for fire-and-forget, 
where its not critical that every bit of data gets through.  A classic 
example is high-volume logging.  Once you start thinking about any kind 
of acknowledgements, you should just switch to TCP.  The alternative is 
that you will slowly end up reinventing TCP (and doing a bad job of it).

> NAT is the most common cause of breakage, but any router does have the
> power to monitor and drop connections on any basis it likes. (I can't
> imagine any reason a non-NAT router would want to prevent connections
> from going idle, but it could be done.)

It's common in corporate networks for routers to forcibly close idle 
connections (i.e. inject RST packets).  Often, the IT security guys 
think idle connections represent some kind of threat (and good luck 
trying to argue with them).

[toc] | [prev] | [next] | [standalone]

#80890

On 2014-11-16, Roy Smith <roy@panix.com> wrote:
> In article <mailman.15887.1416150791.18130.python-list@python.org>,
>  Chris Angelico <rosuav@gmail.com> wrote:
>
>> UDP for anything more than your network's MTU is inefficient
>
> Why do you say it's inefficient?  Sure, the UDP datagram will get 
> fragmented and re-assembled at the other end, but it's not like TCP 
> would do any better.  One way or another, your data is going to be 
> transmitted in packet that fit into the MTU.
>
>> plus you'd need to roll your own acknowledgement system so you know 
>> when the client got the data, at which point you're basically 
>> recreating TCP.
>
> That I certainly agree with.  UDP should be used for fire-and-forget, 
> where its not critical that every bit of data gets through.  A classic 
> example is high-volume logging.  Once you start thinking about any kind 
> of acknowledgements, you should just switch to TCP.  The alternative is 
> that you will slowly end up reinventing TCP (and doing a bad job of it).
>
>> NAT is the most common cause of breakage, but any router does have the
>> power to monitor and drop connections on any basis it likes. (I can't
>> imagine any reason a non-NAT router would want to prevent connections
>> from going idle, but it could be done.)
>
> It's common in corporate networks for routers to forcibly close idle 
> connections (i.e. inject RST packets).  Often, the IT security guys 
> think idle connections represent some kind of threat (and good luck 
> trying to argue with them).

According to the OP, the communication is "on a small local subnet,
over Ethernet."  To me that means no router, no firewall: just an
Ethernet switch (or two).

If that is the case, and the master and slaves can't maintain TCP
connectivity, then something is seriously broken in the master or
slaves and needs to get fixed.

What I don't understand is if TCP connections are going away (which
can happen if either end gets restarted, or cables get unplugged, or
switches lose power), why the app doesn't just open a new one and
resume operation?

-- 
Grant

[toc] | [prev] | [next] | [standalone]

#80914

On Mon, Nov 17, 2014 at 4:21 AM, Roy Smith <roy@panix.com> wrote:
> In article <mailman.15887.1416150791.18130.python-list@python.org>,
>  Chris Angelico <rosuav@gmail.com> wrote:
>
>> UDP for anything more than your network's MTU is inefficient
>
> Why do you say it's inefficient?  Sure, the UDP datagram will get
> fragmented and re-assembled at the other end, but it's not like TCP
> would do any better.  One way or another, your data is going to be
> transmitted in packet that fit into the MTU.

Sorry, is less efficient. I was responding to the point about 64KB
being way too big for UDP, when really it's perfectly possible. It's
common to keep UDP packet sizes as low as possible to take advantage
of non-fragmentation efficiency (hence the DNS TCP switchover at 512
bytes), but I've sent some huge UDP packets around.

>> plus you'd need to roll your own acknowledgement system so you know
>> when the client got the data, at which point you're basically
>> recreating TCP.
>
> That I certainly agree with.  UDP should be used for fire-and-forget,
> where its not critical that every bit of data gets through.  A classic
> example is high-volume logging.  Once you start thinking about any kind
> of acknowledgements, you should just switch to TCP.  The alternative is
> that you will slowly end up reinventing TCP (and doing a bad job of it).

Another good use of UDP is where it's not 100% critical that *every*
update get to *every* client instantly, but a client can "catch up"
when it gets the next notification. (Since clients can be down, you'd
need some kind of catch-up mechanic anyway.) If the data involved can
fit inside a single UDP packet, this is trivially easy: just overwrite
with the new state. Otherwise, some kind of update counter works too
("hmm, I got update 5, this one's update 7, I guess I'd better catch
up"). If most clients collect most UDP updates, but occasionally they
establish a TCP connection to do the full catch-up, you can get some
handy conveniences.

>> NAT is the most common cause of breakage, but any router does have the
>> power to monitor and drop connections on any basis it likes. (I can't
>> imagine any reason a non-NAT router would want to prevent connections
>> from going idle, but it could be done.)
>
> It's common in corporate networks for routers to forcibly close idle
> connections (i.e. inject RST packets).  Often, the IT security guys
> think idle connections represent some kind of threat (and good luck
> trying to argue with them).

Yeah... but I can't imagine any reason for those connections to be a
threat. I've heard of it happening but I cannot fathom the reasoning
behind it. (You're also suggesting a much nicer approach than the
worst-case I was talking about: with RST injection, at least one end
will be immediately aware of the closure. If the router just starts
dropping packets, much harder.)

ChrisA

[toc] | [prev] | [next] | [standalone]

#80972

On Mon, 17 Nov 2014 08:56:43 +1100, Chris Angelico wrote:

> On Mon, Nov 17, 2014 at 4:21 AM, Roy Smith <roy@panix.com> wrote:
>> In article <mailman.15887.1416150791.18130.python-list@python.org>,
>>  Chris Angelico <rosuav@gmail.com> wrote:
>>
>>> UDP for anything more than your network's MTU is inefficient
>>
>> Why do you say it's inefficient?  Sure, the UDP datagram will get
>> fragmented and re-assembled at the other end, but it's not like TCP
>> would do any better.  One way or another, your data is going to be
>> transmitted in packet that fit into the MTU.
> 
> Sorry, is less efficient. I was responding to the point about 64KB being
> way too big for UDP, when really it's perfectly possible. It's common to
> keep UDP packet sizes as low as possible to take advantage of
> non-fragmentation efficiency (hence the DNS TCP switchover at 512
> bytes), but I've sent some huge UDP packets around.
> 
>>> plus you'd need to roll your own acknowledgement system so you know
>>> when the client got the data, at which point you're basically
>>> recreating TCP.
>>
>> That I certainly agree with.  UDP should be used for fire-and-forget,
>> where its not critical that every bit of data gets through.  A classic
>> example is high-volume logging.  Once you start thinking about any kind
>> of acknowledgements, you should just switch to TCP.  The alternative is
>> that you will slowly end up reinventing TCP (and doing a bad job of
>> it).
> 
> Another good use of UDP is where it's not 100% critical that *every*
> update get to *every* client instantly, but a client can "catch up" when
> it gets the next notification. (Since clients can be down, you'd need
> some kind of catch-up mechanic anyway.) If the data involved can fit
> inside a single UDP packet, this is trivially easy: just overwrite with
> the new state. Otherwise, some kind of update counter works too ("hmm, I
> got update 5, this one's update 7, I guess I'd better catch up"). If
> most clients collect most UDP updates, but occasionally they establish a
> TCP connection to do the full catch-up, you can get some handy
> conveniences.
> 
>>> NAT is the most common cause of breakage, but any router does have the
>>> power to monitor and drop connections on any basis it likes. (I can't
>>> imagine any reason a non-NAT router would want to prevent connections
>>> from going idle, but it could be done.)
>>
>> It's common in corporate networks for routers to forcibly close idle
>> connections (i.e. inject RST packets).  Often, the IT security guys
>> think idle connections represent some kind of threat (and good luck
>> trying to argue with them).
> 
> Yeah... but I can't imagine any reason for those connections to be a
> threat. I've heard of it happening but I cannot fathom the reasoning
> behind it. (You're also suggesting a much nicer approach than the
> worst-case I was talking about: with RST injection, at least one end
> will be immediately aware of the closure. If the router just starts
> dropping packets, much harder.)
> 
> ChrisA

I have recently been caught by this issues with call logging software for 
a PBX I maintain.

TCP sessions can get closed for a multitude of reasons. they are commonly 
closed by the TCP/IP stack because they have been idle too long & the 
resources are needed elsewhere. ideally a close message should be sent 
but this does not always happen (& is perfectly within specification)

if a sending devise finds that its connection is no longer open it should 
re-initiate the connection with a three way handshake.

Certain versions of the PBX firmware did not re initiate the connection 
but just kept retrying indefinitely meaning that the call logger did not 
get the data until the PBX output was stopped & restarted.

(a small python script to emulate things helped me prove what was 
happening :-) )
 



-- 
A horse!  A horse!  My kingdom for a horse!
		-- Wm. Shakespeare, "Henry VI"

[toc] | [prev] | [next] | [standalone]

#80908

Hi All
    Thanks for the various and interesting responses so far. A bit of 
fleshing out in a few areas:

The problems of maintaining the long-term TCP connection is something I'd 
like to leave to one side, for now at least. There are some non-technical 
project issues here which is why I am considering some alternatives. It may 
be that this gets revisited at a later date.

It is possible that the final deployment will, like the prototype, be 
written in Python; again, non-technical issues come into play here.

I use the term 'devices' a bit loosely; some of the 'slave' machines are 
fully-fledged and relatively powerful embedded machines running linux; some 
of them are simpler embedded machines running a scheduler-based system and a 
simple TCP/IP stack. The capability of the latter may be a consideration in 
the final design; they won't be running Python, for instance.

When a 'slave' (not necessarily acting in the client role, AFAICT) is 
unavailable, the behaviour should be that the data is lost, but that it 
should sync up to the next 'block' of data as soon as possible after 
(physical) reconnection, power up etc. An analogy might be with a master and 
multiple slave devices sharing a serial RS-485 bus,.

I have control over the format of the data to be send, so there can and 
should be some indication of the beginning and end of a data 'block'. In 
actual fact the data is very likely to be in JSON.

UDP has been mentioned but I am expecting the comms to be TCP-based, I 
think. The 65KB block size is indicative but I am reluctant to say that this 
is a hard limit.

Chris asks why the slave devices can't periodically connect to the master 
and request data. The reason is that it is unknown when data destined for a 
given slave will be available. The devices would need to connect perhaps 
once a second to get sufficient time resolution, and this seems over-
complicated to me at the moment.

(FWIW there will also have to be some (UDP?) transmission from the slaves to 
the master, for purposes of device discovery)

Thanks for the mention of SCTP; I was aware that there were other protocols 
to consider but haven't yet looked hard at them. One likely issue here is 
that our simpler devices won't be able to support such protocols 
straightforwardly.

I hope this gives sufficient background info to progress the discussion...

    Cheers
    Jon N

[toc] | [prev] | [next] | [standalone]

#80979

On 2014-11-16, jkn <jkn_gg@nicorp.f9.co.uk> wrote:

> An analogy might be with a master and multiple slave devices sharing
> a serial RS-485 bus.

If that's the model you _want_, then UDP multicast matches it almost
exactly.  ;)

> I have control over the format of the data to be send, so there can
> and should be some indication of the beginning and end of a data
> 'block'. In actual fact the data is very likely to be in JSON.

Using TCP with human-readable ASCII data sure makes testing,
prototyping, and troubleshooting a lot easier: you'd be surprised what
you can do with some dummy data files, a shell script, and nc.

> UDP has been mentioned but I am expecting the comms to be TCP-based,
> I think. The 65KB block size is indicative but I am reluctant to say
> that this is a hard limit.
>
> Chris asks why the slave devices can't periodically connect to the
> master and request data. The reason is that it is unknown when data
> destined for a given slave will be available. The devices would need
> to connect perhaps once a second to get sufficient time resolution,
> and this seems over- complicated to me at the moment.

When the TCP connections are failing, does the master know they have
failed?

> (FWIW there will also have to be some (UDP?) transmission from the
> slaves to the master, for purposes of device discovery)

UDP broadcast works pretty well for discoverying nodes (I'm assuming
IPV4 at this point).  One hint for Linux platforms: you have to
disable reverse path filtering <http://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.kernel.rpf.html>
in order to receive a UDP packet whose source address doesn't match
the subnets of the interface on which it was received.  This can come
into play when you need to discover and configure a newly installed
device which hasn't yet been configured to match the subnet.

UDP discovery and configuration is actually much simpler to implement
under IPv6.  If you're doing IPv6, then UDP link-local all-nodes
multicast is trivial, since all nodes _have_ a valid link-local
address, you don't have to worrry about reverse-path filtering.

> Thanks for the mention of SCTP; I was aware that there were other
> protocols to consider but haven't yet looked hard at them. One likely
> issue here is that our simpler devices won't be able to support such
> protocols straightforwardly.

If simple helps, then UDP again fits the bill nicely.

-- 
Grant Edwards               grant.b.edwards        Yow! I love ROCK 'N ROLL!
                                  at               I memorized the all WORDS
                              gmail.com            to "WIPE-OUT" in 1965!!

[toc] | [prev] | [next] | [standalone]

#80886

On 2014-11-16, jkn <jkn_gg@nicorp.f9.co.uk> wrote:

> I have a use case of a single 'master' machine which will need to
> periodically 'push' data to a variety of 'slave' devices on a small
> local subnet, over Ethernet. We are talking perhaps a dozen devices
> in all with comms occurring perhaps once very few seconds, to much
> less often - once per half an hour, or less. There is probably an
> upper bound of 64KB or so of data that is likely to be sent on each
> occasion.
>
> Previous similar systems have attempted to do this by maintaining
> multiple long-term TCP connections from the master to all the slave
> devices. The Master is the server and the slaves periodically check
> the master to see what has changed.

Why "check the master"?  Why not just have the master send updates to
the slaves whenever something changes?

> Although this ... works ..., we have had trouble maintaining the
> connection, for reasons ... I am not yet fully aware of.

You need to find out what's wrong and fix it.

> We are now considering an alternative approach where the master
> maintains a list of slave devices and acts as a client. Each device
> is a server from the point of view of data transfer. We would then
> use a short-lived TCP connection when data is available; the Master
> would connect to each slave which needed the data, send it, and
> close the connection.

Why close the connection?  Just leave it open and re-use it the next
time an update needs to be sent.  If the connection has "gone away"
you can simply open a new one.

But, mostly you need to figure out why you can't maintain a TCP
connection between master and slave and fix that problem.

I think the best approach is for the "master" to act as the server.
Clients connect to the server and then wait for updates to be sent
from the server.  If a connection is dropped (e.g. server restarts),
the client should re-establish a new connection.

If there is very little TCP traffic, it can be difficult to detect an
unplugged cable.  The solution for that is either
  
 1) Enable TCP keepalive.  In a no-traffic situation, that will detect
    an unplugged cable within an hour or two.

 2) If detecting an unplugged cable needs to happen faster, then add a
    heartbeat message to your protocol that gets sent often enough
    that you can detect an unplugged cable in within the required
    amount of time.

-- 
Grant

[toc] | [prev] | [next] | [standalone]

#80965

jkn wrote:

> Hi all
>     This is a little bit OT for this newsgroup, but I intend to use python
> for prototyping at least, and I know there are a lot of knowledgeable
> people using Python in a Network context here...
> 
> I have a use case of a single 'master' machine which will need to
> periodically 'push' data to a variety of 'slave' devices on a small local
> subnet, over Ethernet. We are talking perhaps a dozen devices in all with
> comms occurring perhaps once very few seconds, to much less often - once
> per half an hour, or less. There is probably an upper bound of 64KB or so
> of data that is likely to be sent on each occasion.
> 
> Previous similar systems have attempted to do this by maintaining multiple
> long-term TCP connections from the master to all the slave devices. The
> Master is the server and the slaves periodically check the master to see
> what has changed. Although this ... works ..., we have had trouble
> maintaining the connection, for reasons ... I am not yet fully aware of.
> 
> We are now considering an alternative approach where the master maintains
> a list of slave devices and acts as a client. Each device is a server from
> the point of view of data transfer. We would then use a short-lived TCP
> connection when data is available; the Master would connect to each slave
> which needed the data, send it, and close the connection.
> 
> I should also add that we desire our system to be 'robust' in the face of
> situations such as cable unplugging, device power cycles, etc.
> 
> Although this might get round some of our current problems, I can see that
> we might end up with new problems to deal with. I am wondering if this
> scenario rings bells with anyone, and seeking pointers to what has been
> done elsewhere. As I say, I am expecting to prototype it in Python so any
> specifics also welcome!
> 
> (suggestions as to a better forum to ask for previous experience also
> gratefully received)
> 
> Thanks al lot for any thoughts/suggestions

You said *any* suggestions, so:

I know nothing about them, but would a messaging infrastructure like 
RabbitMQ or ZeroMQ be useful?

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web