Re: client-server parallellised number crunching

Path	csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!news.albasani.net!news.teledata-fn.de!newsfeed.arcor.de!newsspool3.arcor-online.net!news.arcor.de.POSTED!not-for-mail
From	Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de>
Newsgroups	comp.lang.python
Subject	Re: client-server parallellised number crunching
Date	Wed, 27 Apr 2011 11:35:16 +0200
Organization	A newly installed InterNetNews server
Message-ID	<ip8o0q$p4f$1@r03.glglgl.eu> (permalink)
References	<8ikj88-bs1.ln1@svn.schaathun.net>
Mime-Version	1.0
Content-Type	text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding	7bit
User-Agent	Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.15) Gecko/20110303 Lightning/1.0b2 Thunderbird/3.1.9 Hamster/2.1.0.11
In-Reply-To	<8ikj88-bs1.ln1@svn.schaathun.net>
Lines	29
NNTP-Posting-Date	27 Apr 2011 11:40:02 CEST
NNTP-Posting-Host	8fcf0b98.newsspool2.arcor-online.net
X-Trace	DXC=;VkjiQ7SSC\5TOT9_N5i<VA9EHlD;3YcR4Fo<]lROoRQ8kF<OcfhCO[S4UU<=N:EQPK8FCa6^2FWROLF_]FfFg8_P2BWBc2:;UX
X-Complaints-To	usenet-abuse@arcor.de
Xref	x330-a1.tempe.blueboxinc.net comp.lang.python:4121

Show key headers only | View raw

Am 26.04.2011 21:55, schrieb Hans Georg Schaathun:

> Now, I would like to use remote hosts as well, more precisely, student
> lab boxen which are rather unreliable.  By experience I'd expect to
> lose roughly 4-5 jobs in 100 CPU hours on average.  Thus I need some
> way of detecting lost connections and requeue unfinished tasks,
> avoiding any serious delays in this detection.  What is the best way to
> do this in python?

As far as I understand, you acquire a job, send it to a remote host via 
a socket and then wait for the answer. Is that correct?

In this case, I would put running jobs together with the respective 
socket in a "running queue". If you detect a broken connection, put that 
job into the "todo" queue again.


> ... if I could detect disconnects and
> requeue the tasks from the networking threads.  Is that possible
> using python sockets?

Of course, why not? It might depend on some settings you set (keepalive 
etc.); but generally you should get an exception when trying a 
communication over a disconnected connection (over a disconnection? ;-))

When going over tne network, aviod pickling. Better use an own protocol.


Thomas

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

client-server parallellised number crunching Hans Georg Schaathun <georg@schaathun.net> - 2011-04-26 20:55 +0100
  Re: client-server parallellised number crunching Chris Angelico <rosuav@gmail.com> - 2011-04-27 06:20 +1000
  Re: client-server parallellised number crunching Dan Stromberg <drsalists@gmail.com> - 2011-04-26 13:31 -0700
  Re: client-server parallellised number crunching Dan Stromberg <drsalists@gmail.com> - 2011-04-26 13:33 -0700
    Re: client-server parallellised number crunching Hans Georg Schaathun <hg@schaathun.net> - 2011-04-26 21:47 +0100
      Re: client-server parallellised number crunching Chris Angelico <rosuav@gmail.com> - 2011-04-27 07:07 +1000
  Re: client-server parallellised number crunching Chris Angelico <rosuav@gmail.com> - 2011-04-27 06:35 +1000
  Re: client-server parallellised number crunching geremy condra <debatem1@gmail.com> - 2011-04-26 14:31 -0700
    Re: client-server parallellised number crunching Hans Georg Schaathun <georg@schaathun.net> - 2011-04-27 06:58 +0100
      Re: client-server parallellised number crunching geremy condra <debatem1@gmail.com> - 2011-04-26 23:54 -0700
        Re: client-server parallellised number crunching Hans Georg Schaathun <georg@schaathun.net> - 2011-04-27 10:57 +0100
  Re: client-server parallellised number crunching Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2011-04-27 11:35 +0200
    Re: client-server parallellised number crunching Hans Georg Schaathun <hg@schaathun.net> - 2011-04-27 13:21 +0100
      Re: client-server parallellised number crunching Chris Angelico <rosuav@gmail.com> - 2011-04-27 23:35 +1000
        Re: client-server parallellised number crunching Hans Georg Schaathun <hg@schaathun.net> - 2011-04-27 15:15 +0100
          Re: client-server parallellised number crunching Chris Angelico <rosuav@gmail.com> - 2011-04-28 00:58 +1000
            Re: client-server parallellised number crunching Hans Georg Schaathun <hg@schaathun.net> - 2011-04-27 19:28 +0100

csiph-web