Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=jMKOhLkjLQ9i81Nwa5teRWjmmmMTy0ykN0bL+0V2hmi5UeRt9f61Nz851K3nFHADF2 6+HlT+XPYAEQwtN4SXrrN1E8dLqM2NJYhJnsK/XBetpagaq4Pre/a2mwwOT0P8G/77bD wNO5FPp8oFgildIiLn8xkGeRYQbwFYGojS/RY=
MIME-Version: 1.0
In-Reply-To: <8ikj88-bs1.ln1@svn.schaathun.net>
References: <8ikj88-bs1.ln1@svn.schaathun.net>
Date: Tue, 26 Apr 2011 14:31:59 -0700
Subject: Re: client-server parallellised number crunching
From: geremy condra <debatem1@gmail.com>
To: Hans Georg Schaathun <georg@schaathun.net>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: python-list@python.org
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.879.1303853522.9059.python-list@python.org>
Lines: 49
NNTP-Posting-Host: 82.94.164.166
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:4094

On Tue, Apr 26, 2011 at 12:55 PM, Hans Georg Schaathun
<georg@schaathun.net> wrote:
> I wonder if anyone has any experience with this ...
>
> I try to set up a simple client-server system to do some number
> crunching, using a simple ad hoc protocol over TCP/IP. =A0I use
> two Queue objects on the server side to manage the input and the output
> of the client process. =A0A basic system running seemingly fine on a sing=
le
> quad-core box was surprisingly simple to set up, and it seems to give
> me a reasonable speed-up of a factor of around 3-3.5 using four client
> processes in addition to the master process. =A0(If anyone wants more
> details, please ask.)
>
> Now, I would like to use remote hosts as well, more precisely, student
> lab boxen which are rather unreliable. =A0By experience I'd expect to
> lose roughly 4-5 jobs in 100 CPU hours on average. =A0Thus I need some
> way of detecting lost connections and requeue unfinished tasks,
> avoiding any serious delays in this detection. =A0What is the best way to
> do this in python?
>
> It is, of course, possible for the master thread upon processing the
> results, to requeue the tasks for any missing results, but it seems
> to me to be a cleaner solution if I could detect disconnects and
> requeue the tasks from the networking threads. =A0Is that possible
> using python sockets?
>
> Somebody will probably ask why I am not using one of the multiprocessing
> libraries. =A0I have tried at least two, and got trapped by the overhead
> of passing complex pickled objects across. =A0Doing it myself has at leas=
t
> helped me clarify what can be parallelised effectively. =A0Now,
> understanding the parallelisable subproblems better, I could try again,
> if I can trust that these libraries can robustly handle lost clients.
> That I don't know if I can.

Without knowledge of what you're doing it's hard to comment
intelligently, but I'd try something like CHAOS or OpenSSI to see if
you can't get what you need for free, if that doesn't do it then try
dropping a liveCD with Hadoop on it in each machine and running it
that way. If that can't work, try MPI. If you've gotten that far and
nothing does the trick then you're probably going to have to give more
details.

On a side note, there are a reasonable number of examples of how to do
similar things on a GPU in "GPU Gems"- depending on your needs you
might be able to just apply a copypasta solver ;)

Geremy Condra