Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <CALR-gidSJ9sDxnQVFETDmZJL1AfLFmkH0cZfyMOvhpF_ubTi=Q@mail.gmail.com>
References: <CALR-gidSJ9sDxnQVFETDmZJL1AfLFmkH0cZfyMOvhpF_ubTi=Q@mail.gmail.com>
Date: Wed, 3 Aug 2011 16:51:20 +0100
Subject: Re: python distributed computing
From: Chris Angelico <rosuav@gmail.com>
To: python-list@python.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.1842.1312386683.1164.python-list@python.org>
Lines: 37
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:10810

On Wed, Aug 3, 2011 at 4:37 PM, julien godin <t4rtine@gmail.com> wrote:
> I have a HUUUUUGE(everything is relative) amount of data coming from UDP/=
514
> ( good old syslog), and one process alone could not handle them ( it's
> full=A0of regex, comparison and counting. all this at about 100K lines of=
 data
> per seconds and each line is about 1500o long. I tried to handle it on on=
e
> process, 100% core time in htop. )
> So i said to myself : why not distribute the computing ?

You could brute-force it by forking out to multiple computers, but
that would entail network traffic, which might defeat the purpose. If
you have a multi-core CPU or multi-CPU computer, a better option would
be to look into the multiprocessing module for some simple ways to
divide the work between cores/CPUs. Otherwise, there's a few things to
try.

First and most important thing: Optimize your algorithms! Can you do
less work and still get the same result? For instance, can you replace
the regex with two or three simpler string methods? Profile your code
to find out where most of the time is being spent, and see if you can
recode those parts.

Second: Try PyPy or Cython for higher-performance Python code.

And third, if you still can't get the performance you need, consider
changing languages. Python isn't the best language for fast execution
- it's good for fast coding. Rewrite some or all of your code in C,
Pike, COBOL, raw assembly language... okay, maybe not those last two!
Since you profiled your code up in step 1, you'll know which parts are
the best candidates for C code.

I think there are quite a few options better than forking across
computers; although distributed computing IS a lot of fun.

Chris Angelico