Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!news.albasani.net!feeder.news-service.com!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=f3BliAsy5N48qti3+ytktmHf/ZkRRqBjbgv9C0ng1EGDplJlbiXFVUbCANoshVcyqV CGesU5uARR7SCiNN+tCslByhMcrTLDQmolIzPyx0huG637lboVsrQzVWKvq+Q6f8VaOV xD789au6gss32meN13hdOpSaa89aT9k/7W07w=
MIME-Version: 1.0
Date: Tue, 17 May 2011 11:14:21 -0400
Subject: portable multiprocessing code
From: Eric Frederich <eric.frederich@gmail.com>
To: python-list@python.org
Content-Type: text/plain; charset=ISO-8859-1
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.1674.1305645262.9059.python-list@python.org>
Lines: 61
NNTP-Posting-Host: 82.94.164.166
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:5570

I have written some code using Python 2.7 but I'd like these scripts
to be able to run on Red Hat 5's 2.4.3 version of Python which doesn't
have multiprocessing.
I can try to import multiprocessing and set a flag as to whether it is
available.  Then I can create a Queue.Queue instead of a
multiprocessing.Queue for the arg_queue and result_queue.
Without actually trying this yet it seems like things would work okay
except for the Worker class.  It seems I can conditionally replace
multiprocessing.Queue with Queue.Queue, but is there anything to
replace multiprocessing.Process with?

Are there any best practices for doing something like this?
Below is a dumb example that just counts lines in files.
What would be the best way to make this runnable in older (2.4.3)
versions of Python?


#!/usr/bin/env python

import sys
import os
import multiprocessing
import Queue

fnames = sys.argv[1:]

def SimpleWorker(func):
    class SimpleWorker_wrapped(multiprocessing.Process):
        def __init__(self, arg_queue, result_queue):
            super(SimpleWorker_wrapped, self).__init__()
            self.arg_queue    = arg_queue
            self.result_queue = result_queue
        def run(self):
            while True:
                try:
                    args = self.arg_queue.get_nowait()
                except Queue.Empty:
                    break
                self.result_queue.put(func(*args))
    return SimpleWorker_wrapped

@SimpleWorker
def line_counter(fname):
    lc = len(open(fname).read().splitlines())
    return fname, lc

arg_queue = multiprocessing.Queue()
result_queue = multiprocessing.Queue()
for fname in fnames:
    arg_queue.put((fname,))

for i in range(multiprocessing.cpu_count()):
    w = line_counter(arg_queue, result_queue)
    w.start()

results = {}
for fname in sorted(fnames):
    while fname not in results:
        n, i = result_queue.get()
        results[n] = i
    print "%-40s %d" % (fname, results[fname])