Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #5570 > unrolled thread

portable multiprocessing code

Back to article view | Back to comp.lang.python

Contents

  portable multiprocessing code Eric Frederich <eric.frederich@gmail.com> - 2011-05-17 11:14 -0400

#5570 — portable multiprocessing code

I have written some code using Python 2.7 but I'd like these scripts
to be able to run on Red Hat 5's 2.4.3 version of Python which doesn't
have multiprocessing.
I can try to import multiprocessing and set a flag as to whether it is
available.  Then I can create a Queue.Queue instead of a
multiprocessing.Queue for the arg_queue and result_queue.
Without actually trying this yet it seems like things would work okay
except for the Worker class.  It seems I can conditionally replace
multiprocessing.Queue with Queue.Queue, but is there anything to
replace multiprocessing.Process with?

Are there any best practices for doing something like this?
Below is a dumb example that just counts lines in files.
What would be the best way to make this runnable in older (2.4.3)
versions of Python?


#!/usr/bin/env python

import sys
import os
import multiprocessing
import Queue

fnames = sys.argv[1:]

def SimpleWorker(func):
    class SimpleWorker_wrapped(multiprocessing.Process):
        def __init__(self, arg_queue, result_queue):
            super(SimpleWorker_wrapped, self).__init__()
            self.arg_queue    = arg_queue
            self.result_queue = result_queue
        def run(self):
            while True:
                try:
                    args = self.arg_queue.get_nowait()
                except Queue.Empty:
                    break
                self.result_queue.put(func(*args))
    return SimpleWorker_wrapped

@SimpleWorker
def line_counter(fname):
    lc = len(open(fname).read().splitlines())
    return fname, lc

arg_queue = multiprocessing.Queue()
result_queue = multiprocessing.Queue()
for fname in fnames:
    arg_queue.put((fname,))

for i in range(multiprocessing.cpu_count()):
    w = line_counter(arg_queue, result_queue)
    w.start()

results = {}
for fname in sorted(fnames):
    while fname not in results:
        n, i = result_queue.get()
        results[n] = i
    print "%-40s %d" % (fname, results[fname])

[toc] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web