Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Fri, 24 Feb 2012 18:36:10 +0000
From: MRAB <python@mrabarnett.plus.com>
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: multiprocessing, what am I doing wrong?
References: <CAAoZyYMHv=JCZ-PE4ufuuKmcR5Y3rZNNkDz24kv4uJdframKdA@mail.gmail.com> <4F46A49D.9030905@mrabarnett.plus.com> <CAAoZyYM43CjGuTKS9T65dPuSGjU-JrPNR9H2yC5K2z8RoVM=ew@mail.gmail.com>
In-Reply-To: <CAAoZyYM43CjGuTKS9T65dPuSGjU-JrPNR9H2yC5K2z8RoVM=ew@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Reply-To: python-list@python.org
Newsgroups: comp.lang.python
Message-ID: <mailman.135.1330108574.3037.python-list@python.org>
Lines: 70
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:20828

On 24/02/2012 17:00, Eric Frederich wrote:
> I can sill get it to freeze and nothing is printed out from the other
> except block.
> Does it look like I'm doing anything wrong here?
>
[snip]
I don't normally use multiprocessing, so I forgot about a critical
detail. :-(

When the multiprocessing module starts a process, that process
_imports_ the module which contains the function which is to be run, so
what's happening is that when your script is run, it creates and starts
workers, the multiprocessing module makes a new process for each
worker, each of those processes then imports the script, which creates
and starts workers, etc, leading to an ever-increasing number of
processes.

The solution is to ensure that the script/module distinguishes between
being run as the main script and being imported as a module:

#!/usr/bin/env python

import sys
import Queue
import multiprocessing
import time

def FOO(a, b, c):
     print 'foo', a, b, c
     return (a + b) * c

class MyWorker(multiprocessing.Process):
     def __init__(self, inbox, outbox):
         super(MyWorker, self).__init__()
         self.inbox = inbox
         self.outbox = outbox
         print >> sys.stderr, '1' * 80; sys.stderr.flush()
     def run(self):
         print >> sys.stderr, '2' * 80; sys.stderr.flush()
         while True:
             try:
                 args = self.inbox.get_nowait()
             except Queue.Empty:
                 break
             self.outbox.put(FOO(*args))

if __name__ == '__main__':
     # This file is being run as the main script. This part won't be
     # run if the file is imported.
     todo = multiprocessing.Queue()

     for i in xrange(100):
         todo.put((i, i+1, i+2))

     print >> sys.stderr, 'a' * 80; sys.stderr.flush()
     result_queue = multiprocessing.Queue()

     print >> sys.stderr, 'b' * 80; sys.stderr.flush()
     w1 = MyWorker(todo, result_queue)
     print >> sys.stderr, 'c' * 80; sys.stderr.flush()
     w2 = MyWorker(todo, result_queue)

     print >> sys.stderr, 'd' * 80; sys.stderr.flush()
     w1.start()
     print >> sys.stderr, 'e' * 80; sys.stderr.flush()
     w2.start()
     print >> sys.stderr, 'f' * 80; sys.stderr.flush()

     for i in xrange(100):
         print result_queue.get()