Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Wed, 24 Apr 2013 19:26:36 -0400
From: Dave Angel <davea@davea.name>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130329 Thunderbird/17.0.5
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Finding the source of an exception in a python multiprocessing program
References: <mailman.1032.1366835159.3114.python-list@python.org> <atqtpaFa3niU2@mid.individual.net> <5BC4D9D9-D757-40F3-81CA-CF4FA5C73A96@mac.com>
In-Reply-To: <5BC4D9D9-D757-40F3-81CA-CF4FA5C73A96@mac.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.1037.1366846014.3114.python-list@python.org>
Lines: 47
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:44300

On 04/24/2013 05:09 PM, William Ray Wing wrote:
> On Apr 24, 2013, at 4:31 PM, Neil Cerutti <neilc@norwich.edu> wrote:
>
>> On 2013-04-24, William Ray Wing <wrw@mac.com> wrote:
>>> When I look at the pool module, the error is occurring in
>>> get(self, timeout=None) on the line after the final else:
>>>
>>>     def get(self, timeout=None):
>>>         self.wait(timeout)
>>>         if not self._ready:
>>>             raise TimeoutError
>>>         if self._success:
>>>             return self._value
>>>         else:
>>>             raise self._value
>>
>> The code that's failing is in self.wait. Somewhere in there you
>> must be masking an exception and storing it in self._value
>> instead of letting it propogate and crash your program. This is
>> hiding the actual context.
>>
>> --
>> Neil Cerutti
>> --
>> http://mail.python.org/mailman/listinfo/python-list
>
> I'm sorry, I'm not following you.  The "get" routine (and thus self.wait) is part of the "pool" module in the Python multiprocessing library.
> None of my code has a class or function named "get".
>
> -Bill
>

My question is why bother with multithreading?  Why not just do these as 
separate processes?  You said "they in no way interact with each other" 
and that's a clear clue that separate processes would be cleaner.

Without knowing anything about those libraries, I'd guess that somewhere 
they do store state in a global attribute or equivalent, and when that 
is accessed by both threads, it can crash.

Separate processes will find it much more difficult to interact, which 
is a good thing most of the time.  Further, they seem to be scheduled 
more efficiently because of the GIL, though that may not make that much 
difference when you're time-limited by network data.

-- 
DaveA