Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #44303

Re: Finding the source of an exception in a python multiprocessing program

Show key headers only | View raw

On 25 April 2013 00:26, Dave Angel <davea@davea.name> wrote:
> On 04/24/2013 05:09 PM, William Ray Wing wrote:
>>
>> On Apr 24, 2013, at 4:31 PM, Neil Cerutti <neilc@norwich.edu> wrote:
>>
>>> On 2013-04-24, William Ray Wing <wrw@mac.com> wrote:
>>>>
>>>> When I look at the pool module, the error is occurring in
>>>> get(self, timeout=None) on the line after the final else:
>>>>
>>>>     def get(self, timeout=None):
>>>>         self.wait(timeout)
>>>>         if not self._ready:
>>>>             raise TimeoutError
>>>>         if self._success:
>>>>             return self._value
>>>>         else:
>>>>             raise self._value
>>>
>>>
>>> The code that's failing is in self.wait. Somewhere in there you
>>> must be masking an exception and storing it in self._value
>>> instead of letting it propogate and crash your program. This is
>>> hiding the actual context.
>>>
>>> --
>>> Neil Cerutti
>>> --
>>> http://mail.python.org/mailman/listinfo/python-list
>>
>>
>> I'm sorry, I'm not following you.  The "get" routine (and thus self.wait)
>> is part of the "pool" module in the Python multiprocessing library.
>> None of my code has a class or function named "get".
>>
>> -Bill
>>
>
> My question is why bother with multithreading?  Why not just do these as
> separate processes?  You said "they in no way interact with each other" and
> that's a clear clue that separate processes would be cleaner.

It's using multiprocessing rather than threads: they are separate processes.

>
> Without knowing anything about those libraries, I'd guess that somewhere
> they do store state in a global attribute or equivalent, and when that is
> accessed by both threads, it can crash.

It's state that is passed to it by the subprocess and should only be
accessed by the top-level process after the subprocess completes (I
think!).

>
> Separate processes will find it much more difficult to interact, which is a
> good thing most of the time.  Further, they seem to be scheduled more
> efficiently because of the GIL, though that may not make that much
> difference when you're time-limited by network data.

They are separate processes and do not share the GIL (unless I'm very
much mistaken). Also I think the underlying program is limited by the
call to sleep for 15 seconds.


Oscar

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Finding the source of an exception in a python multiprocessing program William Ray Wing <wrw@mac.com> - 2013-04-24 15:25 -0400
  Re: Finding the source of an exception in a python multiprocessing program Neil Cerutti <neilc@norwich.edu> - 2013-04-24 20:31 +0000
    Re: Finding the source of an exception in a python multiprocessing program William Ray Wing <wrw@mac.com> - 2013-04-24 17:09 -0400
      Re: Finding the source of an exception in a python multiprocessing program Neil Cerutti <neilc@norwich.edu> - 2013-04-25 12:37 +0000
    Re: Finding the source of an exception in a python multiprocessing program Dave Angel <davea@davea.name> - 2013-04-24 19:26 -0400
    Re: Finding the source of an exception in a python multiprocessing program Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-04-25 01:00 +0100
    Re: Finding the source of an exception in a python multiprocessing program Dave Angel <davea@davea.name> - 2013-04-24 20:11 -0400

csiph-web