Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #58311 > unrolled thread

multiprocessing: child process race to answer

Back to article view | Back to comp.lang.python

Contents

  multiprocessing: child process race to answer smhall05 <smhall05@gmail.com> - 2013-11-01 19:35 -0700
    Re: multiprocessing: child process race to answer MRAB <python@mrabarnett.plus.com> - 2013-11-02 02:52 +0000
      Re: multiprocessing: child process race to answer smhall05 <smhall05@gmail.com> - 2013-11-01 22:03 -0700
        Re: multiprocessing: child process race to answer William Ray Wing <wrw@mac.com> - 2013-11-02 08:17 -0400
        Re: multiprocessing: child process race to answer Sherard Hall <smhall05@gmail.com> - 2013-11-02 11:44 -0400
        Re: multiprocessing: child process race to answer (forgot to Cc: the list) William Ray Wing <wrw@mac.com> - 2013-11-02 23:07 -0400
    Re: multiprocessing: child process race to answer cappleman@gmail.com - 2013-11-03 02:10 -0800
      Re: multiprocessing: child process race to answer Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-11-03 10:24 +0000

#58311 — multiprocessing: child process race to answer

I am using a basic multiprocessing snippet I found:

#-----------------------------------------------------
from multiprocessing import Pool

def  f(x):
    return x*x

if __name__ == '__main__':
    pool = Pool(processes=4)              # start 4 worker processes
    result = pool.apply_async(f, [10])    # evaluate "f(10)" asynchronously
    print result.get(timeout=1)           
    print pool.map(f, range(10))          # prints "[0, 1, 4,..., 81]"
#---------------------------------------------------------

I am using this code to have each process go off and solve the same problem, just with different inputs to the problem. I need to be able to kill all processes once 1 of n processes has come up with the solution. There will only be one answer.

I have tried:

sys.exit(0) #this causes the program to hang
pool.close()
pool.terminate

These still allow further processing before the program terminates. What else can I try? I am not able to share the exact code at this time. I can provide more detail if I am unclear. Thank you

[toc] | [next] | [standalone]

#58312

On 02/11/2013 02:35, smhall05 wrote:
> I am using a basic multiprocessing snippet I found:
>
> #-----------------------------------------------------
> from multiprocessing import Pool
>
> def  f(x):
>      return x*x
>
> if __name__ == '__main__':
>      pool = Pool(processes=4)              # start 4 worker processes
>      result = pool.apply_async(f, [10])    # evaluate "f(10)" asynchronously
>      print result.get(timeout=1)
>      print pool.map(f, range(10))          # prints "[0, 1, 4,..., 81]"
> #---------------------------------------------------------
>
> I am using this code to have each process go off and solve the same problem, just with different inputs to the problem. I need to be able to kill all processes once 1 of n processes has come up with the solution. There will only be one answer.
>
> I have tried:
>
> sys.exit(0) #this causes the program to hang
> pool.close()
> pool.terminate
>
Did you actually mean "pool.terminate", or is that a typo for
"pool.terminate()"?

> These still allow further processing before the program terminates. What else can I try? I am not able to share the exact code at this time. I can provide more detail if I am unclear. Thank you
>

[toc] | [prev] | [next] | [standalone]

#58315

On Friday, November 1, 2013 10:52:40 PM UTC-4, MRAB wrote:
> On 02/11/2013 02:35, smhall05 wrote:
> 
> > I am using a basic multiprocessing snippet I found:
> 
> >
> 
> > #-----------------------------------------------------
> 
> > from multiprocessing import Pool
> 
> >
> 
> > def  f(x):
> 
> >      return x*x
> 
> >
> 
> > if __name__ == '__main__':
> 
> >      pool = Pool(processes=4)              # start 4 worker processes
> 
> >      result = pool.apply_async(f, [10])    # evaluate "f(10)" asynchronously
> 
> >      print result.get(timeout=1)
> 
> >      print pool.map(f, range(10))          # prints "[0, 1, 4,..., 81]"
> 
> > #---------------------------------------------------------
> 
> >
> 
> > I am using this code to have each process go off and solve the same problem, just with different inputs to the problem. I need to be able to kill all processes once 1 of n processes has come up with the solution. There will only be one answer.
> 
> >
> 
> > I have tried:
> 
> >
> 
> > sys.exit(0) #this causes the program to hang
> 
> > pool.close()
> 
> > pool.terminate
> 
> >
> 
> Did you actually mean "pool.terminate", or is that a typo for
> 
> "pool.terminate()"?
> 
> 
> 
> > These still allow further processing before the program terminates. What else can I try? I am not able to share the exact code at this time. I can provide more detail if I am unclear. Thank you
> 
> >

I am not sure to be honest, however it turns out that I can't use pool.terminate() because pool is defined in main and not accessible under my def in which I check for the correct answer.

[toc] | [prev] | [next] | [standalone]

#58324

On Nov 2, 2013, at 1:03 AM, smhall05 <smhall05@gmail.com> wrote:

> On Friday, November 1, 2013 10:52:40 PM UTC-4, MRAB wrote:
>> On 02/11/2013 02:35, smhall05 wrote:
>> 
>>> I am using a basic multiprocessing snippet I found:
>>> 
>>> #-----------------------------------------------------
>>> from multiprocessing import Pool
>>> 
>>> def  f(x):
>>>     return x*x
>>> 
>>> if __name__ == '__main__':
>>>     pool = Pool(processes=4)              # start 4 worker processes
>>>     result = pool.apply_async(f, [10])    # evaluate "f(10)" asynchronously
>>>     print result.get(timeout=1)
>>>     print pool.map(f, range(10))          # prints "[0, 1, 4,..., 81]"
>>> #---------------------------------------------------------
>>> 
>>> I am using this code to have each process go off and solve the same problem, just with different inputs to the problem. I need to be able to kill all processes once 1 of n processes has come up with the solution. There will only be one answer.
>>> 
>>> I have tried:
>>> 
>>> sys.exit(0) #this causes the program to hang
>>> pool.close()
>>> pool.terminate
>>> 
>> 
>> Did you actually mean "pool.terminate", or is that a typo for
>> 
>> "pool.terminate()"?
>> 
>>> These still allow further processing before the program terminates. What else can I try? I am not able to share the exact code at this time. I can provide more detail if I am unclear. Thank you
>>> 
> 
> I am not sure to be honest, however it turns out that I can't use pool.terminate() because pool is defined in main and not accessible under my def in which I check for the correct answer.
> -- 
> https://mail.python.org/mailman/listinfo/python-list

So, the simplest solution to that situation is to have whichever subprocess that finds the correct answer set a flag which the calling process can check.  Depending on your OS, that flag can be anything from setting a lock to something as simple as creating a file which the calling process periodically wakes up and looks for, maybe just a file in which the subprocess has written the answer.

Bill

[toc] | [prev] | [next] | [standalone]

#58328

[Multipart message — attachments visible in raw view] — view raw

Thank you for the response. Processing time is very important so I suspect
having to write to disk will take more time than letting the other
processes complete without finding the answer. So I did some profiling one
process finds the answer in about 250ms, but since I can't stop the other
processes, it takes about 800ms before I can use the answer.  Do you
recommend a global variable flag? Any other suggestions?
On Nov 2, 2013 8:17 AM, "William Ray Wing" <wrw@mac.com> wrote:

> On Nov 2, 2013, at 1:03 AM, smhall05 <smhall05@gmail.com> wrote:
>
> > On Friday, November 1, 2013 10:52:40 PM UTC-4, MRAB wrote:
> >> On 02/11/2013 02:35, smhall05 wrote:
> >>
> >>> I am using a basic multiprocessing snippet I found:
> >>>
> >>> #-----------------------------------------------------
> >>> from multiprocessing import Pool
> >>>
> >>> def  f(x):
> >>>     return x*x
> >>>
> >>> if __name__ == '__main__':
> >>>     pool = Pool(processes=4)              # start 4 worker processes
> >>>     result = pool.apply_async(f, [10])    # evaluate "f(10)"
> asynchronously
> >>>     print result.get(timeout=1)
> >>>     print pool.map(f, range(10))          # prints "[0, 1, 4,..., 81]"
> >>> #---------------------------------------------------------
> >>>
> >>> I am using this code to have each process go off and solve the same
> problem, just with different inputs to the problem. I need to be able to
> kill all processes once 1 of n processes has come up with the solution.
> There will only be one answer.
> >>>
> >>> I have tried:
> >>>
> >>> sys.exit(0) #this causes the program to hang
> >>> pool.close()
> >>> pool.terminate
> >>>
> >>
> >> Did you actually mean "pool.terminate", or is that a typo for
> >>
> >> "pool.terminate()"?
> >>
> >>> These still allow further processing before the program terminates.
> What else can I try? I am not able to share the exact code at this time. I
> can provide more detail if I am unclear. Thank you
> >>>
> >
> > I am not sure to be honest, however it turns out that I can't use
> pool.terminate() because pool is defined in main and not accessible under
> my def in which I check for the correct answer.
> > --
> > https://mail.python.org/mailman/listinfo/python-list
>
> So, the simplest solution to that situation is to have whichever
> subprocess that finds the correct answer set a flag which the calling
> process can check.  Depending on your OS, that flag can be anything from
> setting a lock to something as simple as creating a file which the calling
> process periodically wakes up and looks for, maybe just a file in which the
> subprocess has written the answer.
>
> Bill
>
>

[toc] | [prev] | [next] | [standalone]

#58357 — Re: multiprocessing: child process race to answer (forgot to Cc: the list)

[Multipart message — attachments visible in raw view] — view raw

On Nov 2, 2013, at 11:44 AM, Sherard Hall <smhall05@gmail.com> wrote:

> Thank you for the response. Processing time is very important so I suspect having to write to disk will take more time than letting the other processes complete without finding the answer. So I did some profiling one process finds the answer in about 250ms, but since I can't stop the other processes, it takes about 800ms before I can use the answer.  Do you recommend a global variable flag? Any other suggestions?
> 
> On Nov 2, 2013 8:17 AM, "William Ray Wing" <wrw@mac.com> wrote:
> On Nov 2, 2013, at 1:03 AM, smhall05 <smhall05@gmail.com> wrote:
> 
> > On Friday, November 1, 2013 10:52:40 PM UTC-4, MRAB wrote:
> >> On 02/11/2013 02:35, smhall05 wrote:
> >>
> >>> I am using a basic multiprocessing snippet I found:
> >>>
> >>> #-----------------------------------------------------
> >>> from multiprocessing import Pool
> >>>
> >>> def  f(x):
> >>>     return x*x
> >>>
> >>> if __name__ == '__main__':
> >>>     pool = Pool(processes=4)              # start 4 worker processes
> >>>     result = pool.apply_async(f, [10])    # evaluate "f(10)" asynchronously
> >>>     print result.get(timeout=1)
> >>>     print pool.map(f, range(10))          # prints "[0, 1, 4,..., 81]"
> >>> #---------------------------------------------------------
> >>>
> >>> I am using this code to have each process go off and solve the same problem, just with different inputs to the problem. I need to be able to kill all processes once 1 of n processes has come up with the solution. There will only be one answer.
> >>>
> >>> I have tried:
> >>>
> >>> sys.exit(0) #this causes the program to hang
> >>> pool.close()
> >>> pool.terminate
> >>>
> >>
> >> Did you actually mean "pool.terminate", or is that a typo for
> >>
> >> "pool.terminate()"?
> >>
> >>> These still allow further processing before the program terminates. What else can I try? I am not able to share the exact code at this time. I can provide more detail if I am unclear. Thank you
> >>>
> >
> > I am not sure to be honest, however it turns out that I can't use pool.terminate() because pool is defined in main and not accessible under my def in which I check for the correct answer.
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> 
> So, the simplest solution to that situation is to have whichever subprocess that finds the correct answer set a flag which the calling process can check.  Depending on your OS, that flag can be anything from setting a lock to something as simple as creating a file which the calling process periodically wakes up and looks for, maybe just a file in which the subprocess has written the answer.
> 
> Bill
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

Well, the multiprocessing library provides listeners and clients that wrap BSD style sockets and allow you to send (push) arbitrary python objects to a listener, i.e., the master.  There might be something better that was OS specific, but this will keep it pure python.  I've not tested it, but there is a simple example here on Stackoverflow: 

http://stackoverflow.com/questions/6920858/interprocess-communication-in-python

-Bill

[toc] | [prev] | [next] | [standalone]

#58384

On Friday, November 1, 2013 10:35:47 PM UTC-4, smhall05 wrote:
> I am using a basic multiprocessing snippet I found:
> 
> 
> 
> #-----------------------------------------------------
> 
> from multiprocessing import Pool
> 
> 
> 
> def  f(x):
> 
>     return x*x
> 
> 
> 
> if __name__ == '__main__':
> 
>     pool = Pool(processes=4)              # start 4 worker processes
> 
>     result = pool.apply_async(f, [10])    # evaluate "f(10)" asynchronously
> 
>     print result.get(timeout=1)           
> 
>     print pool.map(f, range(10))          # prints "[0, 1, 4,..., 81]"
> 
> #---------------------------------------------------------
> 
> 
> 
> I am using this code to have each process go off and solve the same problem, just with different inputs to the problem. I need to be able to kill all processes once 1 of n processes has come up with the solution. There will only be one answer.
> 
> 
> 
> I have tried:
> 
> 
> 
> sys.exit(0) #this causes the program to hang
> 
> pool.close()
> 
> pool.terminate
> 
> 
> 
> These still allow further processing before the program terminates. What else can I try? I am not able to share the exact code at this time. I can provide more detail if I am unclear. Thank you

You could take a look at the Mutiprocessing module capabilities for exchanging objects between processes:

http://docs.python.org/3.3/library/multiprocessing.html#exchanging-objects-between-processes

[toc] | [prev] | [next] | [standalone]

#58386

On 03/11/2013 10:10, cappleman@gmail.com wrote:
> On Friday, November 1, 2013 10:35:47 PM UTC-4, smhall05 wrote:
>> I am using a basic multiprocessing snippet I found:
>>
>>
>>
>> #-----------------------------------------------------
>>
>> from multiprocessing import Pool
>>
>>
>>
>> def  f(x):
>>
>>      return x*x
>>
>>
>>
>> if __name__ == '__main__':
>>
>>      pool = Pool(processes=4)              # start 4 worker processes
>>
>>      result = pool.apply_async(f, [10])    # evaluate "f(10)" asynchronously
>>
>>      print result.get(timeout=1)
>>
>>      print pool.map(f, range(10))          # prints "[0, 1, 4,..., 81]"
>>
>> #---------------------------------------------------------
>>
>>
>>
>> I am using this code to have each process go off and solve the same problem, just with different inputs to the problem. I need to be able to kill all processes once 1 of n processes has come up with the solution. There will only be one answer.
>>
>>
>>
>> I have tried:
>>
>>
>>
>> sys.exit(0) #this causes the program to hang
>>
>> pool.close()
>>
>> pool.terminate
>>
>>
>>
>> These still allow further processing before the program terminates. What else can I try? I am not able to share the exact code at this time. I can provide more detail if I am unclear. Thank you
>
> You could take a look at the Mutiprocessing module capabilities for exchanging objects between processes:
>
> http://docs.python.org/3.3/library/multiprocessing.html#exchanging-objects-between-processes
>

Would you please read and action this as it prevents the double line 
spacing that you can observe above, thanks 
https://wiki.python.org/moin/GoogleGroupsPython

-- 
Python is the second best programming language in the world.
But the best has yet to be invented.  Christian Tismer

Mark Lawrence

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web