Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.008 X-Spam-Evidence: '*H*': 0.98; '*S*': 0.00; 'mrab': 0.05; '__name__': 0.09; 'friday,': 0.09; 'snippet': 0.09; 'subject:process': 0.09; 'whichever': 0.09; 'cc:addr:python-list': 0.11; 'def': 0.12; '#this': 0.16; '>>': 0.16; "'__main__':": 0.16; 'check.': 0.16; 'finds': 0.16; 'for,': 0.16; 'inputs': 0.16; 'letting': 0.16; 'simplest': 0.16; 'subject:skip:m 10': 0.16; 'sys.exit(0)': 0.16; 'tried:': 0.16; 'try?': 0.16; 'typo': 0.16; 'wrote:': 0.18; 'variable': 0.18; 'solution.': 0.20; 'written': 0.21; '>>>': 0.22; 'import': 0.22; 'email addr:gmail.com>': 0.22; 'cc:addr:python.org': 0.22; 'print': 0.22; 'creating': 0.23; '>>>': 0.24; 'looks': 0.24; 'cc:2**0': 0.24; 'cc:no real name:2**0': 0.24; '>': 0.26; 'defined': 0.27; 'header:In-Reply- To:1': 0.27; 'correct': 0.29; 'am,': 0.29; 'message- id:@mail.gmail.com': 0.30; 'url:mailman': 0.30; 'code': 0.31; 'os,': 0.31; 'prints': 0.31; 'ray': 0.31; 'file': 0.32; 'url:python': 0.33; 'maybe': 0.34; 'basic': 0.35; "can't": 0.35; 'problem.': 0.35; 'something': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'there': 0.35; 'accessible': 0.36; 'disk': 0.36; 'url:listinfo': 0.36; 'url:org': 0.36; 'detail': 0.37; 'so,': 0.37; 'skip:& 10': 0.38; 'thank': 0.38; 'nov': 0.38; 'anything': 0.39; 'skip:& 20': 0.39; 'bill': 0.39; 'sure': 0.39; 'url:mail': 0.40; 'solve': 0.60; 'simple': 0.61; 'further': 0.61; 'complete': 0.62; 'provide': 0.64; 'more': 0.64; 'different': 0.65; 'situation': 0.65; 'hang': 0.67; 'response.': 0.68; 'answer.': 0.68; 'periodically': 0.68; 'evaluate': 0.72; 'wakes': 0.84; '2013,': 0.91; 'processes,': 0.91; 'subject:answer': 0.95; '2013': 0.98 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=7TYrk2Y4XkBI26BnkOPuNExqWM0BGfaq//RFnYOnOTw=; b=CG19y2/qGqI47kg2Sz3EfeugqjyHsqNcDqH9ulN/L1+hSXqlRmrjxm9NtAofm92aMm Oba3q1dIJEw61WPI7VTxnydCSZL55n6A5tQPS3qp8CR9ZxT/PvglCoj1dCXQDL1tP08b xa0mOkiQVggf7RJupfykVb46VqFjdRG2iU0RnyMgq2PloDwVKuReVeI/gV8gvi68OkBb PhwJpKx/XMwIR7U6rdgkY6wpWlJmCZs59MhL/EHKQieAzM+Qw+s2nMZLqdOKtjOnn64s m50VnlYPYGbz5dQgoad+Dq3gtCFodLIJ3gANaikIwJQrhh6gYjQ6FAYZmIHQEikhmHbY qgyA== MIME-Version: 1.0 X-Received: by 10.112.143.3 with SMTP id sa3mr4920673lbb.12.1383407068615; Sat, 02 Nov 2013 08:44:28 -0700 (PDT) In-Reply-To: <6276BD6E-B0A2-4443-9274-E27BBC3C0152@mac.com> References: <6276BD6E-B0A2-4443-9274-E27BBC3C0152@mac.com> Date: Sat, 2 Nov 2013 11:44:28 -0400 Subject: Re: multiprocessing: child process race to answer From: Sherard Hall To: William Ray Wing Content-Type: multipart/alternative; boundary=089e011827aee799eb04ea338e08 Cc: python-list@python.org X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 152 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1383407076 news.xs4all.nl 15991 [2001:888:2000:d::a6]:54667 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:58328 --089e011827aee799eb04ea338e08 Content-Type: text/plain; charset=ISO-8859-1 Thank you for the response. Processing time is very important so I suspect having to write to disk will take more time than letting the other processes complete without finding the answer. So I did some profiling one process finds the answer in about 250ms, but since I can't stop the other processes, it takes about 800ms before I can use the answer. Do you recommend a global variable flag? Any other suggestions? On Nov 2, 2013 8:17 AM, "William Ray Wing" wrote: > On Nov 2, 2013, at 1:03 AM, smhall05 wrote: > > > On Friday, November 1, 2013 10:52:40 PM UTC-4, MRAB wrote: > >> On 02/11/2013 02:35, smhall05 wrote: > >> > >>> I am using a basic multiprocessing snippet I found: > >>> > >>> #----------------------------------------------------- > >>> from multiprocessing import Pool > >>> > >>> def f(x): > >>> return x*x > >>> > >>> if __name__ == '__main__': > >>> pool = Pool(processes=4) # start 4 worker processes > >>> result = pool.apply_async(f, [10]) # evaluate "f(10)" > asynchronously > >>> print result.get(timeout=1) > >>> print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]" > >>> #--------------------------------------------------------- > >>> > >>> I am using this code to have each process go off and solve the same > problem, just with different inputs to the problem. I need to be able to > kill all processes once 1 of n processes has come up with the solution. > There will only be one answer. > >>> > >>> I have tried: > >>> > >>> sys.exit(0) #this causes the program to hang > >>> pool.close() > >>> pool.terminate > >>> > >> > >> Did you actually mean "pool.terminate", or is that a typo for > >> > >> "pool.terminate()"? > >> > >>> These still allow further processing before the program terminates. > What else can I try? I am not able to share the exact code at this time. I > can provide more detail if I am unclear. Thank you > >>> > > > > I am not sure to be honest, however it turns out that I can't use > pool.terminate() because pool is defined in main and not accessible under > my def in which I check for the correct answer. > > -- > > https://mail.python.org/mailman/listinfo/python-list > > So, the simplest solution to that situation is to have whichever > subprocess that finds the correct answer set a flag which the calling > process can check. Depending on your OS, that flag can be anything from > setting a lock to something as simple as creating a file which the calling > process periodically wakes up and looks for, maybe just a file in which the > subprocess has written the answer. > > Bill > > --089e011827aee799eb04ea338e08 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Thank you for the response. Processing time is very importan= t so I suspect having to write to disk will take more time than letting the= other processes complete without finding the answer. So I did some profili= ng one process finds the answer in about 250ms, but since I can't stop = the other processes, it takes about 800ms before I can use the answer.=A0 D= o you recommend a global variable flag? Any other suggestions?

On Nov 2, 2013 8:17 AM, "William Ray Wing&q= uot; <wrw@mac.com> wrote:
On Nov 2, 2013, at 1:03 AM, smhall05 <smhall05@gmail.com> wrote:

> On Friday, November 1, 2013 10:52:40 PM UTC-4, MRAB wrote:
>> On 02/11/2013 02:35, smhall05 wrote:
>>
>>> I am using a basic multiprocessing snippet I found:
>>>
>>> #-----------------------------------------------------
>>> from multiprocessing import Pool
>>>
>>> def =A0f(x):
>>> =A0 =A0 return x*x
>>>
>>> if __name__ =3D=3D '__main__':
>>> =A0 =A0 pool =3D Pool(processes=3D4) =A0 =A0 =A0 =A0 =A0 =A0 = =A0# start 4 worker processes
>>> =A0 =A0 result =3D pool.apply_async(f, [10]) =A0 =A0# evaluate= "f(10)" asynchronously
>>> =A0 =A0 print result.get(timeout=3D1)
>>> =A0 =A0 print pool.map(f, range(10)) =A0 =A0 =A0 =A0 =A0# prin= ts "[0, 1, 4,..., 81]"
>>> #---------------------------------------------------------
>>>
>>> I am using this code to have each process go off and solve the= same problem, just with different inputs to the problem. I need to be able= to kill all processes once 1 of n processes has come up with the solution.= There will only be one answer.
>>>
>>> I have tried:
>>>
>>> sys.exit(0) #this causes the program to hang
>>> pool.close()
>>> pool.terminate
>>>
>>
>> Did you actually mean "pool.terminate", or is that a typ= o for
>>
>> "pool.terminate()"?
>>
>>> These still allow further processing before the program termin= ates. What else can I try? I am not able to share the exact code at this ti= me. I can provide more detail if I am unclear. Thank you
>>>
>
> I am not sure to be honest, however it turns out that I can't use = pool.terminate() because pool is defined in main and not accessible under m= y def in which I check for the correct answer.
> --
> https://mail.python.org/mailman/listinfo/python-list

So, the simplest solution to that situation is to have whichever subprocess= that finds the correct answer set a flag which the calling process can che= ck. =A0Depending on your OS, that flag can be anything from setting a lock = to something as simple as creating a file which the calling process periodi= cally wakes up and looks for, maybe just a file in which the subprocess has= written the answer.

Bill

--089e011827aee799eb04ea338e08--