Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!ecngs!feeder2.ecngs.de!newsfeeder.ewetel.de!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.002 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'python.': 0.02; 'mrab': 0.05; '__name__': 0.09; 'friday,': 0.09; 'snippet': 0.09; 'subject:process': 0.09; 'whichever': 0.09; 'python': 0.11; 'def': 0.12; '#this': 0.16; '>>': 0.16; "'__main__':": 0.16; 'bsd': 0.16; 'check.': 0.16; 'finds': 0.16; 'for,': 0.16; 'i.e.,': 0.16; 'inputs': 0.16; 'letting': 0.16; 'master.': 0.16; 'received:mac.com': 0.16; 'simplest': 0.16; 'sockets': 0.16; 'specific,': 0.16; 'subject: \n ': 0.16; 'subject:skip:m 10': 0.16; 'sys.exit(0)': 0.16; 'tried:': 0.16; 'try?': 0.16; 'typo': 0.16; 'wrote:': 0.18; 'library': 0.18; 'variable': 0.18; 'received:10.0.1': 0.19; 'solution.': 0.20; 'written': 0.21; '>>>': 0.22; 'example': 0.22; 'import': 0.22; 'email addr:gmail.com>': 0.22; 'to:name:python-list@python.org': 0.22; 'print': 0.22; 'creating': 0.23; '>>>': 0.24; 'looks': 0.24; 'cc:2**0': 0.24; "i've": 0.25; '>': 0.26; 'hall': 0.26; 'defined': 0.27; 'to:2**1': 0.27; 'correct': 0.29; 'am,': 0.29; 'subject:list': 0.30; 'url:mailman': 0.30; 'code': 0.31; 'os,': 0.31; 'prints': 0.31; 'ray': 0.31; 'file': 0.32; 'url:python': 0.33; 'style': 0.33; 'maybe': 0.34; 'subject:the': 0.34; 'subject: (': 0.35; 'basic': 0.35; "can't": 0.35; 'problem.': 0.35; 'something': 0.35; 'objects': 0.35; 'but': 0.35; 'there': 0.35; 'accessible': 0.36; 'disk': 0.36; 'url:listinfo': 0.36; 'charset :us-ascii': 0.36; 'received:10.0': 0.36; 'url:org': 0.36; 'detail': 0.37; 'so,': 0.37; 'received:10': 0.37; 'thank': 0.38; 'nov': 0.38; 'received:17': 0.38; 'to:addr:python-list': 0.38; 'anything': 0.39; 'bill': 0.39; 'sure': 0.39; 'to:addr:python.org': 0.39; 'url:mail': 0.40; 'solve': 0.60; 'simple': 0.61; 'further': 0.61; 'complete': 0.62; 'provide': 0.64; 'more': 0.64; 'different': 0.65; 'situation': 0.65; 'to:addr:gmail.com': 0.65; 'here': 0.66; 'hang': 0.67; 'response.': 0.68; 'answer.': 0.68; 'periodically': 0.68; 'evaluate': 0.72; '11:44': 0.84; 'header:In-reply-to:1': 0.84; 'wakes': 0.84; '2013,': 0.91; 'processes,': 0.91; 'subject:answer': 0.95; '2013': 0.98 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8794,1.0.431,0.0.0000 definitions=2013-11-02_03:2013-11-02,2013-11-02,1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=7.0.1-1308280000 definitions=main-1311020307 Content-type: multipart/alternative; boundary="Apple-Mail=_EBCEF874-6E9A-4302-954D-FAAA074826C9" MIME-version: 1.0 (Mac OS X Mail 6.6 \(1510\)) Subject: Re: multiprocessing: child process race to answer (forgot to Cc: the list) From: William Ray Wing In-reply-to: Date: Sat, 02 Nov 2013 23:07:42 -0400 References: <6276BD6E-B0A2-4443-9274-E27BBC3C0152@mac.com> To: Sherard Hall , "python-list@python.org" X-Mailer: Apple Mail (2.1510) Cc: William Ray Wing X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 147 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1383448097 news.xs4all.nl 15892 [2001:888:2000:d::a6]:54187 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:58357 --Apple-Mail=_EBCEF874-6E9A-4302-954D-FAAA074826C9 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On Nov 2, 2013, at 11:44 AM, Sherard Hall wrote: > Thank you for the response. Processing time is very important so I = suspect having to write to disk will take more time than letting the = other processes complete without finding the answer. So I did some = profiling one process finds the answer in about 250ms, but since I can't = stop the other processes, it takes about 800ms before I can use the = answer. Do you recommend a global variable flag? Any other suggestions? >=20 > On Nov 2, 2013 8:17 AM, "William Ray Wing" wrote: > On Nov 2, 2013, at 1:03 AM, smhall05 wrote: >=20 > > On Friday, November 1, 2013 10:52:40 PM UTC-4, MRAB wrote: > >> On 02/11/2013 02:35, smhall05 wrote: > >> > >>> I am using a basic multiprocessing snippet I found: > >>> > >>> #----------------------------------------------------- > >>> from multiprocessing import Pool > >>> > >>> def f(x): > >>> return x*x > >>> > >>> if __name__ =3D=3D '__main__': > >>> pool =3D Pool(processes=3D4) # start 4 worker = processes > >>> result =3D pool.apply_async(f, [10]) # evaluate "f(10)" = asynchronously > >>> print result.get(timeout=3D1) > >>> print pool.map(f, range(10)) # prints "[0, 1, 4,..., = 81]" > >>> #--------------------------------------------------------- > >>> > >>> I am using this code to have each process go off and solve the = same problem, just with different inputs to the problem. I need to be = able to kill all processes once 1 of n processes has come up with the = solution. There will only be one answer. > >>> > >>> I have tried: > >>> > >>> sys.exit(0) #this causes the program to hang > >>> pool.close() > >>> pool.terminate > >>> > >> > >> Did you actually mean "pool.terminate", or is that a typo for > >> > >> "pool.terminate()"? > >> > >>> These still allow further processing before the program = terminates. What else can I try? I am not able to share the exact code = at this time. I can provide more detail if I am unclear. Thank you > >>> > > > > I am not sure to be honest, however it turns out that I can't use = pool.terminate() because pool is defined in main and not accessible = under my def in which I check for the correct answer. > > -- > > https://mail.python.org/mailman/listinfo/python-list >=20 > So, the simplest solution to that situation is to have whichever = subprocess that finds the correct answer set a flag which the calling = process can check. Depending on your OS, that flag can be anything from = setting a lock to something as simple as creating a file which the = calling process periodically wakes up and looks for, maybe just a file = in which the subprocess has written the answer. >=20 > Bill >=20 > --=20 > https://mail.python.org/mailman/listinfo/python-list Well, the multiprocessing library provides listeners and clients that = wrap BSD style sockets and allow you to send (push) arbitrary python = objects to a listener, i.e., the master. There might be something = better that was OS specific, but this will keep it pure python. I've = not tested it, but there is a simple example here on Stackoverflow:=20 = http://stackoverflow.com/questions/6920858/interprocess-communication-in-p= ython -Bill= --Apple-Mail=_EBCEF874-6E9A-4302-954D-FAAA074826C9 Content-Transfer-Encoding: 7bit Content-Type: text/html; charset=us-ascii
On Nov 2, 2013, at 11:44 AM, Sherard Hall <smhall05@gmail.com> wrote:

Thank you for the response. Processing time is very important so I suspect having to write to disk will take more time than letting the other processes complete without finding the answer. So I did some profiling one process finds the answer in about 250ms, but since I can't stop the other processes, it takes about 800ms before I can use the answer.  Do you recommend a global variable flag? Any other suggestions?

On Nov 2, 2013 8:17 AM, "William Ray Wing" <wrw@mac.com> wrote:
On Nov 2, 2013, at 1:03 AM, smhall05 <smhall05@gmail.com> wrote:

> On Friday, November 1, 2013 10:52:40 PM UTC-4, MRAB wrote:
>> On 02/11/2013 02:35, smhall05 wrote:
>>
>>> I am using a basic multiprocessing snippet I found:
>>>
>>> #-----------------------------------------------------
>>> from multiprocessing import Pool
>>>
>>> def  f(x):
>>>     return x*x
>>>
>>> if __name__ == '__main__':
>>>     pool = Pool(processes=4)              # start 4 worker processes
>>>     result = pool.apply_async(f, [10])    # evaluate "f(10)" asynchronously
>>>     print result.get(timeout=1)
>>>     print pool.map(f, range(10))          # prints "[0, 1, 4,..., 81]"
>>> #---------------------------------------------------------
>>>
>>> I am using this code to have each process go off and solve the same problem, just with different inputs to the problem. I need to be able to kill all processes once 1 of n processes has come up with the solution. There will only be one answer.
>>>
>>> I have tried:
>>>
>>> sys.exit(0) #this causes the program to hang
>>> pool.close()
>>> pool.terminate
>>>
>>
>> Did you actually mean "pool.terminate", or is that a typo for
>>
>> "pool.terminate()"?
>>
>>> These still allow further processing before the program terminates. What else can I try? I am not able to share the exact code at this time. I can provide more detail if I am unclear. Thank you
>>>
>
> I am not sure to be honest, however it turns out that I can't use pool.terminate() because pool is defined in main and not accessible under my def in which I check for the correct answer.
> --
> https://mail.python.org/mailman/listinfo/python-list

So, the simplest solution to that situation is to have whichever subprocess that finds the correct answer set a flag which the calling process can check.  Depending on your OS, that flag can be anything from setting a lock to something as simple as creating a file which the calling process periodically wakes up and looks for, maybe just a file in which the subprocess has written the answer.

Bill

--
https://mail.python.org/mailman/listinfo/python-list

Well, the multiprocessing library provides listeners and clients that wrap BSD style sockets and allow you to send (push) arbitrary python objects to a listener, i.e., the master.  There might be something better that was OS specific, but this will keep it pure python.  I've not tested it, but there is a simple example here on Stackoverflow: 


-Bill
--Apple-Mail=_EBCEF874-6E9A-4302-954D-FAAA074826C9--