Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!goblin2!goblin.stu.neva.ru!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.001 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'argument': 0.05; 'filename:fname piece:py': 0.07; 'modify': 0.07; 'none:': 0.07; 'welcome.': 0.07; '__name__': 0.09; 'main()': 0.09; 'skip:% 20': 0.09; 'subject:while': 0.09; 'wrapper': 0.09; 'python': 0.11; 'def': 0.12; 'assume': 0.14; 'wrote': 0.14; "'__main__':": 0.16; 'empty.': 0.16; 'ideally,': 0.16; 'logger': 0.16; 'loop.': 0.16; 'main():': 0.16; 'pulling': 0.16; 'subject:skip:m 10': 0.16; 'thanks,': 0.17; 'module': 0.19; "python's": 0.19; 'examples': 0.20; 'import': 0.22; 'to:name:python-list@python.org': 0.22; 'skip:% 10': 0.24; 'specify': 0.24; 'header:Received:8': 0.24; "i've": 0.25; 'script': 0.25; 'logging': 0.26; 'function': 0.29; 'external': 0.29; "i'm": 0.30; 'publicly': 0.31; 'testing.': 0.31; 'quite': 0.32; 'skip:# 10': 0.33; "i'd": 0.34; 'done.': 0.35; 'accessible': 0.36; 'yield': 0.36; 'doing': 0.36; 'next': 0.36; 'subject:?': 0.36; 'hi,': 0.36; 'ahead': 0.38; 'handle': 0.38; 'to:addr:python-list': 0.38; 'sure': 0.39; 'to:addr:python.org': 0.39; 'skip:p 20': 0.39; 'how': 0.40; 'skip:u 10': 0.60; 'read': 0.60; 'deleting': 0.60; 'most': 0.60; 'times': 0.62; 'high': 0.63; 'connecting': 0.64; 'header:Reply-To:1': 0.67; 'captures': 0.84; 'hours)': 0.84; 'johan': 0.84; 'received:98.138.229': 0.84 X-Yahoo-Newman-Property: ymail-4 X-Yahoo-Newman-Id: 588650.29567.bm@omp1055.mail.ne1.yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1399401940; bh=AqFa0G+s/ecXjuSLkTNVoUlLxJ83kFoFEDkhoc5HeGk=; h=Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=1iZHiYagDEouZHmCZ4BGwzfcvwM+6p2s0Qv2+FOfsT6j9exCoQJxrQFG8UeUFnnt2/Al2BxeqWtz96wWhtcyLYispe7OUdNQGhXQ5yL6X3l/2sbXnPqzy+LzCuGgr0QGYM6gvMihuwxOuV2FGOAnkfg3E5le/FUiwCwXpY25/H4= X-YMail-OSG: WQL2h2AVM1k4h8zAf5._cgFATOaI9Ypf1RCVKBvOCy.PJnW gmFaSfHqDOdjnLf6e4ORCDSpHghsP.EUmI29XMe0zgetmt7Lvx3s4NthzKWY X2sLFYu.Cl.Xj2D1m6Llh5fK61JQ9._tNY2plcXV63Ju1LjXuC0WklSk5NTf bK6BXSoIj6dNP_xR4Y5qFjv1zevVHawOqqVLuoSMY28MNOKgIPI9pP9l2Q_0 UGMl9v3KOUnL8pz8AWdQLh_nM3BV.PQLK4_wVWHmbjKMxaVddnwPPxD8fNul MygVtIX56Y8Vx0gI5Gx8gAT__txZhqFa0fAlHgyX6wMJGH1vmdBt1AImQEyK .Hf89ijdMGx6kaFxySRgytHJbe17VrCcwrQqRc2.W7JBCdSh.htgV7Ar5bBi d0IinY6YU_c59iuEVnKxWwDlvHSMod4czjxRXOQ_aLDB8f_ToVat6A5xvj.r Tb2Kyg7_orbMhSDTSaAkXatTIFnZ9yeM5Mx.Maox4l3jA36jrFNVrWJlB7ne LdZriSkyBaeZkyNqgWxFHQzmPP6Gd2V.KQ84EL903mXTMrPzWCbJQA7FuRGB Yhf8I1tfkPzvoxkiLQEpE X-Rocket-MIMEInfo: 002.001, aGksIEkgYW0gc3RydWdnbGluZyB0byB1bmRlcnN0YW5kIGhvdyB0byBsZXZlcmFnZSBweXRob24ncyBtdWx0aXByb2Nlc3NpbmcgbW9kdWxlIGluIGEgd2hpbGUgbG9vcC7CoCB0aGUgZXhhbXBsZXMgSSBoYXZlIGZvdW5kIHNlZW0gdG8gYXNzdW1lIGl0IGlzIGtub3duIGFoZWFkIG9mIHRpbWUgaG93IG1hbnkgaXRlbXMgbmVlZCB0byBiZSBwcm9jZXNzZWQuCgpzcGVjaWZpY2FsbHksIEkgYW0gcmVhZGluZyBmcm9tIGFuIGV4dGVybmFsIHF1ZXVlLsKgIEkgY3VycmVudGx5IHByb2Nlc3MgaXRlbXMgb25lIGEBMAEBAQE- X-Mailer: YahooMailWebService/0.8.188.663 Date: Tue, 6 May 2014 11:45:40 -0700 (PDT) From: Johan Llewellyn Subject: multiprocessing in a while loop? To: "python-list@python.org" MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="1078491548-840768961-1399401940=:38145" X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Johan Llewellyn List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 83 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1399402109 news.xs4all.nl 2934 [2001:888:2000:d::a6]:35412 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:70978 --1078491548-840768961-1399401940=:38145 Content-Type: multipart/alternative; boundary="1078491548-2104499605-1399401940=:38145" --1078491548-2104499605-1399401940=:38145 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable hi, I am struggling to understand how to leverage python's multiprocessing = module in a while loop.=A0 the examples I have found seem to assume it is k= nown ahead of time how many items need to be processed.=0A=0Aspecifically, = I am reading from an external queue.=A0 I currently process items one at a = time until the queue is empty.=A0 I wrote a wrapper function to handle conn= ecting to the queue, pulling the next message, and deleting it when I am do= ne.=A0 ideally, I'd like to modify this wrapper function to take an additio= nal argument (processes) to specify the number of messages to process simul= taneously.=0A=0AI've attached a script that captures what I am doing now.= =A0 unfortunately, the external queue object is not publicly accessible and= I'm not quite sure how to set up a local object that would support testing= .=A0 any suggestions would be most welcome.=0A=0A=0Athanks,=0AJohan=0A --1078491548-2104499605-1399401940=:38145 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable
hi, I am struggling to understand = how to leverage python's multiprocessing module in a while loop.  the = examples I have found seem to assume it is known ahead of time how many ite= ms need to be processed.

specifically,= I am reading from an external queue.  I currently process items one a= t a time until the queue is empty.  I wrote a wrapper function to hand= le connecting to the queue, pulling the next message, and deleting it when = I am done.  ideally, I'd like to modify this wrapper function to take = an additional argument (processes) to specify the number of messages to proce= ss simultaneously.
I've attached a scri= pt that captures what I am doing now.  unfortunately, the external que= ue object is not publicly accessible and I'm not quite sure how to set up a= local object that would support testing.  any suggestions would be mo= st welcome.

thank= s,
Johan
--1078491548-2104499605-1399401940=:38145-- --1078491548-840768961-1399401940=:38145 Content-Type: text/x-python; name="mp_test.py" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="mp_test.py" IyEvdXNyL2Jpbi9lbnYgcHl0aG9uCgppbXBvcnQgYm90bwppbXBvcnQgbG9n Z2luZwoKTE9HX0ZNVCA9ICclKGFzY3RpbWUpcyAtICUobGV2ZWxuYW1lKXMg LSAlKG1vZHVsZSlzLiUoZnVuY05hbWUpcyAtICUobWVzc2FnZSlzJwpRVUVV RSA9ICdteS1xdWV1ZScKCmxvZ2dlciA9IGxvZ2dpbmcuZ2V0TG9nZ2VyKF9f bmFtZV9fKQoKCiMgcmVhZCBmcm9tIGFuIGV4dGVybmFsIHF1ZXVlOyBpdGVt cyBtYXkgYmUgYWRkZWQgZHVyaW5nIHByb2Nlc3NpbmcKZGVmIGNvbnN1bWVf cXVldWUocXVldWVfbmFtZSk6CiAgY29ubiA9IGJvdG8uY29ubmVjdF9zcXMo KQogIHEgPSBjb25uLmdldF9xdWV1ZShxdWV1ZV9uYW1lKQogIG0gPSBxLnJl YWQoKQogIHdoaWxlIG0gaXMgbm90IE5vbmU6CiAgICB5aWVsZCBtCiAgICBx LmRlbGV0ZV9tZXNzYWdlKG0pCiAgICBsb2dnZXIuZGVidWcoJ21lc3NhZ2Ug ZGVsZXRlZCcpCiAgICBtID0gcS5yZWFkKCkKCgojIGhpZ2ggdmFyaWFiaWxp dHkgaW4gbWVzc2FnZSBwcm9jZXNzaW5nIHRpbWVzIChzZWNvbmRzIC0gaG91 cnMpCmRlZiBoYW5kbGVfbWVzc2FnZShtZXNzYWdlKQogIHMgPSBtZXNzYWdl LmdldF9ib2R5KCkKICBsb2dnZXIuaW5mbyhzKQoKCmRlZiBtYWluKCk6CiAg Zm9yIG1lc3NhZ2UgaW4gY29uc3VtZV9xdWV1ZShRVUVVRSk6CiAgICBoYW5k bGVfbWVzc2FnZShtZXNzYWdlKQoKCmlmIF9fbmFtZV9fID09ICdfX21haW5f Xyc6CiAgbG9nZ2luZy5iYXNpY0NvbmZpZyhsZXZlbD1sb2dnaW5nLkRFQlVH LCBmb3JtYXQ9TE9HX0ZNVCkKICBtYWluKCkKCg== --1078491548-840768961-1399401940=:38145--