Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.python > #70978
| Path | csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!goblin2!goblin.stu.neva.ru!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail |
|---|---|
| Return-Path | <johan.llewellyn@yahoo.com> |
| X-Original-To | python-list@python.org |
| Delivered-To | python-list@mail.python.org |
| X-Spam-Status | OK 0.001 |
| X-Spam-Evidence | '*H*': 1.00; '*S*': 0.00; 'argument': 0.05; 'filename:fname piece:py': 0.07; 'modify': 0.07; 'none:': 0.07; 'welcome.': 0.07; '__name__': 0.09; 'main()': 0.09; 'skip:% 20': 0.09; 'subject:while': 0.09; 'wrapper': 0.09; 'python': 0.11; 'def': 0.12; 'assume': 0.14; 'wrote': 0.14; "'__main__':": 0.16; 'empty.': 0.16; 'ideally,': 0.16; 'logger': 0.16; 'loop.': 0.16; 'main():': 0.16; 'pulling': 0.16; 'subject:skip:m 10': 0.16; 'thanks,': 0.17; 'module': 0.19; "python's": 0.19; 'examples': 0.20; 'import': 0.22; 'to:name:python-list@python.org': 0.22; 'skip:% 10': 0.24; 'specify': 0.24; 'header:Received:8': 0.24; "i've": 0.25; 'script': 0.25; 'logging': 0.26; 'function': 0.29; 'external': 0.29; "i'm": 0.30; 'publicly': 0.31; 'testing.': 0.31; 'quite': 0.32; 'skip:# 10': 0.33; "i'd": 0.34; 'done.': 0.35; 'accessible': 0.36; 'yield': 0.36; 'doing': 0.36; 'next': 0.36; 'subject:?': 0.36; 'hi,': 0.36; 'ahead': 0.38; 'handle': 0.38; 'to:addr:python-list': 0.38; 'sure': 0.39; 'to:addr:python.org': 0.39; 'skip:p 20': 0.39; 'how': 0.40; 'skip:u 10': 0.60; 'read': 0.60; 'deleting': 0.60; 'most': 0.60; 'times': 0.62; 'high': 0.63; 'connecting': 0.64; 'header:Reply-To:1': 0.67; 'captures': 0.84; 'hours)': 0.84; 'johan': 0.84; 'received:98.138.229': 0.84 |
| X-Yahoo-Newman-Property | ymail-4 |
| X-Yahoo-Newman-Id | 588650.29567.bm@omp1055.mail.ne1.yahoo.com |
| DKIM-Signature | v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1399401940; bh=AqFa0G+s/ecXjuSLkTNVoUlLxJ83kFoFEDkhoc5HeGk=; h=Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=1iZHiYagDEouZHmCZ4BGwzfcvwM+6p2s0Qv2+FOfsT6j9exCoQJxrQFG8UeUFnnt2/Al2BxeqWtz96wWhtcyLYispe7OUdNQGhXQ5yL6X3l/2sbXnPqzy+LzCuGgr0QGYM6gvMihuwxOuV2FGOAnkfg3E5le/FUiwCwXpY25/H4= |
| X-YMail-OSG | WQL2h2AVM1k4h8zAf5._cgFATOaI9Ypf1RCVKBvOCy.PJnW gmFaSfHqDOdjnLf6e4ORCDSpHghsP.EUmI29XMe0zgetmt7Lvx3s4NthzKWY X2sLFYu.Cl.Xj2D1m6Llh5fK61JQ9._tNY2plcXV63Ju1LjXuC0WklSk5NTf bK6BXSoIj6dNP_xR4Y5qFjv1zevVHawOqqVLuoSMY28MNOKgIPI9pP9l2Q_0 UGMl9v3KOUnL8pz8AWdQLh_nM3BV.PQLK4_wVWHmbjKMxaVddnwPPxD8fNul MygVtIX56Y8Vx0gI5Gx8gAT__txZhqFa0fAlHgyX6wMJGH1vmdBt1AImQEyK .Hf89ijdMGx6kaFxySRgytHJbe17VrCcwrQqRc2.W7JBCdSh.htgV7Ar5bBi d0IinY6YU_c59iuEVnKxWwDlvHSMod4czjxRXOQ_aLDB8f_ToVat6A5xvj.r Tb2Kyg7_orbMhSDTSaAkXatTIFnZ9yeM5Mx.Maox4l3jA36jrFNVrWJlB7ne LdZriSkyBaeZkyNqgWxFHQzmPP6Gd2V.KQ84EL903mXTMrPzWCbJQA7FuRGB Yhf8I1tfkPzvoxkiLQEpE |
| X-Rocket-MIMEInfo | 002.001, aGksIEkgYW0gc3RydWdnbGluZyB0byB1bmRlcnN0YW5kIGhvdyB0byBsZXZlcmFnZSBweXRob24ncyBtdWx0aXByb2Nlc3NpbmcgbW9kdWxlIGluIGEgd2hpbGUgbG9vcC7CoCB0aGUgZXhhbXBsZXMgSSBoYXZlIGZvdW5kIHNlZW0gdG8gYXNzdW1lIGl0IGlzIGtub3duIGFoZWFkIG9mIHRpbWUgaG93IG1hbnkgaXRlbXMgbmVlZCB0byBiZSBwcm9jZXNzZWQuCgpzcGVjaWZpY2FsbHksIEkgYW0gcmVhZGluZyBmcm9tIGFuIGV4dGVybmFsIHF1ZXVlLsKgIEkgY3VycmVudGx5IHByb2Nlc3MgaXRlbXMgb25lIGEBMAEBAQE- |
| X-Mailer | YahooMailWebService/0.8.188.663 |
| Date | Tue, 6 May 2014 11:45:40 -0700 (PDT) |
| From | Johan Llewellyn <johan.llewellyn@yahoo.com> |
| Subject | multiprocessing in a while loop? |
| To | "python-list@python.org" <python-list@python.org> |
| MIME-Version | 1.0 |
| Content-Type | multipart/mixed; boundary="1078491548-840768961-1399401940=:38145" |
| X-BeenThere | python-list@python.org |
| X-Mailman-Version | 2.1.15 |
| Precedence | list |
| Reply-To | Johan Llewellyn <johan.llewellyn@yahoo.com> |
| List-Id | General discussion list for the Python programming language <python-list.python.org> |
| List-Unsubscribe | <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe> |
| List-Archive | <http://mail.python.org/pipermail/python-list/> |
| List-Post | <mailto:python-list@python.org> |
| List-Help | <mailto:python-list-request@python.org?subject=help> |
| List-Subscribe | <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe> |
| Newsgroups | comp.lang.python |
| Message-ID | <mailman.9704.1399402109.18130.python-list@python.org> (permalink) |
| Lines | 83 |
| NNTP-Posting-Host | 2001:888:2000:d::a6 |
| X-Trace | 1399402109 news.xs4all.nl 2934 [2001:888:2000:d::a6]:35412 |
| X-Complaints-To | abuse@xs4all.nl |
| Xref | csiph.com comp.lang.python:70978 |
Show key headers only | View raw
[Multipart message — attachments visible in raw view] - view raw
hi, I am struggling to understand how to leverage python's multiprocessing module in a while loop. the examples I have found seem to assume it is known ahead of time how many items need to be processed. specifically, I am reading from an external queue. I currently process items one at a time until the queue is empty. I wrote a wrapper function to handle connecting to the queue, pulling the next message, and deleting it when I am done. ideally, I'd like to modify this wrapper function to take an additional argument (processes) to specify the number of messages to process simultaneously. I've attached a script that captures what I am doing now. unfortunately, the external queue object is not publicly accessible and I'm not quite sure how to set up a local object that would support testing. any suggestions would be most welcome. thanks, Johan
Back to comp.lang.python | Previous | Next — Next in thread | Find similar | Unroll thread
multiprocessing in a while loop? Johan Llewellyn <johan.llewellyn@yahoo.com> - 2014-05-06 11:45 -0700 Re: multiprocessing in a while loop? Jesse Adam <jaahush@gmail.com> - 2014-06-27 16:27 -0700
csiph-web