Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.python > #92898
| References | <mm17as$i8m$1@speranza.aioe.org> <55843842$0$1644$c3e8da3$5496439d@news.astraweb.com> |
|---|---|
| Date | 2015-06-20 06:58 +1000 |
| Subject | Re: Catching exceptions with multi-processing |
| From | Chris Angelico <rosuav@gmail.com> |
| Newsgroups | comp.lang.python |
| Message-ID | <mailman.647.1434747496.13271.python-list@python.org> (permalink) |
On Sat, Jun 20, 2015 at 1:41 AM, Steven D'Aprano <steve@pearwood.info> wrote: > On Sat, 20 Jun 2015 12:01 am, Fabien wrote: > >> Folks, >> >> I am developing a tool which works on individual entities (glaciers) and >> do a lot of operations on them. There are many tasks to do, one after >> each other, and each task follows the same interface: > > I'm afraid your description is contradictory. Here you say the tasks run one > after another, but then you say: > >> This way, the tasks can be run in parallel very easily: > > and then later still you contradict this: > >> Also, the task2 should not be run if task1 threw an error. > > > If task2 relies on task1, then you *cannot* run them in parallel. You have > to run them one after the other, sequentially. AIUI what he's doing is all the subparts of task1 in parallel, then all the subparts of task2: pool.map(task1, dirs, chunksize=1) pool.map(task2, dirs, chunksize=1) pool.map(task3, dirs, chunksize=1) task1 can be done on all of dirs in parallel, as no instance of task1 depends on any other instance of task1; but task2 should be started only if all task1s finish successfully. OP, is this how it is? If not, I apologize for the noise. ChrisA
Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
Catching exceptions with multi-processing Fabien <fabien.maussion@gmail.com> - 2015-06-19 16:01 +0200
Re: Catching exceptions with multi-processing Andres Riancho <andres.riancho@gmail.com> - 2015-06-19 11:25 -0300
Re: Catching exceptions with multi-processing Fabien <fabien.maussion@gmail.com> - 2015-06-19 18:16 +0200
Re: Catching exceptions with multi-processing Cameron Simpson <cs@zip.com.au> - 2015-06-20 13:14 +1000
Re: Catching exceptions with multi-processing Fabien <fabien.maussion@gmail.com> - 2015-06-20 11:03 +0200
Re: Catching exceptions with multi-processing Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2015-06-19 16:01 +0100
Re: Catching exceptions with multi-processing Steven D'Aprano <steve@pearwood.info> - 2015-06-20 01:41 +1000
Re: Catching exceptions with multi-processing Fabien <fabien.maussion@gmail.com> - 2015-06-19 18:07 +0200
Re: Catching exceptions with multi-processing Chris Angelico <rosuav@gmail.com> - 2015-06-20 06:58 +1000
Re: Catching exceptions with multi-processing Fabien <fabien.maussion@gmail.com> - 2015-06-20 11:01 +0200
Re: Catching exceptions with multi-processing Paul Rubin <no.email@nospam.invalid> - 2015-06-21 13:27 -0700
csiph-web