Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #92898

Re: Catching exceptions with multi-processing

Path csiph.com!usenet.pasdenom.info!news.redatomik.org!newsfeed.xs4all.nl!newsfeed8.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path <rosuav@gmail.com>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.064
X-Spam-Evidence '*H*': 0.87; '*S*': 0.00; 'cc:addr:python-list': 0.10; 'do,': 0.15; 'dirs,': 0.16; 'folks,': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'threw': 0.16; 'wrote:': 0.16; 'later': 0.16; 'cc:2**0': 0.21; 'cc:addr:python.org': 0.21; 'am,': 0.23; '2015': 0.23; 'sat,': 0.23; 'this:': 0.23; 'header:In-Reply-To:1': 0.24; 'developing': 0.25; 'not,': 0.27; 'message-id:@mail.gmail.com': 0.28; "i'm": 0.29; 'follows': 0.29; 'other,': 0.29; 'relies': 0.29; 'task': 0.31; 'error.': 0.31; 'operations': 0.31; 'run': 0.32; "d'aprano": 0.33; "he's": 0.33; 'is?': 0.33; 'steven': 0.33; 'received:google.com': 0.34; 'done': 0.35; 'instance': 0.35; 'tasks': 0.35; 'but': 0.36; 'tool': 0.36; 'there': 0.36; 'depends': 0.36; 'should': 0.37; 'subject:: ': 0.37; 'say': 0.38; 'doing': 0.38; 'subject:-': 0.39; 'subject:with': 0.40; 'your': 0.60; 'afraid': 0.63; 'here': 0.66; '20,': 0.66; 'apologize': 0.69; 'aiui': 0.84; 'chrisa': 0.84; 'interface:': 0.84; 'to:none': 0.90
DKIM-Signature v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type; bh=YmPLN3z29w6/wg/RgwmYJwKOZhv90k3Zt9H4aLai3KI=; b=wuldgmsFxzb4hIM9/+EC3JCrQIZJXfjXgNDrN5g3MAGkr5NOw3n0dBpR7CklcflxfQ IbqMdDFOcZMgSO7BUuYeRQwzDN2ZGBeXRXuAqCVcR2r64oRfPdknEBxs0jrHMEb6M5RW EcdtKiwiR4AIVaiStV/F8szuRVCqYovzYStOVUoB2o2s/Og6cS62822ZlmILqW0Iukxg yUswuO30vyANvdhj63okp3OC947WHFh8Emq5UCVLmLVPHlIago6iAzAn3BdZoeaoOb6E z2jmABCSaWGzQHz9NjMvqiR2JkbjqTHI7fhpFV7HuT32jwju9hr7eQEMkD1hI932TENC aleg==
MIME-Version 1.0
X-Received by 10.42.43.199 with SMTP id y7mr13970361ice.12.1434747493326; Fri, 19 Jun 2015 13:58:13 -0700 (PDT)
In-Reply-To <55843842$0$1644$c3e8da3$5496439d@news.astraweb.com>
References <mm17as$i8m$1@speranza.aioe.org> <55843842$0$1644$c3e8da3$5496439d@news.astraweb.com>
Date Sat, 20 Jun 2015 06:58:13 +1000
Subject Re: Catching exceptions with multi-processing
From Chris Angelico <rosuav@gmail.com>
Cc "python-list@python.org" <python-list@python.org>
Content-Type text/plain; charset=UTF-8
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.20+
Precedence list
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list/>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups comp.lang.python
Message-ID <mailman.647.1434747496.13271.python-list@python.org> (permalink)
Lines 35
NNTP-Posting-Host 2001:888:2000:d::a6
X-Trace 1434747496 news.xs4all.nl 2914 [2001:888:2000:d::a6]:47599
X-Complaints-To abuse@xs4all.nl
Xref csiph.com comp.lang.python:92898

Show key headers only | View raw

On Sat, Jun 20, 2015 at 1:41 AM, Steven D'Aprano <steve@pearwood.info> wrote:
> On Sat, 20 Jun 2015 12:01 am, Fabien wrote:
>
>> Folks,
>>
>> I am developing a tool which works on individual entities (glaciers) and
>> do a lot of operations on them. There are many tasks to do, one after
>> each other, and each task follows the same interface:
>
> I'm afraid your description is contradictory. Here you say the tasks run one
> after another, but then you say:
>
>> This way, the tasks can be run in parallel very easily:
>
> and then later still you contradict this:
>
>> Also, the task2 should not be run if task1 threw an error.
>
>
> If task2 relies on task1, then you *cannot* run them in parallel. You have
> to run them one after the other, sequentially.

AIUI what he's doing is all the subparts of task1 in parallel, then
all the subparts of task2:

pool.map(task1, dirs, chunksize=1)
pool.map(task2, dirs, chunksize=1)
pool.map(task3, dirs, chunksize=1)

task1 can be done on all of dirs in parallel, as no instance of task1
depends on any other instance of task1; but task2 should be started
only if all task1s finish successfully. OP, is this how it is? If not,
I apologize for the noise.

ChrisA

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Catching exceptions with multi-processing Fabien <fabien.maussion@gmail.com> - 2015-06-19 16:01 +0200
  Re: Catching exceptions with multi-processing Andres Riancho <andres.riancho@gmail.com> - 2015-06-19 11:25 -0300
    Re: Catching exceptions with multi-processing Fabien <fabien.maussion@gmail.com> - 2015-06-19 18:16 +0200
      Re: Catching exceptions with multi-processing Cameron Simpson <cs@zip.com.au> - 2015-06-20 13:14 +1000
        Re: Catching exceptions with multi-processing Fabien <fabien.maussion@gmail.com> - 2015-06-20 11:03 +0200
  Re: Catching exceptions with multi-processing Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2015-06-19 16:01 +0100
  Re: Catching exceptions with multi-processing Steven D'Aprano <steve@pearwood.info> - 2015-06-20 01:41 +1000
    Re: Catching exceptions with multi-processing Fabien <fabien.maussion@gmail.com> - 2015-06-19 18:07 +0200
    Re: Catching exceptions with multi-processing Chris Angelico <rosuav@gmail.com> - 2015-06-20 06:58 +1000
      Re: Catching exceptions with multi-processing Fabien <fabien.maussion@gmail.com> - 2015-06-20 11:01 +0200
  Re: Catching exceptions with multi-processing Paul Rubin <no.email@nospam.invalid> - 2015-06-21 13:27 -0700

csiph-web