Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #7977

Re: threading : make stop the caller

Path csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!selfless.tophat.at!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path <rosuav@gmail.com>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.012
X-Spam-Evidence '*H*': 0.98; '*S*': 0.00; 'raises': 0.07; 'adjusting': 0.09; 'am,': 0.14; 'received:209.85.214.174': 0.14; 'received:mail-iw0-f174.google.com': 0.14; 'wrote:': 0.14; 'arbitrary.': 0.16; 'disk.': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'set,': 0.16; 'stop.': 0.16; 'subject:stop': 0.16; 'mon,': 0.17; 'header:In-Reply-To:1': 0.21; 'figure': 0.21; '(b)': 0.23; 'subject: : ': 0.26; 'fixed': 0.27; 'message-id:@mail.gmail.com': 0.28; '(the': 0.28; 'problem': 0.28; 'received:209.85.214': 0.28; 'disk': 0.29; 'fact': 0.30; 'terminate': 0.30; 'threads': 0.30; 'worker': 0.30; 'does': 0.33; 'to:addr:python-list': 0.33; 'copying': 0.33; 'starting': 0.33; 'error': 0.33; 'file': 0.34; 'fail': 0.34; '(a)': 0.35; 'flag': 0.35; 'probably': 0.36; 'too.': 0.37; 'received:google.com': 0.37; 'received:209.85': 0.37; '20,': 0.37; 'thread': 0.37; 'subject:: ': 0.38; "i'd": 0.39; 'received:209': 0.39; 'empty': 0.39; 'to:addr:python.org': 0.39; 'current': 0.40; 'stop': 0.62; 'huge': 0.62; 'fighting': 0.67; 'subject:make': 0.73; 'optimum': 0.91
DKIM-Signature v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=4j+HU5sDmYK3isrM9Bp9arWNMEhsEKNjffGqgthmC+o=; b=gu4iWrBHHHbysGMsSSLSAazzN9kWpLHeYbBc1gxX8+LuhYWtrbb9/8OMfJU8wxIDFD C2Q51Bn6GhOmb3XpOkDjCWf41MIIaEOlsCi0+d4oqRhShKvq/UjoW2vFomasHXrhEVoD +OFl5PQZQ7TdOYHy8/biv/rUJBBwVe0OqYmTY=
DomainKey-Signature a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=so1I9uzaKZswtG9NfWd7F7cWBVJDqLXIFasc7S21CUYcUKaSmcofs4zzZ381QrSoXw aePTWmYm2BWcdt2NxSu8k/AYONVcqW3BpJo0l8ooyCSK9a3ftr/sqaqQ6EHtYxTl0BRj Ry1N/ZY4S+0rhyr6DUywQIhQmA9m4LG2HoA/U=
MIME-Version 1.0
In-Reply-To <itl56g$aut$1@news.univ-fcomte.fr>
References <itl1sc$ap0$1@news.univ-fcomte.fr> <mailman.155.1308496777.1164.python-list@python.org> <itl56g$aut$1@news.univ-fcomte.fr>
Date Mon, 20 Jun 2011 02:03:11 +1000
Subject Re: threading : make stop the caller
From Chris Angelico <rosuav@gmail.com>
To python-list@python.org
Content-Type text/plain; charset=ISO-8859-1
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.12
Precedence list
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups comp.lang.python
Message-ID <mailman.158.1308499394.1164.python-list@python.org> (permalink)
Lines 19
NNTP-Posting-Host 82.94.164.166
X-Trace 1308499395 news.xs4all.nl 49178 [::ffff:82.94.164.166]:49526
X-Complaints-To abuse@xs4all.nl
Xref x330-a1.tempe.blueboxinc.net comp.lang.python:7977

Show key headers only | View raw

On Mon, Jun 20, 2011 at 1:39 AM, Laurent Claessens <moky.math@gmail.com> wrote:
> My problem is that when FileToCopyTask raises an error, the program does not
> stop.
> In fact when the error is Disk Full, I want to stop the whole program
> because I know that the next task will fail too.

If you're starting a thread for every file you're copying, you're
starting a huge number of threads that probably will just end up
fighting over the disk. To get a reasonably efficient early-abort, I'd
recommend having a fixed pool of worker threads (say, ten of them),
and have each thread (a) check if the early-abort flag is set, and
then (b) start copying the next file in queue. Once the queue's empty
or the early-abort flag is set, all ten threads will terminate when
they finish their current transfers.

(The ten threads figure is arbitrary. Optimum value for performance
will come by adjusting this.)

ChrisA

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

threading : make stop the caller Laurent Claessens <moky.math@gmail.com> - 2011-06-19 16:42 +0200
  Re: threading : make stop the caller Chris Angelico <rosuav@gmail.com> - 2011-06-20 01:19 +1000
    Re: threading : make stop the caller Laurent Claessens <moky.math@gmail.com> - 2011-06-19 17:39 +0200
      Re: threading : make stop the caller Laurent Claessens <moky.math@gmail.com> - 2011-06-19 17:54 +0200
      Re: threading : make stop the caller Chris Angelico <rosuav@gmail.com> - 2011-06-20 02:03 +1000
        Re: threading : make stop the caller Laurent Claessens <moky.math@gmail.com> - 2011-06-19 18:08 +0200
      Re: threading : make stop the caller Terry Reedy <tjreedy@udel.edu> - 2011-06-19 12:38 -0400
        Re: threading : make stop the caller Laurent Claessens <moky.math@gmail.com> - 2011-06-19 18:52 +0200
          Re: threading : make stop the caller Lie Ryan <lie.1296@gmail.com> - 2011-06-20 03:04 +1000
      Re: threading : make stop the caller Terry Reedy <tjreedy@udel.edu> - 2011-06-19 12:58 -0400

csiph-web