Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #7984

Re: threading : make stop the caller

Show all headers | View raw

On 6/19/2011 12:03 PM, Chris Angelico wrote:
> On Mon, Jun 20, 2011 at 1:39 AM, Laurent Claessens<moky.math@gmail.com>  wrote:
>> My problem is that when FileToCopyTask raises an error, the program does not
>> stop.
>> In fact when the error is Disk Full, I want to stop the whole program
>> because I know that the next task will fail too.
>
> If you're starting a thread for every file you're copying, you're
> starting a huge number of threads that probably will just end up
> fighting over the disk. To get a reasonably efficient early-abort, I'd
> recommend having a fixed pool of worker threads (say, ten of them),
> and have each thread (a) check if the early-abort flag is set, and
> then (b) start copying the next file in queue. Once the queue's empty
> or the early-abort flag is set, all ten threads will terminate when
> they finish their current transfers.
>
> (The ten threads figure is arbitrary. Optimum value for performance
> will come by adjusting this.)

I an not convinced that much of anything is gained by having multiple 
copying threads. It certainly adds complication -- hence this thread. I 
would just use shutil to copy files or directories of files.

Note that copytree has an option to pass a filter function:
"If ignore is given, it must be a callable that will receive as its 
arguments the directory being visited by copytree(), and a list of its 
contents, as returned by os.listdir(). Since copytree() is called 
recursively, the ignore callable will be called once for each directory 
that is copied. The callable must return a sequence of directory and 
file names relative to the current directory (i.e. a subset of the items 
in its second argument); these names will then be ignored in the copy 
process. ignore_patterns() can be used to create such a callable that 
ignores names based on glob-style patterns."

-- 
Terry Jan Reedy

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

threading : make stop the caller Laurent Claessens <moky.math@gmail.com> - 2011-06-19 16:42 +0200
  Re: threading : make stop the caller Chris Angelico <rosuav@gmail.com> - 2011-06-20 01:19 +1000
    Re: threading : make stop the caller Laurent Claessens <moky.math@gmail.com> - 2011-06-19 17:39 +0200
      Re: threading : make stop the caller Laurent Claessens <moky.math@gmail.com> - 2011-06-19 17:54 +0200
      Re: threading : make stop the caller Chris Angelico <rosuav@gmail.com> - 2011-06-20 02:03 +1000
        Re: threading : make stop the caller Laurent Claessens <moky.math@gmail.com> - 2011-06-19 18:08 +0200
      Re: threading : make stop the caller Terry Reedy <tjreedy@udel.edu> - 2011-06-19 12:38 -0400
        Re: threading : make stop the caller Laurent Claessens <moky.math@gmail.com> - 2011-06-19 18:52 +0200
          Re: threading : make stop the caller Lie Ryan <lie.1296@gmail.com> - 2011-06-20 03:04 +1000
      Re: threading : make stop the caller Terry Reedy <tjreedy@udel.edu> - 2011-06-19 12:58 -0400

csiph-web