Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #7965 > unrolled thread

threading : make stop the caller

Back to article view | Back to comp.lang.python

Contents

  threading : make stop the caller Laurent Claessens <moky.math@gmail.com> - 2011-06-19 16:42 +0200
    Re: threading : make stop the caller Chris Angelico <rosuav@gmail.com> - 2011-06-20 01:19 +1000
      Re: threading : make stop the caller Laurent Claessens <moky.math@gmail.com> - 2011-06-19 17:39 +0200
        Re: threading : make stop the caller Laurent Claessens <moky.math@gmail.com> - 2011-06-19 17:54 +0200
        Re: threading : make stop the caller Chris Angelico <rosuav@gmail.com> - 2011-06-20 02:03 +1000
          Re: threading : make stop the caller Laurent Claessens <moky.math@gmail.com> - 2011-06-19 18:08 +0200
        Re: threading : make stop the caller Terry Reedy <tjreedy@udel.edu> - 2011-06-19 12:38 -0400
          Re: threading : make stop the caller Laurent Claessens <moky.math@gmail.com> - 2011-06-19 18:52 +0200
            Re: threading : make stop the caller Lie Ryan <lie.1296@gmail.com> - 2011-06-20 03:04 +1000
        Re: threading : make stop the caller Terry Reedy <tjreedy@udel.edu> - 2011-06-19 12:58 -0400

#7965 — threading : make stop the caller

    Hello


I've a list of tasks to perform. Each of them is a threading.Thread. 
Basically I have :

while task_list :
    task = task_list[0]
    task.run()
    task_list.remove(task)


Now I want, in some circumstance to raise errors that make the loop stop.

In order IOError to make stop the loop, I tried this :

no_error = True
while task_list and no_error:
    task = task_list[0]
    try :
       task.run()
    except IOError :
       no_error = False
    task_list.remove(task)

But it does not work (I think the exception is raised too late).

I prefer that the taske are not aware to be included in a loop, but I 
can make them raise personnal exceptions.
How can I do ?


Thanks
Laurent

[toc] | [next] | [standalone]

#7970

On Mon, Jun 20, 2011 at 12:42 AM, Laurent Claessens <moky.math@gmail.com> wrote:
>   Hello
>
>
> I've a list of tasks to perform. Each of them is a threading.Thread.
> Basically I have :
>
> while task_list :
>   task = task_list[0]
>   task.run()
>   task_list.remove(task)

I'm not understanding what you're doing with threads here. Are you
using threading.Thread but then calling its run() method
synchronously?

Normally threads are used for asynchronous operations. You would then
use the start() method to spin the thread off; it will return almost
immediately, and the thread will run to completion in parallel with
you. But then you can't halt the main loop, because it will have
already finished by the time you detect the IOError (starting a bunch
of threads is pretty quick). On the other hand, the code you're
showing seems to simply call each thread's run() method one by one,
which should propagate any exceptions in the same way that function
calls usually do.

Can you share the code for one of the tasks, and show what happens
when it raises an exception?

Chris Angelico

[toc] | [prev] | [next] | [standalone]

#7972

Le 19/06/2011 17:19, Chris Angelico a écrit :
> On Mon, Jun 20, 2011 at 12:42 AM, Laurent Claessens<moky.math@gmail.com>  wrote:
>>     Hello
>>
>>
>>  I've a list of tasks to perform. Each of them is a threading.Thread.
>>  Basically I have :
>>
>>  while task_list :
>>     task = task_list[0]
>>     task.run()
>>     task_list.remove(task)
>
> I'm not understanding what you're doing with threads here. Are you
> using threading.Thread but then calling its run() method
> synchronously?

Woops yes. I missprinted my example. I was using task.start() of course.

The aim is to copy the content of a repertory (with some conditions on 
each file, so I cannot use shutils or something).

I've one thread that runs over the repertory and fill the list 
'task_list' with taskes from the following class :


class FileCopyTask(threading.Thread):
     def __init__(self,source,destination,old_version):
         threading.Thread.__init__(self)
         self.source = source
         self.destination = destination
     def run(self):
         try :
             shutil.copy(self.source,self.destination)
         except (IOError,OSError),data :
             <WHAT TO PUT HERE ??>
         else :
             print "file copied"


In the same time I've a thread that read the list and perform the 
operations:

def run():
    while task_list :
       task = task_list[0]
       task_list.remove(task)
       task.start()


My problem is that when FileToCopyTask raises an error, the program does 
not stop.
In fact when the error is Disk Full, I want to stop the whole program 
because I know that the next task will fail too.

thanks for any help
Laurent

[toc] | [prev] | [next] | [standalone]

#7975

I read the library documentation. I think that if I get a trick to kill 
a thread, then I'm done.

Is there a way ?

Laurent


Le 19/06/2011 17:39, Laurent Claessens a écrit :
> Le 19/06/2011 17:19, Chris Angelico a écrit :
>>  On Mon, Jun 20, 2011 at 12:42 AM, Laurent Claessens<moky.math@gmail.com>   wrote:
>>>      Hello
>>>
>>>
>>>   I've a list of tasks to perform. Each of them is a threading.Thread.
>>>   Basically I have :

>

[toc] | [prev] | [next] | [standalone]

#7977

On Mon, Jun 20, 2011 at 1:39 AM, Laurent Claessens <moky.math@gmail.com> wrote:
> My problem is that when FileToCopyTask raises an error, the program does not
> stop.
> In fact when the error is Disk Full, I want to stop the whole program
> because I know that the next task will fail too.

If you're starting a thread for every file you're copying, you're
starting a huge number of threads that probably will just end up
fighting over the disk. To get a reasonably efficient early-abort, I'd
recommend having a fixed pool of worker threads (say, ten of them),
and have each thread (a) check if the early-abort flag is set, and
then (b) start copying the next file in queue. Once the queue's empty
or the early-abort flag is set, all ten threads will terminate when
they finish their current transfers.

(The ten threads figure is arbitrary. Optimum value for performance
will come by adjusting this.)

ChrisA

[toc] | [prev] | [next] | [standalone]

#7978

Le 19/06/2011 18:03, Chris Angelico a écrit :
> On Mon, Jun 20, 2011 at 1:39 AM, Laurent Claessens<moky.math@gmail.com>  wrote:
>>  My problem is that when FileToCopyTask raises an error, the program does not
>>  stop.
>>  In fact when the error is Disk Full, I want to stop the whole program
>>  because I know that the next task will fail too.
>
> If you're starting a thread for every file you're copying, you're
> starting a huge number of threads that probably will just end up
> fighting over the disk. To get a reasonably efficient early-abort, I'd
> recommend having a fixed pool of worker threads (say, ten of them),
> and have each thread (a) check if the early-abort flag is set, and
> then (b) start copying the next file in queue. Once the queue's empty
> or the early-abort flag is set, all ten threads will terminate when
> they finish their current transfers.

Yes, my example was simplified, but I do that :)

Laurent

[toc] | [prev] | [next] | [standalone]

#7981

On 6/19/2011 11:39 AM, Laurent Claessens wrote:

> In the same time I've a thread that read the list and perform the
> operations:
>
> def run():
> while task_list :
> task = task_list[0]
> task_list.remove(task)
> task.start()

Popping task off the end of the list is more efficient:

while task_list:
   task_list.pop().start()

or if the list is static

for task in task_list:
   task.start()


-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#7983

> Popping task off the end of the list is more efficient:

> while task_list:
>     task_list.pop().start()

That's cool. In my case it's better to do
task_list.pop(0).start

in order to pop the first element.

> or if the list is static

No, my list is dynamic and is feeded by an other thread (which also has 
to be closed).

Finally, I solved the problem by using a global flag looked at each 
iteration of the loop. A break is done if the global flag reports an error.

That was suggested by Alain Ketterlin from the French speaking python's 
usenet and Chris Angelico here.

Thanks all
Laurent

[toc] | [prev] | [next] | [standalone]

#7987

On 06/20/11 02:52, Laurent Claessens wrote:
> 
>> Popping task off the end of the list is more efficient:
> 
>> while task_list:
>>     task_list.pop().start()
> 
> That's cool. In my case it's better to do
> task_list.pop(0).start
> 
> in order to pop the first element.

then you really wanted a queue instead of a list.

There is a thread-safe `queue` module in the standard library.

[toc] | [prev] | [next] | [standalone]

#7984

On 6/19/2011 12:03 PM, Chris Angelico wrote:
> On Mon, Jun 20, 2011 at 1:39 AM, Laurent Claessens<moky.math@gmail.com>  wrote:
>> My problem is that when FileToCopyTask raises an error, the program does not
>> stop.
>> In fact when the error is Disk Full, I want to stop the whole program
>> because I know that the next task will fail too.
>
> If you're starting a thread for every file you're copying, you're
> starting a huge number of threads that probably will just end up
> fighting over the disk. To get a reasonably efficient early-abort, I'd
> recommend having a fixed pool of worker threads (say, ten of them),
> and have each thread (a) check if the early-abort flag is set, and
> then (b) start copying the next file in queue. Once the queue's empty
> or the early-abort flag is set, all ten threads will terminate when
> they finish their current transfers.
>
> (The ten threads figure is arbitrary. Optimum value for performance
> will come by adjusting this.)

I an not convinced that much of anything is gained by having multiple 
copying threads. It certainly adds complication -- hence this thread. I 
would just use shutil to copy files or directories of files.

Note that copytree has an option to pass a filter function:
"If ignore is given, it must be a callable that will receive as its 
arguments the directory being visited by copytree(), and a list of its 
contents, as returned by os.listdir(). Since copytree() is called 
recursively, the ignore callable will be called once for each directory 
that is copied. The callable must return a sequence of directory and 
file names relative to the current directory (i.e. a subset of the items 
in its second argument); these names will then be ignored in the copy 
process. ignore_patterns() can be used to create such a callable that 
ignores names based on glob-style patterns."

-- 
Terry Jan Reedy

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web