Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #52441 > unrolled thread

multithreading in python

Back to article view | Back to comp.lang.python

Contents

  multithreading in python samaneh.yahyapour@gmail.com - 2013-08-13 01:06 -0700
    Re: multithreading in python Steven D'Aprano <steve@pearwood.info> - 2013-08-13 08:38 +0000
    Re: multithreading in python Dave Angel <davea@davea.name> - 2013-08-13 11:22 +0000
    Re: multithreading in python Dave Angel <davea@davea.name> - 2013-08-13 11:40 +0000
    Re: multithreading in python Terry Reedy <tjreedy@udel.edu> - 2013-08-13 11:51 -0400

#52441 — multithreading in python

hi 
my program work by 4 thread but when i use more thread it terminates

i use opencv in my image_process.so

my code is : 


#!/usr/bin/python
import sys
import os
import io
import time
import copy
import threading
import ctypes

class MyClass():
    
    def __init__(self):
        i = 0
        while i<10: 
            thread1 = threading.Thread(target=self.item_thread)
            thread1.start()
            i = i+1
            time.sleep(0.01)
        
    def item_thread(self):
        imageAnalyzer=ctypes.CDLL("../so/image_process.so")
        imageAnalyzer.aref_img_score_init("/opt/amniran/etc/face.xml", "/opt/amniran/etc/porn.xml")
        for filename in os.listdir("../script/images/"):
            if filename[-4:] == ".jpg" or filename[-4:] == ".png" or filename[-4:] == ".gif" or filename[-5:] == ".jpeg"    :
                
                path = "../script/images/%s"%filename
                
                fo = file(path, "r")
                content = fo.read()
                score = imageAnalyzer.score_image(content, len(content))
                print "%d : %s " %(score, path)
        print "ENDDDDDDDDDDDDDDDDDDDDDDDDDDDD"
                

                
                  
   
   
x = MyClass()         
 
 
 
how can i solve this problem 

[toc] | [next] | [standalone]

#52446

On Tue, 13 Aug 2013 01:06:01 -0700, samaneh.yahyapour wrote:

> hi
> my program work by 4 thread but when i use more thread it terminates

Is that a problem? Isn't it supposed to terminate, when it has finished?

If it raises an exception, or crashes, you should tell us.



> i use opencv in my image_process.so
> 
> my code is :
> 
> 
> #!/usr/bin/python
> import sys
> import os
> import io
> import time
> import copy
> import threading
> import ctypes
> 
> class MyClass():
>     
>     def __init__(self):
>         i = 0
>         while i<10:
>             thread1 = threading.Thread(target=self.item_thread)
>             thread1.start()
>             i = i+1
>             time.sleep(0.01)


This is better written as:

    def __init__(self):
        self.threads = []
        for i in range(10):
            thread = threading.Thread(target=self.item_thread)
            thread.start()
            self.threads.append(thread)
            time.sleep(0.01)  # not sure this helps for anything


I think it will also help if you keep references to the threads. That 
will stop them from being garbage collected unexpectedly, and you can 
check their status before exiting the main thread.


>     def item_thread(self):
>         imageAnalyzer=ctypes.CDLL("../so/image_process.so")
>         imageAnalyzer.aref_img_score_init("/opt/amniran/etc/face.xml",
>         "/opt/amniran/etc/porn.xml") for filename in
>         os.listdir("../script/images/"):
>             if filename[-4:] == ".jpg" or filename[-4:] == ".png" or
>             filename[-4:] == ".gif" or filename[-5:] == ".jpeg"    :
>                 
>                 path = "../script/images/%s"%filename
>                 
>                 fo = file(path, "r")
>                 content = fo.read()
>                 score = imageAnalyzer.score_image(content, len(content))
>                 print "%d : %s " %(score, path)
>         print "ENDDDDDDDDDDDDDDDDDDDDDDDDDDDD"
>                 
>    
> x = MyClass()


I suspect that when the main thread exits, and your other threads are 
still running, you may be in trouble. But I'm not a threading expert, so 
I could be wrong. However, I would put something like this at the end:

for thread in x.threads:
    x.join()


that way the main thread cannot finish until each of the subthreads are.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#52452

samaneh.yahyapour@gmail.com wrote:

> hi 
> my program work by 4 thread but when i use more thread it terminates
>
> 

I simplified your code so anybody could run it, and tested it inside
Komodo IDE, on Python 2.7

#!/usr/bin/env python


import sys
import os
import time
import threading

class MyClass():

    def __init__(self):
        i = 0
        while i<10:
            work = WorkClass(i)
            thread1 = threading.Thread(target=work.item_thread)
            thread1.start()
            i = i+1
            time.sleep(0.01)

class WorkClass():
    def __init__(self, parm):
        self.parm = str(parm)
    def item_thread(self):
        print "beginning thread", self.parm
        for filename in os.listdir("."):
            data =  "thread " +  self.parm + " filename " + filename + "\n"
            print data
            time.sleep(0.5)
        print "ENDDDDDDDDDDDDDDDDDDDDDDDDDDDD " + self.parm

x = MyClass()         
print "Finishing main thread"

When the
-- 
Signature file not found

[toc] | [prev] | [next] | [standalone]

#52453

samaneh.yahyapour@gmail.com wrote:

> hi 
> my program work by 4 thread but when i use more thread it terminates
>    <snip>
> how can i solve this problem 

I simplified the code so I could actually run it, and tested it in
Python 2.7, both under Komodo IDE and in the terminal.

The code:

#!/usr/bin/env python


import sys
import os
import time
import threading

class MyClass():

    def __init__(self):
        self.threads = []
        i = 0
        while i<10:
            work = WorkClass(i)
            thread1 = threading.Thread(target=work.item_thread)
            self.threads.append(thread1)
            thread1.start()
            i = i+1
            time.sleep(0.01)
        for thread in self.threads:   #wait for all the threads to end
            thread.join()

class WorkClass():
    def __init__(self, parm):
        self.parm = str(parm)
    def item_thread(self):
        print "beginning thread", self.parm
        for filename in os.listdir("."):
            data =  "thread " +  self.parm + " filename " + filename + "\n"
            print data
            time.sleep(0.5)
        print "ENDDDDDDDDDDDDDDDDDDDDDDDDDDDD " + self.parm

x = MyClass()         
print "All done with main thread"

The original code couldn't tell us what threads were running, so the
only clue you got was how many times it printed "ENDDDD"  So I arranged
that each thread had a unique "parm" value.  Typically you must do
something like this so that all the threads aren't doing exactly the
same work.

Another thing i did was to "atomicize" the print statements.  As it
originally was, partial lines from different threads could be
intermixed in the output.

The IDE showed the error message:

"""
ERROR: dbgp.client: 
The main thread of this application is exiting while there are still threads
alive. When the main thread exits, it is system defined whether the other
threads survive.

See Caveats at http://docs.python.org/lib/module-thread.html
"""

which tells you what's wrong.  You need to do a join on the threads
before exiting.

it's traditional (and better) to derive your own class from
threading.Thread, and that's where you can store any additional
attributes that each thread will need. I demonstrated something I
figured was simpler, by making the item_thread() method part of a
separate class.

-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#52464

On 8/13/2013 4:06 AM, samaneh.yahyapour@gmail.com wrote:

Aside from the other comments...

>      def item_thread(self):
>          imageAnalyzer=ctypes.CDLL("../so/image_process.so")
>          imageAnalyzer.aref_img_score_init("/opt/amniran/etc/face.xml", "/opt/amniran/etc/porn.xml")
>          for filename in os.listdir("../script/images/"):
>              if filename[-4:] == ".jpg" or filename[-4:] == ".png" or filename[-4:] == ".gif" or filename[-5:] == ".jpeg"    :
>
>                  path = "../script/images/%s"%filename
>
>                  fo = file(path, "r")

Use 'open' instead of the deprecated 'file' (removed in 3.x) and use it 
with a 'with' statement (this is now standard). This closes the file at 
the end of the block. Not doing so can cause problems on other 
implementations.

                 with open(path, 'rb') as fo:
>                  content = fo.read()

>                  score = imageAnalyzer.score_image(content, len(content))
>                  print "%d : %s " %(score, path)
>          print "ENDDDDDDDDDDDDDDDDDDDDDDDDDDDD"

Do you know how to use queue.Queue to spread work to multiple worker 
threads so each processes different files (and to collect results from 
multiple threads).? (If not, read doc.)

-- 
Terry Jan Reedy

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web