Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #91992 > unrolled thread

Multiple thread program problem

Back to article view | Back to comp.lang.python

Contents

  Multiple thread program problem Mohan Mohta <mohan.mohta@gmail.com> - 2015-06-03 13:41 -0700
    Multiple thread program problem Sam Raker <sam.raker@gmail.com> - 2015-06-03 14:01 -0700
      Re: Multiple thread program problem Mohan Mohta <mohan.mohta@gmail.com> - 2015-06-03 15:23 -0700
        Re: Multiple thread program problem Mohan Mohta <mohan.mohta@gmail.com> - 2015-06-03 16:45 -0700
          Re: Multiple thread program problem sohcahtoa82@gmail.com - 2015-06-03 16:56 -0700
            Re: Multiple thread program problem M2 <mohan.mohta@gmail.com> - 2015-06-03 17:04 -0700
              Re: Multiple thread program problem Cameron Simpson <cs@zip.com.au> - 2015-06-04 10:37 +1000
                Re: Multiple thread program problem M2 <mohan.mohta@gmail.com> - 2015-06-03 19:59 -0700
                  Re: Multiple thread program problem Cameron Simpson <cs@zip.com.au> - 2015-06-04 14:42 +1000
    Re: Multiple thread program problem MRAB <python@mrabarnett.plus.com> - 2015-06-03 21:59 +0100
    Re: Multiple thread program problem Gary Herron <gherron@digipen.edu> - 2015-06-03 15:02 -0700
    Re: Multiple thread program problem M2 <mohan.mohta@gmail.com> - 2015-06-04 10:20 -0700
      Re: Multiple thread program problem Cameron Simpson <cs@zip.com.au> - 2015-06-05 08:58 +1000

#91992 — Multiple thread program problem

Hello
I am trying to create multiple thread through the below program but I am getting an error 

#! /usr/bin/python
import os
import subprocess
import thread
import threading
from thread import start_new_thread

def proc(f) :
        com1="ssh -B "
        com2=line.strip('\n')
        com3= " uname -a"
        co=str("ssh -B ")+ str(com2) + str(" uname -a")
        subprocess.call(co,shell=True)
        print "----------------------------"
        return

f = open('/tmp/python/1')
for line in f:
        t=thread.start_new_thread(proc(f),())
        t.start()
f.close()
c = raw_input(" Type anything to quit")


Execution output:
Linux abc.myhomenetwork.com 2.6.18-348.25.1.el5 #1 SMP Thu Apr 10 06:32:45 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
----------------------------
Traceback (most recent call last):
  File "./readfile1.py", line 19, in <module>
    t=thread.start_new_thread(proc(f),())
TypeError: first arg must be callable

[toc] | [next] | [standalone]

#91994

proc(f) isn't a callable, it's whatever it returns. IIRC, you need to do something like 'start_new_thread(proc, (f,))'

[toc] | [prev] | [next] | [standalone]

#92006

On Wednesday, June 3, 2015 at 4:01:13 PM UTC-5, Sam Raker wrote:
> proc(f) isn't a callable, it's whatever it returns. IIRC, you need to do something like 'start_new_thread(proc, (f,))'


If I execute something like 
t=thread.start_new_thread(proc,(f))

I get:

Traceback (most recent call last):
  File "./readfile1.py", line 19, in <module>
    t=thread.start_new_thread(proc,(f))
TypeError: 2nd arg must be a tuple

[toc] | [prev] | [next] | [standalone]

#92013

On Wednesday, June 3, 2015 at 5:34:31 PM UTC-5, Waffle wrote:
> You think "(f)" makes a tuple, but it does not.
> the parentesis is not the tuple constructor, the comma is
> try:
> t=thread.start_new_thread(proc,(f,))

Thanks for the pointer waffle.
The program executes now but still not the way I want it.
I think I will need to tweak it a bit as the code is executing with the same argument from the file /tmp/python/1 multiple times whereas it needs to be executed only ones but in parallel. Let me figure that out.


Once again thanks for all the help provided on this thread.

[toc] | [prev] | [next] | [standalone]

#92014

On Wednesday, June 3, 2015 at 4:45:52 PM UTC-7, M2 wrote:
> On Wednesday, June 3, 2015 at 5:34:31 PM UTC-5, Waffle wrote:
> > You think "(f)" makes a tuple, but it does not.
> > the parentesis is not the tuple constructor, the comma is
> > try:
> > t=thread.start_new_thread(proc,(f,))
> 
> Thanks for the pointer waffle.
> The program executes now but still not the way I want it.
> I think I will need to tweak it a bit as the code is executing with the same argument from the file /tmp/python/1 multiple times whereas it needs to be executed only ones but in parallel. Let me figure that out.
> 
> 
> Once again thanks for all the help provided on this thread.

Check your usages of "line" and "f".  You have spots where you probably meant "line" instead of "f", and others where you have "f" where you probably meant "line".

[toc] | [prev] | [next] | [standalone]

#92015

On Wednesday, June 3, 2015 at 6:56:47 PM UTC-5, sohca...@gmail.com wrote:
> On Wednesday, June 3, 2015 at 4:45:52 PM UTC-7, M2 wrote:
> > On Wednesday, June 3, 2015 at 5:34:31 PM UTC-5, Waffle wrote:
> > > You think "(f)" makes a tuple, but it does not.
> > > the parentesis is not the tuple constructor, the comma is
> > > try:
> > > t=thread.start_new_thread(proc,(f,))
> > 
> > Thanks for the pointer waffle.
> > The program executes now but still not the way I want it.
> > I think I will need to tweak it a bit as the code is executing with the same argument from the file /tmp/python/1 multiple times whereas it needs to be executed only ones but in parallel. Let me figure that out.
> > 
> > 
> > Once again thanks for all the help provided on this thread.
> 
> Check your usages of "line" and "f".  You have spots where you probably meant "line" instead of "f", and others where you have "f" where you probably meant "line".

Here is my logic:
f is where the entire file is getting loaded
which is also passed as argument in the function proc
line has a single line from the file which is then stripped off the new line character and assigned to com2 variable which helps in using it in the subprocess.call

[toc] | [prev] | [next] | [standalone]

#92016

On 03Jun2015 17:04, M2 <mohan.mohta@gmail.com> wrote:
>On Wednesday, June 3, 2015 at 6:56:47 PM UTC-5, sohca...@gmail.com wrote:
>> On Wednesday, June 3, 2015 at 4:45:52 PM UTC-7, M2 wrote:
>> > On Wednesday, June 3, 2015 at 5:34:31 PM UTC-5, Waffle wrote:
>> > > You think "(f)" makes a tuple, but it does not.
>> > > the parentesis is not the tuple constructor, the comma is
>> > > try:
>> > > t=thread.start_new_thread(proc,(f,))
>> >
>> > Thanks for the pointer waffle.
>> > The program executes now but still not the way I want it.
>> > I think I will need to tweak it a bit as the code is executing with the same argument from the file /tmp/python/1 multiple times whereas it needs to be executed only ones but in parallel. Let me figure that out.
>> >
>> >
>> > Once again thanks for all the help provided on this thread.
>>
>> Check your usages of "line" and "f".  You have spots where you probably meant "line" instead of "f", and others where you have "f" where you probably meant "line".
>
>Here is my logic:
>f is where the entire file is getting loaded

In the main code, yes.

>which is also passed as argument in the function proc

But why? f is not using in proc. Only line is.

>line has a single line from the file which is then stripped off the new line character and assigned to com2 variable which helps in using it in the subprocess.call

That end is fine.

I would be passing only "line" to proc, not "f" at all.

Suggestion: move your main code into its own function. That will make all the 
variables in it "local". Your proc function is presently relying on "line" 
being global, which generally bad and a recipe for disaster in multithreaded 
code.

Moving the main code into its own function will (1) get rid of the global 
variables and (2) force you to consider exactly what you need to pass to 
"proc", and that will help reveal various logic issues.

Cheers,
Cameron Simpson <cs@zip.com.au>

>>>How do you blip the throttle and wave? Do you blip it real high, then wave
>>>before the revs drop back?
>>Blip = right hand; Wave = left hand.  Do both simultaneously.  QED.
>Doesnt this make the bike lurch forward thru the intersection?
Not if the disk lock is in place...
        - Dean Woodward <deanw@agora.rdrop.com>

[toc] | [prev] | [next] | [standalone]

#92021

On Wednesday, June 3, 2015 at 7:38:22 PM UTC-5, Cameron Simpson wrote:
> On 03Jun2015 17:04, M2 <mohan.mohta@gmail.com> wrote:
> >On Wednesday, June 3, 2015 at 6:56:47 PM UTC-5, sohca...@gmail.com wrote:
> >> On Wednesday, June 3, 2015 at 4:45:52 PM UTC-7, M2 wrote:
> >> > On Wednesday, June 3, 2015 at 5:34:31 PM UTC-5, Waffle wrote:
> >> > > You think "(f)" makes a tuple, but it does not.
> >> > > the parentesis is not the tuple constructor, the comma is
> >> > > try:
> >> > > t=thread.start_new_thread(proc,(f,))
> >> >
> >> > Thanks for the pointer waffle.
> >> > The program executes now but still not the way I want it.
> >> > I think I will need to tweak it a bit as the code is executing with the same argument from the file /tmp/python/1 multiple times whereas it needs to be executed only ones but in parallel. Let me figure that out.
> >> >
> >> >
> >> > Once again thanks for all the help provided on this thread.
> >>
> >> Check your usages of "line" and "f".  You have spots where you probably meant "line" instead of "f", and others where you have "f" where you probably meant "line".
> >
> >Here is my logic:
> >f is where the entire file is getting loaded
> 
> In the main code, yes.
> 
> >which is also passed as argument in the function proc
> 
> But why? f is not using in proc. Only line is.
> 
> >line has a single line from the file which is then stripped off the new line character and assigned to com2 variable which helps in using it in the subprocess.call
> 
> That end is fine.
> 
> I would be passing only "line" to proc, not "f" at all.
> 
> Suggestion: move your main code into its own function. That will make all the 
> variables in it "local". Your proc function is presently relying on "line" 
> being global, which generally bad and a recipe for disaster in multithreaded 
> code.
> 
> Moving the main code into its own function will (1) get rid of the global 
> variables and (2) force you to consider exactly what you need to pass to 
> "proc", and that will help reveal various logic issues.
> 
> Cheers,
> Cameron Simpson <cs@zip.com.au>
> 
> >>>How do you blip the throttle and wave? Do you blip it real high, then wave
> >>>before the revs drop back?
> >>Blip = right hand; Wave = left hand.  Do both simultaneously.  QED.
> >Doesnt this make the bike lurch forward thru the intersection?
> Not if the disk lock is in place...
>         - Dean Woodward <deanw@agora.rdrop.com>

Thanks Cameron.
I do not see the duplication in the execution now.
I do see it is not consistent by executing all the threads ; it might be due to the fact I am using 
        subprocess.call(co,shell=True) 
Per my understanding the above does not keep track of threads it just spawns a thread and leaves it there.
I might need to use the function start(), join() to ensure it picks up all the argument

For the record now my new code is 
#! /usr/bin/python
import os
import subprocess
import thread
import threading
import sys
from thread import start_new_thread
 
def proc(col) :
        subprocess.call(col,shell=True)
        return
 
f = open('/tmp/python/1')
for line in f:
        com1="ssh -B "
        com2=line.strip('\n')
        com3= " uname -a  "
        co=str("ssh -B ")+ str(com2) + str(" uname -a")
        t=thread.start_new_thread(proc,(co,))
f.close()


Thanks again for the help

[toc] | [prev] | [next] | [standalone]

#92028

On 03Jun2015 19:59, M2 <mohan.mohta@gmail.com> wrote:
>On Wednesday, June 3, 2015 at 7:38:22 PM UTC-5, Cameron Simpson wrote:
>> I would be passing only "line" to proc, not "f" at all.
>>
>> Suggestion: move your main code into its own function. That will make all the
>> variables in it "local". Your proc function is presently relying on "line"
>> being global, which generally bad and a recipe for disaster in multithreaded
>> code.
>>
>> Moving the main code into its own function will (1) get rid of the global
>> variables and (2) force you to consider exactly what you need to pass to
>> "proc", and that will help reveal various logic issues.
[...]
>
>Thanks Cameron.
>I do not see the duplication in the execution now.
>I do see it is not consistent by executing all the threads ; it might be due to the fact I am using
>        subprocess.call(co,shell=True)
>Per my understanding the above does not keep track of threads it just spawns a thread and leaves it there.

subprocess does not start a thread at all. It starts a new external process.  
Your variable "t" is the Thread.

>I might need to use the function start(), join() to ensure it picks up all the argument

Because you have called .start_new_thread, you do not need to call .start().

Whether you call .join() on the Thread later is up to you. When you call 
.join(), the caller will block until the Thread terminates. So you won't want 
to do that immediately if you hope to run several subprocesses in parallel with 
this setup.

Thank you for including your current code; most helpful. I have some other 
remarks, below the code which I'm keeping here for reference:

>For the record now my new code is
>#! /usr/bin/python
>import os
>import subprocess
>import thread
>import threading
>import sys
>from thread import start_new_thread
>
>def proc(col) :
>        subprocess.call(col,shell=True)
>        return
>
>f = open('/tmp/python/1')
>for line in f:
>        com1="ssh -B "
>        com2=line.strip('\n')
>        com3= " uname -a  "
>        co=str("ssh -B ")+ str(com2) + str(" uname -a")
>        t=thread.start_new_thread(proc,(co,))
>f.close()

First up, your open/read/close stuff is normally written like this in Python:

  with open('/tmp/python/1') as f:
    for line in f:
      ... loop body here ...

This is because the return from open() is in fact a "context manager", an 
object designed to work with the "with" statement. Specificly, the "exit" step 
of this particular context manager calls .close() for you. Aside from being 
more succinct, a context manager has the other advantage that the exit action 
is _always_ called when control leaves the scope of the "with". Consider this 
example function:

  def foo(filename):
    with open(filename) as f:
      for line in f:
        ... do stuff with line ...
        if "quit" in line:
          return

That will return from the function if the string "quit" occurs on one of the 
lines in the file, and not process any following lines. The important point 
here is that if you use "with" then the open file will be closed automatically 
when the return happens. With your original open/loop/close code, the "return" 
would bypass the .close() call, leaving the file open.

The next remark I would make is that while Threads are very nice (I use them a 
lot), if all your thread is doing is dispatching a subprocess then you do not 
need a thread. The separate process that is made by subprocess runs 
automatically, immediately, without waiting for your program. So you can invoke 
a bunch of subprocesses in parallel, and not go near a thread.

However, because you have invoked the subprocess with .call(), your "proc" 
function inherently waits for the subprocess to complete before returning and 
therefore you need a thread to do these in parallel.

The .call() function is a convenience function; the alternative is to use the 
.Popen() constructor directly:

  def proc(col):
    P = subprocess.Popen(col, shell=True)
    return P

Then you can have your main loop replace "t=..." with:

  P = proc(co)

At this point, proc starts the subprocess but does not wait for it (versus 
"call", which does the wait for you). So you can dispatch all these 
subprocesses in parallel. Then after your loop you can wait for them to finish 
at your leisure.

Cheers,
Cameron Simpson <cs@zip.com.au>

Nonsense. Space is blue, and birds fly through it.      - Heisenberg

[toc] | [prev] | [next] | [standalone]

#91995

On 2015-06-03 21:41, Mohan Mohta wrote:
> Hello
> I am trying to create multiple thread through the below program but I am getting an error
>
> #! /usr/bin/python
> import os
> import subprocess
> import thread
> import threading
> from thread import start_new_thread
>
> def proc(f) :
>          com1="ssh -B "
>          com2=line.strip('\n')
>          com3= " uname -a"
>          co=str("ssh -B ")+ str(com2) + str(" uname -a")
>          subprocess.call(co,shell=True)
>          print "----------------------------"
>          return
>
> f = open('/tmp/python/1')
> for line in f:
>          t=thread.start_new_thread(proc(f),())
>          t.start()
> f.close()
> c = raw_input(" Type anything to quit")
>
>
> Execution output:
> Linux abc.myhomenetwork.com 2.6.18-348.25.1.el5 #1 SMP Thu Apr 10 06:32:45 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
> ----------------------------
> Traceback (most recent call last):
>    File "./readfile1.py", line 19, in <module>
>      t=thread.start_new_thread(proc(f),())
> TypeError: first arg must be callable
>
The first argument should be the function. You're calling proc(f) and 
the passing its result.

Also, start_new_thread starts the thread running immediately and
returns its identifier, an int.

[toc] | [prev] | [next] | [standalone]

#92035

On 06/03/2015 01:41 PM, Mohan Mohta wrote:
> Hello
> I am trying to create multiple thread through the below program but I am getting an error
>
> #! /usr/bin/python
> import os
> import subprocess
> import thread
> import threading
> from thread import start_new_thread
>
> def proc(f) :
>          com1="ssh -B "
>          com2=line.strip('\n')
>          com3= " uname -a"
>          co=str("ssh -B ")+ str(com2) + str(" uname -a")
>          subprocess.call(co,shell=True)
>          print "----------------------------"
>          return
>
> f = open('/tmp/python/1')
> for line in f:
>          t=thread.start_new_thread(proc(f),())

You are calling the function f yourself, but what you want here is for 
the thread to call f once it's running.
So do

     t=thread.start_new_thread(proc, (f,)) # Procedure to call and a 
tuple of args for it.

>          t.start()

I don't think the thread has a start method.

> f.close()
> c = raw_input(" Type anything to quit")
>
>
> Execution output:
> Linux abc.myhomenetwork.com 2.6.18-348.25.1.el5 #1 SMP Thu Apr 10 06:32:45 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
> ----------------------------
> Traceback (most recent call last):
>    File "./readfile1.py", line 19, in <module>
>      t=thread.start_new_thread(proc(f),())
> TypeError: first arg must be callable

You should probably also consider using the higher-level threading 
module rather than the lower level thread module.

(Also consider using Python3 instead of Python2.)

Gary Herron





-- 
Dr. Gary Herron
Department of Computer Science
DigiPen Institute of Technology
(425) 895-4418

[toc] | [prev] | [next] | [standalone]

#92070

On Wednesday, June 3, 2015 at 3:41:15 PM UTC-5, M2 wrote:
> Hello
> I am trying to create multiple thread through the below program but I am getting an error 
> 
> #! /usr/bin/python
> import os
> import subprocess
> import thread
> import threading
> from thread import start_new_thread
> 
> def proc(f) :
>         com1="ssh -B "
>         com2=line.strip('\n')
>         com3= " uname -a"
>         co=str("ssh -B ")+ str(com2) + str(" uname -a")
>         subprocess.call(co,shell=True)
>         print "----------------------------"
>         return
> 
> f = open('/tmp/python/1')
> for line in f:
>         t=thread.start_new_thread(proc(f),())
>         t.start()
> f.close()
> c = raw_input(" Type anything to quit")
> 
> 
> Execution output:
> Linux abc.myhomenetwork.com 2.6.18-348.25.1.el5 #1 SMP Thu Apr 10 06:32:45 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
> ----------------------------
> Traceback (most recent call last):
>   File "./readfile1.py", line 19, in <module>
>     t=thread.start_new_thread(proc(f),())
> TypeError: first arg must be callable

Awesome Cameron.
It works the way I want it to work.

Thanks a lot guys.
Here is the new code:
#! /usr/bin/python
import os
import subprocess
import thread
import threading
from thread import start_new_thread

def proc(col) :
        P=subprocess.Popen(col, shell=True)
        return

f = open('/tmp/python/1')
for line in f:
        com1="ssh -B "
        com2=line.strip('\n')
        #com3= " /var/scripts/health/health.sh"
        com3= " uname -a "
        co=str("ssh -B ")+ str(com2) + str(com3)
        P=proc(co)
f.close()


Thanks a ton guys this should get me started on the greater program I am trying to write.

[toc] | [prev] | [next] | [standalone]

#92099

On 04Jun2015 10:20, M2 <mohan.mohta@gmail.com> wrote:
>Awesome Cameron.
>It works the way I want it to work.

Glad to hear it. A few small remarks:

>Thanks a lot guys.
>Here is the new code:
[...]
>from thread import start_new_thread

You're not using this any more. You may want to tidy this up.

>def proc(col) :
>        P=subprocess.Popen(col, shell=True)
>        return

You are not returning P. But ...

>        co=str("ssh -B ")+ str(com2) + str(com3)
>        P=proc(co)

Here you are saving the return value as P (local to this function). My 
suggestion, I know. The point here is that in later code you might save all the 
P values and call P.wait() for them at some point after the loop, waiting for 
them to terminate. Or you may not care. Your call.

>f.close()

Consider adding the suggested "with" form. Shorter, easier to read, more 
reliable.

Cheers,
Cameron Simpson <cs@zip.com.au>

Ride fast
Die fast
Leave no usable organs
        - Tom Warner <tom@dfind.demon.co.uk>

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web