Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #80693 > unrolled thread

Help with Python Multiprocessing

Back to article view | Back to comp.lang.python

Contents

  Help with Python Multiprocessing Anurag <anuragpatibandla7@gmail.com> - 2014-11-13 10:07 -0800
    Re: Help with Python Multiprocessing Anurag <anuragpatibandla7@gmail.com> - 2014-11-13 10:10 -0800
      Re: Help with Python Multiprocessing MRAB <python@mrabarnett.plus.com> - 2014-11-13 18:45 +0000
    Re: Help with Python Multiprocessing sohcahtoa82@gmail.com - 2014-11-13 11:18 -0800
      Re: Help with Python Multiprocessing Anurag <anuragpatibandla7@gmail.com> - 2014-11-13 15:22 -0800
        Re: Help with Python Multiprocessing sohcahtoa82@gmail.com - 2014-11-14 11:41 -0800
    Re: Help with Python Multiprocessing Gary Herron <gary.herron@islandtraining.com> - 2014-11-13 11:12 -0800
      Re: Help with Python Multiprocessing Anurag <anuragpatibandla7@gmail.com> - 2014-11-13 15:21 -0800
    Re: Help with Python Multiprocessing Anurag <anuragpatibandla7@gmail.com> - 2014-11-23 12:56 -0800
      Re: Help with Python Multiprocessing sohcahtoa82@gmail.com - 2014-11-24 12:58 -0800

#80693 — Help with Python Multiprocessing

I am having trouble understanding the Multiprocessing module.
I need to run three different files 'Worker1' , 'Worker2', 'Worker3' all at once. Currently I am doing this :

from multiprocessing import Process

import Worker1.py
import Worker2.py
import Worker3.py



p1 = Process(target=Worker1.py) 
p1.start()
p2 = Process(target=Worker2.py)
p2.start()
p3 = Process(target=Worker3.py)
p3.start()

But this will only start the 'Worker1'. How do I execute all the three files at once?

Thanks

[toc] | [next] | [standalone]

#80694

On Thursday, November 13, 2014 1:07:56 PM UTC-5, Anurag wrote:
> I am having trouble understanding the Multiprocessing module.
> I need to run three different files 'Worker1' , 'Worker2', 'Worker3' all at once. Currently I am doing this :
> 
> from multiprocessing import Process
> 
> import Worker1.py
> import Worker2.py
> import Worker3.py
> 
> 
> 
> p1 = Process(target=Worker1.py) 
> p1.start()
> p2 = Process(target=Worker2.py)
> p2.start()
> p3 = Process(target=Worker3.py)
> p3.start()
> 
> But this will only start the 'Worker1'. How do I execute all the three files at once?
> 
> Thanks

I am using Python 2.7 and each of my 'Worker' has a never ending loop that runs continuously.

[toc] | [prev] | [next] | [standalone]

#80696

On 2014-11-13 18:10, Anurag wrote:
> On Thursday, November 13, 2014 1:07:56 PM UTC-5, Anurag wrote:
>> I am having trouble understanding the Multiprocessing module.
>> I need to run three different files 'Worker1' , 'Worker2', 'Worker3' all at once. Currently I am doing this :
>>
>> from multiprocessing import Process
>>
>> import Worker1.py
>> import Worker2.py
>> import Worker3.py
>>
>>
>>
>> p1 = Process(target=Worker1.py)
>> p1.start()
>> p2 = Process(target=Worker2.py)
>> p2.start()
>> p3 = Process(target=Worker3.py)
>> p3.start()
>>
>> But this will only start the 'Worker1'. How do I execute all the three files at once?
>>
>> Thanks
>
> I am using Python 2.7 and each of my 'Worker' has a never ending loop that runs continuously.
>
The documentation has this example:


from multiprocessing import Process

def f(name):
     print 'hello', name

if __name__ == '__main__':
     p = Process(target=f, args=('bob',))
     p.start()
     p.join()


Suppose this is in a module called 'example.py'.

When 'example.py' is run as the main module, it executes:

     p = Process(target=f, args=('bob',))
     p.start()

The multiprocessing module will _import_ 'example.py' in a new process,
and because it's only imported (and not run as a main module), that
code won't be executed. The multiprocessing module will then call the
'f' function.

The way you've written your module, it'll spawn a new process even when
it's only imported, causing the spawned process to spawn another
process, which will, in turn, spawn another...

[toc] | [prev] | [next] | [standalone]

#80697

On Thursday, November 13, 2014 10:07:56 AM UTC-8, Anurag wrote:
> I am having trouble understanding the Multiprocessing module.
> I need to run three different files 'Worker1' , 'Worker2', 'Worker3' all at once. Currently I am doing this :
> 
> from multiprocessing import Process
> 
> import Worker1.py
> import Worker2.py
> import Worker3.py
> 
> 
> 
> p1 = Process(target=Worker1.py) 
> p1.start()
> p2 = Process(target=Worker2.py)
> p2.start()
> p3 = Process(target=Worker3.py)
> p3.start()
> 
> But this will only start the 'Worker1'. How do I execute all the three files at once?
> 
> Thanks

Do your WorkerX.py files have a main() function or anything like that?  If not, they should.  Then, you'd set the targets to WorkerX.main.

[toc] | [prev] | [next] | [standalone]

#80716

On Thursday, November 13, 2014 2:18:50 PM UTC-5, sohca...@gmail.com wrote:
> On Thursday, November 13, 2014 10:07:56 AM UTC-8, Anurag wrote:
> > I am having trouble understanding the Multiprocessing module.
> > I need to run three different files 'Worker1' , 'Worker2', 'Worker3' all at once. Currently I am doing this :
> > 
> > from multiprocessing import Process
> > 
> > import Worker1.py
> > import Worker2.py
> > import Worker3.py
> > 
> > 
> > 
> > p1 = Process(target=Worker1.py) 
> > p1.start()
> > p2 = Process(target=Worker2.py)
> > p2.start()
> > p3 = Process(target=Worker3.py)
> > p3.start()
> > 
> > But this will only start the 'Worker1'. How do I execute all the three files at once?
> > 
> > Thanks
> 
> Do your WorkerX.py files have a main() function or anything like that?  If not, they should.  Then, you'd set the targets to WorkerX.main.

My Worker files have three different functions

[toc] | [prev] | [next] | [standalone]

#80780

On Thursday, November 13, 2014 3:22:49 PM UTC-8, Anurag wrote:
> On Thursday, November 13, 2014 2:18:50 PM UTC-5, sohca...@gmail.com wrote:
> > On Thursday, November 13, 2014 10:07:56 AM UTC-8, Anurag wrote:
> > > I am having trouble understanding the Multiprocessing module.
> > > I need to run three different files 'Worker1' , 'Worker2', 'Worker3' all at once. Currently I am doing this :
> > > 
> > > from multiprocessing import Process
> > > 
> > > import Worker1.py
> > > import Worker2.py
> > > import Worker3.py
> > > 
> > > 
> > > 
> > > p1 = Process(target=Worker1.py) 
> > > p1.start()
> > > p2 = Process(target=Worker2.py)
> > > p2.start()
> > > p3 = Process(target=Worker3.py)
> > > p3.start()
> > > 
> > > But this will only start the 'Worker1'. How do I execute all the three files at once?
> > > 
> > > Thanks
> > 
> > Do your WorkerX.py files have a main() function or anything like that?  If not, they should.  Then, you'd set the targets to WorkerX.main.
> 
> My Worker files have three different functions

What I mean is that your code should probably look more like this:

# Contents of main.py
from multiprocessing import Process
import Worker1
import Worker2
import Worker3

p1 = Process(target=Worker1.main)
p1.start()
p2 = Process(target=Worker2.main)
p2.start()
p3 = Process(target=Worker3.main)
p3.start

# Contents of Worker1.py
def main():
    # Do worker1 stuff...


# Contents of Worker2.py
def main():
    # Do worker2 stuff...


# Contents of Worker3.py
def main():
    # Do worker3 stuff...

Alternatively, you could have a single worker.py, import that, but have 3 main() functions, main1(), main2(), main3(), and set the targets for each process to those functions.

Maybe its because I'm less experienced as most people on this group, but setting a Process's target to a module and not a specific function in that module seems slightly strange and ambiguous to me.

[toc] | [prev] | [next] | [standalone]

#80698

On 11/13/2014 10:07 AM, Anurag wrote:
> I am having trouble understanding the Multiprocessing module.
> I need to run three different files 'Worker1' , 'Worker2', 'Worker3' all at once. Currently I am doing this :
>
> from multiprocessing import Process
>
> import Worker1.py
> import Worker2.py
> import Worker3.py
>
>
>
> p1 = Process(target=Worker1.py)
> p1.start()
> p2 = Process(target=Worker2.py)
> p2.start()
> p3 = Process(target=Worker3.py)
> p3.start()
>
> But this will only start the 'Worker1'. How do I execute all the three files at once?
>
> Thanks

I doubt that is your actual code.  Python imports do not include .py 
extension.  Please show us your actual code.  (Use cut and paste 
please.)  And then take the time to tell us how you determine only the 
first is started.  Then we can probably help.

As an aside: To be sure, one could make the above imports work by having 
files named py.py and __init__.py in a directory named Worker1 (and the 
same for directories Worker2 and Worker3).   I think it's more likely 
that you miss-typed the above code.

Gary Herron

[toc] | [prev] | [next] | [standalone]

#80715

On Thursday, November 13, 2014 2:22:29 PM UTC-5, Gary Herron wrote:
> On 11/13/2014 10:07 AM, Anurag wrote:
> > I am having trouble understanding the Multiprocessing module.
> > I need to run three different files 'Worker1' , 'Worker2', 'Worker3' all at once. Currently I am doing this :
> >
> > from multiprocessing import Process
> >
> > import Worker1.py
> > import Worker2.py
> > import Worker3.py
> >
> >
> >
> > p1 = Process(target=Worker1.py)
> > p1.start()
> > p2 = Process(target=Worker2.py)
> > p2.start()
> > p3 = Process(target=Worker3.py)
> > p3.start()
> >
> > But this will only start the 'Worker1'. How do I execute all the three files at once?
> >
> > Thanks
> 
> I doubt that is your actual code.  Python imports do not include .py 
> extension.  Please show us your actual code.  (Use cut and paste 
> please.)  And then take the time to tell us how you determine only the 
> first is started.  Then we can probably help.
> 
> As an aside: To be sure, one could make the above imports work by having 
> files named py.py and __init__.py in a directory named Worker1 (and the 
> same for directories Worker2 and Worker3).   I think it's more likely 
> that you miss-typed the above code.
> 
> Gary Herron

That is the actual code I am using in a file named 'ex.py'. 
My Worker files are also named with a '.py' extension.
When I run my 'ex.py' through a command prompt, the 'Worker1.py' is executed in the same command prompt and starts an infinite loop. It never gets to the 'Worker2.py' part.
But I want my Workers to be executed in different command prompts.

[toc] | [prev] | [next] | [standalone]

#81402

Hey Socha,
Your solution works. But then, all my 3 workers are running in a single command window. How do I make them run in three different command windows?

[toc] | [prev] | [next] | [standalone]

#81470

On Sunday, November 23, 2014 12:56:51 PM UTC-8, Anurag wrote:
> Hey Socha,
> Your solution works. But then, all my 3 workers are running in a single command window. How do I make them run in three different command windows?

That, I don't know.  You would probably need to open a new command window with subprocess.Popen (The exact command line would depend on your operating system) and write your script to include a command line option to tell it which worker it is, but that starts to get really complex fast, especially if data needs to be passed between the worker and the main process.

Otherwise, I don't know if what you really want to do can be done.  You might be better off looking into creating an actual GUI.  Maybe have a window with three text boxes with each worker adding text to their own box?

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web