Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #18670 > unrolled thread

Parallel Processing

Back to article view | Back to comp.lang.python

Contents

  Parallel Processing Yigit Turgut <y.turgut@gmail.com> - 2012-01-08 05:23 -0800
    Re: Parallel Processing Dave Angel <d@davea.name> - 2012-01-08 09:34 -0500
      Re: Parallel Processing Yigit Turgut <y.turgut@gmail.com> - 2012-01-08 07:45 -0800
        Re: Parallel Processing Chris Angelico <rosuav@gmail.com> - 2012-01-09 03:00 +1100
          Re: Parallel Processing Yigit Turgut <y.turgut@gmail.com> - 2012-01-08 08:39 -0800
            Re: Parallel Processing Dave Angel <d@davea.name> - 2012-01-08 17:02 -0500

#18670 — Parallel Processing

Hi all,

I am trying to run two functions at the same time with Parallel
Processing (pp) as following ;

import pygame
import sys
import time
import math
import pp

screen = pygame.display.set_mode((0, 0), pygame.FULLSCREEN)
timer = pygame.time.Clock()
white = True
start = time.time()
end = time.time() - start

def test1():
  global end
  while(end<5):
    end = time.time() - start
    timer.tick(4) #FPS
    screen.fill((255,255,255) if white else (0, 0, 0))
    white = not white
    pygame.display.update()

def test2():
  global end
  while(end2<5):
    end2 = time.time() - start
    print end

ppservers = ()

if len(sys.argv) > 1:
    ncpus = int(sys.argv[1])
    # Creates jobserver with ncpus workers
    job_server = pp.Server(ncpus, ppservers=ppservers)
else:
    # Creates jobserver with automatically detected number of workers
    job_server = pp.Server(ppservers=ppservers)
print "Starting PP with", job_server.get_ncpus(), "workers"

job1 = job_server.submit(test1,test2)
result = job1()

And the output is ;

Starting PP with 2 workers
Traceback (most recent call last):
  File "fl.py", line 46, in <module>
    job1 = job_server.submit(test1,test2)
  File "/usr/lib/python2.6/site-packages/pp.py", line 402, in submit
    raise TypeError("args argument must be a tuple")
TypeError: args argument must be a tuple


When I change job1 to just to see if it will run one function only;
job1 = job_server.submit(test1)

I get an output of;

NameError: global name 'end' is not defined

end variable is defined as global but I get a NameError. Anyone has an
idea what's going on ?

[toc] | [next] | [standalone]

#18671

On 01/08/2012 08:23 AM, Yigit Turgut wrote:
> Hi all,
>
> I am trying to run two functions at the same time with Parallel
> Processing (pp) as following ;
>
> import pygame
> import sys
> import time
> import math
> import pp
>
> screen = pygame.display.set_mode((0, 0), pygame.FULLSCREEN)
> timer = pygame.time.Clock()
> white = True
> start = time.time()
> end = time.time() - start
>
> def test1():
>    global end
>    while(end<5):
>      end = time.time() - start
>      timer.tick(4) #FPS
>      screen.fill((255,255,255) if white else (0, 0, 0))
>      white = not white
>      pygame.display.update()
>
> def test2():
>    global end
>    while(end2<5):
>      end2 = time.time() - start
>      print end
>
> ppservers = ()
>
> if len(sys.argv)>  1:
>      ncpus = int(sys.argv[1])
>      # Creates jobserver with ncpus workers
>      job_server = pp.Server(ncpus, ppservers=ppservers)
> else:
>      # Creates jobserver with automatically detected number of workers
>      job_server = pp.Server(ppservers=ppservers)
> print "Starting PP with", job_server.get_ncpus(), "workers"
>
> job1 = job_server.submit(test1,test2)
> result = job1()
>
> And the output is ;
>
> Starting PP with 2 workers
> Traceback (most recent call last):
>    File "fl.py", line 46, in<module>
>      job1 = job_server.submit(test1,test2)
>    File "/usr/lib/python2.6/site-packages/pp.py", line 402, in submit
>      raise TypeError("args argument must be a tuple")
> TypeError: args argument must be a tuple
>
>
> When I change job1 to just to see if it will run one function only;
> job1 = job_server.submit(test1)
>
> I get an output of;
>
> NameError: global name 'end' is not defined
>
> end variable is defined as global but I get a NameError. Anyone has an
> idea what's going on ?
First, please tell us about any external dependencies (eg. imports).  
Pygame may not matter, since there are many people here using it, but 
each of us has to hunt down something that does parallel processing with 
an interface similar to what you're using.  For the next person, the 
most likely candidate out of the dozens available is:
http://www.parallelpython.com/

Your problem, as stated in the raise statement, is that the submit 
method is expecting a tuple for its "args argument".  Now I had to go to 
the website to find an example, but it appears that your second argument 
should have been a tuple of the arguments for your function.  See
*      submit*(self, func, args=(), depfuncs=(), modules=(), 
callback=None, callbackargs=(), group='default', globals=None)

on page: http://www.parallelpython.com/content/view/15/30/#API

So you're passing it two function objects, and the second one is in the 
place where it's expecting a tuple of arguments.  Apparently you should 
be calling submit multiple times, once for each function.

As for the error you get when you do that, please post the full 
traceback. I suspect that the message you did post has a typo in it, or 
that the error occurred when your source code was not as you show it 
here.  You have two variables end and end2, and only the first one is 
ever declared global.  func2() should get a different error when you run 
it; since it tries to use a local end2 before defining it.

I recommend always testing your code with a single thread before trying 
to launch more complex multithread stuff.  Just call func() and func2(), 
and see if they complete, and get reasonable answers.

-- 

DaveA

[toc] | [prev] | [next] | [standalone]

#18672

On Jan 8, 4:34 pm, Dave Angel <d...@davea.name> wrote:
> On 01/08/2012 08:23 AM, Yigit Turgut wrote:
>
>
>
>
>
>
>
> > Hi all,
>
> > I am trying to run two functions at the same time with Parallel
> > Processing (pp) as following ;
>
> > import pygame
> > import sys
> > import time
> > import math
> > import pp
>
> > screen = pygame.display.set_mode((0, 0), pygame.FULLSCREEN)
> > timer = pygame.time.Clock()
> > white = True
> > start = time.time()
> > end = time.time() - start
>
> > def test1():
> >    global end
> >    while(end<5):
> >      end = time.time() - start
> >      timer.tick(4) #FPS
> >      screen.fill((255,255,255) if white else (0, 0, 0))
> >      white = not white
> >      pygame.display.update()
>
> > def test2():
> >    global end
> >    while(end2<5):
> >      end2 = time.time() - start
> >      print end
>
> > ppservers = ()
>
> > if len(sys.argv)>  1:
> >      ncpus = int(sys.argv[1])
> >      # Creates jobserver with ncpus workers
> >      job_server = pp.Server(ncpus, ppservers=ppservers)
> > else:
> >      # Creates jobserver with automatically detected number of workers
> >      job_server = pp.Server(ppservers=ppservers)
> > print "Starting PP with", job_server.get_ncpus(), "workers"
>
> > job1 = job_server.submit(test1,test2)
> > result = job1()
>
> > And the output is ;
>
> > Starting PP with 2 workers
> > Traceback (most recent call last):
> >    File "fl.py", line 46, in<module>
> >      job1 = job_server.submit(test1,test2)
> >    File "/usr/lib/python2.6/site-packages/pp.py", line 402, in submit
> >      raise TypeError("args argument must be a tuple")
> > TypeError: args argument must be a tuple
>
> > When I change job1 to just to see if it will run one function only;
> > job1 = job_server.submit(test1)
>
> > I get an output of;
>
> > NameError: global name 'end' is not defined
>
> > end variable is defined as global but I get a NameError. Anyone has an
> > idea what's going on ?
>
> First, please tell us about any external dependencies (eg. imports).
> Pygame may not matter, since there are many people here using it, but
> each of us has to hunt down something that does parallel processing with
> an interface similar to what you're using.  For the next person, the
> most likely candidate out of the dozens available is:http://www.parallelpython.com/
>
> Your problem, as stated in the raise statement, is that the submit
> method is expecting a tuple for its "args argument".  Now I had to go to
> the website to find an example, but it appears that your second argument
> should have been a tuple of the arguments for your function.  See
> *      submit*(self, func, args=(), depfuncs=(), modules=(),
> callback=None, callbackargs=(), group='default', globals=None)
>
> on page:http://www.parallelpython.com/content/view/15/30/#API
>
> So you're passing it two function objects, and the second one is in the
> place where it's expecting a tuple of arguments.  Apparently you should
> be calling submit multiple times, once for each function.
>
> As for the error you get when you do that, please post the full
> traceback. I suspect that the message you did post has a typo in it, or
> that the error occurred when your source code was not as you show it
> here.  You have two variables end and end2, and only the first one is
> ever declared global.  func2() should get a different error when you run
> it; since it tries to use a local end2 before defining it.
>
> I recommend always testing your code with a single thread before trying
> to launch more complex multithread stuff.  Just call func() and func2(),
> and see if they complete, and get reasonable answers.
>
> --
>
> DaveA

There are no imports other than defined on the script, which are;

import pygame
import sys
import time
import math
import pp

You are correct about  trying to pass two functions and second one is
in place where a tuple of arguments supposed to be. But what if these
functions don't have any arguments ? I tested functions test1() and
test2() seperately ; they work. Once I figure out how to run these
functions simultaneously, I will add an argument to test2 and try then
on. My main goal is to simultaneously run two functions, one of them
has one argument the other doesn't. To get familiar with parallel
processing I am experimenting now without arguments and then I will
embed the code to my application. I am experimenting with the
following ;

import pygame
import sys
import time
import math
import pp

screen = pygame.display.set_mode((0, 0), pygame.FULLSCREEN)
timer = pygame.time.Clock()
white = True
start = time.time()
end = time.time() - start

def test1():
  global end
  global white
  while(end<5):
    end = time.time() - start
    timer.tick(4) #FPS
    screen.fill((255,255,255) if white else (0, 0, 0))
    white = not white
    pygame.display.update()

def test2():
  global end
  while(end<5):
    end = time.time() - start
    print end

ppservers = ()
job_server = pp.Server(ppservers=ppservers)
print "Starting pp with", job_server.get_ncpus(), "workers"

job1 = job_server.submit(test1())
job2 = job_server.submit(test2())
result = job1()
result2 = job2()

print "Counting...", result2

job_server.print_stats()

test1() works as expected (job1) but test2() doesn't work and I get
the following traceback ;

Traceback (most recent call last):
  File "fl.py", line 33, in <module>
    job1 = job_server.submit(test1())
  File "/usr/lib/python2.6/site-packages/pp.py", line 458, in submit
    sfunc = self.__dumpsfunc((func, ) + depfuncs, modules)
  File "/usr/lib/python2.6/site-packages/pp.py", line 629, in
__dumpsfunc
    sources = [self.__get_source(func) for func in funcs]
  File "/usr/lib/python2.6/site-packages/pp.py", line 696, in
__get_source
    sourcelines = inspect.getsourcelines(func)[0]
  File "/usr/lib/python2.6/inspect.py", line 678, in getsourcelines
    lines, lnum = findsource(object)
  File "/usr/lib/python2.6/inspect.py", line 519, in findsource
    file = getsourcefile(object) or getfile(object)
  File "/usr/lib/python2.6/inspect.py", line 441, in getsourcefile
    filename = getfile(object)
  File "/usr/lib/python2.6/inspect.py", line 418, in getfile
    raise TypeError('arg is not a module, class, method, '
TypeError: arg is not a module, class, method, function, traceback,
frame, or code object

Error is related to test1 not having an argument.  When I leave it
empty as following ;

job1 = job_server.submit(test1,())

test1 doesn't run. When I do ;

job1 = job_server.submit(test1())

Display works but I get;

TypeError: arg is not a module, class, method, function, traceback,
frame, or code object (complete traceback same as above).

And test2 doesn't work also. But when I do;

job1 = job_server.submit(test1,())
job2 = job_server.submit(test2())

I get test2 working but test1 not working. Obviously related to
argument arrangement in submit.

[toc] | [prev] | [next] | [standalone]

#18673

On Mon, Jan 9, 2012 at 2:45 AM, Yigit Turgut <y.turgut@gmail.com> wrote:
> job1 = job_server.submit(test1,())
> job2 = job_server.submit(test2())

The first of these passes test1 and an empty tuple as arguments to
submit(). The second calls test2 with no arguments, then passes its
return value to submit(), which is not what you want to do.

Chris Angelico

[toc] | [prev] | [next] | [standalone]

#18675

On Jan 8, 6:00 pm, Chris Angelico <ros...@gmail.com> wrote:
> On Mon, Jan 9, 2012 at 2:45 AM, Yigit Turgut <y.tur...@gmail.com> wrote:
> > job1 = job_server.submit(test1,())
> > job2 = job_server.submit(test2())
>
> The first of these passes test1 and an empty tuple as arguments to
> submit(). The second calls test2 with no arguments, then passes its
> return value to submit(), which is not what you want to do.
>
> Chris Angelico

Yes that's correct but (test1,()) doesn't do any good since it doesn't
execute the loop.

screen = pygame.display.set_mode((0, 0), pygame.FULLSCREEN)
timer = pygame.time.Clock()
white = True
start = time.time()
end = time.time() - start
end2= time.time() - start

def test1():
  global end
  global white
  while(end<5):
    end = time.time() - start
    timer.tick(4) #FPS
    screen.fill((255,255,255) if white else (0, 0, 0))
    white = not white
    pygame.display.update()

def test2():
  global end2
  while(end2<5):
    end2 = time.time() - start
    print end2

ppservers = ()
job_server = pp.Server(ppservers=ppservers)

job1 = job_server.submit(test1, (), globals=globals())
job2 = job_server.submit(test2, (), globals=globals())
result = job1()
result2 = job2()

print  result2

job_server.print_stats()

This *supposed to* print values of 'end' and simultaneously execute
test1. Eventhough I set globals parameter and nothing seems to be
wrong this code generates the following traceback ;

Starting pp with 2 workers
An error has occured during the function execution
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ppworker.py", line 90, in run
    __result = __f(*__args)
  File "<string>", line 4, in test1
NameError: global name 'end' is not defined
An error has occured during the function execution
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ppworker.py", line 90, in run
    __result = __f(*__args)
  File "<string>", line 3, in test2
NameError: global name 'end2' is not defined

How can this be, what am I missing ?

[toc] | [prev] | [next] | [standalone]

#18680

On 01/08/2012 11:39 AM, Yigit Turgut wrote:
>
> screen = pygame.display.set_mode((0, 0), pygame.FULLSCREEN)
> timer = pygame.time.Clock()
> white = True
> start = time.time()
> end = time.time() - start
> end2= time.time() - start
>
> def test1():
>    global end
>    global white
>    while(end<5):
>      end = time.time() - start
>      timer.tick(4) #FPS
>      screen.fill((255,255,255) if white else (0, 0, 0))
>      white = not white
>      pygame.display.update()
>
> def test2():
>    global end2
>    while(end2<5):
>      end2 = time.time() - start
>      print end2
>
> ppservers = ()
> job_server = pp.Server(ppservers=ppservers)
>
> job1 = job_server.submit(test1, (), globals=globals())
> job2 = job_server.submit(test2, (), globals=globals())
> result = job1()
> result2 = job2()
>
> print  result2
>
> job_server.print_stats()
>
> This *supposed to* print values of 'end' and simultaneously execute
> test1. Eventhough I set globals parameter and nothing seems to be
> wrong this code generates the following traceback ;
>
> Starting pp with 2 workers
> An error has occured during the function execution
> Traceback (most recent call last):
>    File "/usr/lib/python2.6/site-packages/ppworker.py", line 90, in run
>      __result = __f(*__args)
>    File "<string>", line 4, in test1
> NameError: global name 'end' is not defined
> An error has occured during the function execution
> Traceback (most recent call last):
>    File "/usr/lib/python2.6/site-packages/ppworker.py", line 90, in run
>      __result = __f(*__args)
>    File "<string>", line 3, in test2
> NameError: global name 'end2' is not defined
>
> How can this be, what am I missing ?
I don't see anything on the http://www.parallelpython.com 
<http://www.parallelpython.com/> website that indicates how it handles 
globals.  Remember this is creating a separate process, so it can't 
literally share the globals you have.  i would have expected it to 
pickle them when you say globals=globals(), but I dunno. In any case, I 
can't see any value in making end global with the "global" statement.  
I'd move the end= line inside the function, and forget about making it 
global.

The other thing you don't supply is a list of functions that might be 
called by your function.  See the depfuncs argument.  It probably 
handles all the system libraries, but I can't see how it'd be expected 
to handle pygame.

With the limited information supplied by the website, I'd experiment 
first with simpler things.  Make two functions that are self-contained, 
and try them first.  No global statements, and no calls to pygame.  
After that much worked, then I'd try adding arguments, and then return 
values.

Then i'd try calling separate functions (declaring them in depfuncs).  
And finally I'd try some 3rd party library.


-- 

DaveA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web