Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #22131 > unrolled thread

multiprocessing & itertools.product Iterator

Back to article view | Back to comp.lang.python

Contents

  multiprocessing &  itertools.product Iterator Christian <mining.facts@googlemail.com> - 2012-03-24 16:35 -0700
    Re: multiprocessing &  itertools.product Iterator Kiuhnm <kiuhnm03.4t.yahoo.it> - 2012-03-25 16:52 +0200

#22131 — multiprocessing & itertools.product Iterator

Hey,

I struggle to "extend" a multiprocessing example to my problem with a
itertools.product result iterator.
How I have to  assign the combos.next() elements approriate to
Pool.imap/calc functions?

Thanks in advance
Christian


from multiprocessing import Process,Queue,Pool
import Calculation
import DataSimulation
from itertools import product


def produce_example_combos(size=6,na=1000,nb=10):
    data = DataSimulation.DataSimulation()
    a = [data.generate_simple(size) for x in xrange(na)]
    b = [data.generate_simple(size) for x in xrange(nb)]
    it = product(a,b)
    return it

def calc(elements):
    calc.q.put("Doing:" +  elements[0] + elements[1])
    ratio = Calculation.ratio(elements[0],elements[1])
    return ratio

def calc_init(q):
    calc.q = q


if __name__ == '__main__':
    combos = produce_example_combos()
    print "tesdata generated"
    q = Queue()
    p = Pool(10, calc_init, [q])
    results = p.imap(calc,combos.next())
    p.close()

    for i in combos:
        print q.get()
        print results.next()

[toc] | [next] | [standalone]

#22140

On 3/25/2012 0:35, Christian wrote:
> Hey,
>
> I struggle to "extend" a multiprocessing example to my problem with a
> itertools.product result iterator.
> How I have to  assign the combos.next() elements approriate to
> Pool.imap/calc functions?
>
> Thanks in advance
> Christian
>
>
> from multiprocessing import Process,Queue,Pool
> import Calculation
> import DataSimulation
> from itertools import product
>
>
> def produce_example_combos(size=6,na=1000,nb=10):
>      data = DataSimulation.DataSimulation()
>      a = [data.generate_simple(size) for x in xrange(na)]
>      b = [data.generate_simple(size) for x in xrange(nb)]
>      it = product(a,b)
>      return it
>
> def calc(elements):
>      calc.q.put("Doing:" +  elements[0] + elements[1])

This throws because elements[0] isn't a string.
Try with
     calc.q.put("Doing: {} {}".format(elements[0], elements[1]))
or something similar.
To see the error, use something like
     try:
         calc.q.put("Doing:" +  elements[0] + elements[1])
         ratio = Calculation.ratio(elements[0],elements[1])
         return ratio
     except Exception as exc:
         print(exc.__doc__)

>      ratio = Calculation.ratio(elements[0],elements[1])
>      return ratio
>
> def calc_init(q):
>      calc.q = q
>
>
> if __name__ == '__main__':
>      combos = produce_example_combos()
>      print "tesdata generated"
>      q = Queue()
>      p = Pool(10, calc_init, [q])
>      results = p.imap(calc,combos.next())

Why combos.next()?
imap expects an iterable, i.e. combos, not the first element in combos.
That's similar to
   for i in combos.next()

>      p.close()

I don't know whether p.wait() is also needed here.

>      for i in combos:
>          print q.get()
>          print results.next()

That doesn't work, because combos is an iterator and you've already got 
to the end (next() after next()).
Look at this example:
     combos = produce_example_combos()
     for i in combos:
         print("ok")
     for i in combos:
         print("not ok")
That code will print a few "ok" but not even a single "not ok".

You should write
     for r in results:
         print q.get()
         print r

Here's a working example (I had to do some editing):

---->
from multiprocessing import Process,Queue,Pool
#import Calculation
#import DataSimulation
from itertools import product

def produce_example_combos(size=6,na=1000,nb=10):
     #data = DataSimulation.DataSimulation()
     #a = [data.generate_simple(size) for x in xrange(na)]
     #b = [data.generate_simple(size) for x in xrange(nb)]
     a = ['a', 'b', 'c']
     b = [1, 2, 3]
     it = product(a,b)
     return it

def calc(elements):
     calc.q.put("Doing: {}{}".format(elements[0], elements[1]))
     #ratio = Calculation.ratio(elements[0], elements[1])
     ratio = elements
     return ratio

def calc_init(q):
     calc.q = q

if __name__ == '__main__':
     combos = produce_example_combos()
     print("tesdata generated")
     q = Queue()
     p = Pool(10, calc_init, [q])
     results = p.imap(calc, combos)
     p.close()
     p.join()

     for r in results:
         print(q.get())
         print(r)
     input("")
<----

Kiuhnm

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web