Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #75750 > unrolled thread

Pythonic way to iterate through multidimensional space?

Back to article view | Back to comp.lang.python

Contents

  Pythonic way to iterate through multidimensional space? Frank Miles <fpm@u.washington.edu> - 2014-08-05 20:06 +0000
    Re: Pythonic way to iterate through multidimensional space? Peter Otten <__peter__@web.de> - 2014-08-05 22:48 +0200
    Re: Pythonic way to iterate through multidimensional space? Frank Miles <fpm@u.washington.edu> - 2014-08-05 20:57 +0000
      Re: Pythonic way to iterate through multidimensional space? Gayathri J <usethisid2014@gmail.com> - 2014-08-06 11:04 +0530
      Re: Pythonic way to iterate through multidimensional space? Chris Angelico <rosuav@gmail.com> - 2014-08-06 16:25 +1000
      Re: Pythonic way to iterate through multidimensional space? Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-08-06 08:33 +0100
      Re: Pythonic way to iterate through multidimensional space? Peter Otten <__peter__@web.de> - 2014-08-06 09:39 +0200
      Re: Pythonic way to iterate through multidimensional space? Tim Chase <python.list@tim.thechases.com> - 2014-08-06 06:39 -0500
    Re: Pythonic way to iterate through multidimensional space? Wojciech Giel <wojtekgiel@gmail.com> - 2014-08-06 09:04 +0100
    Re: Pythonic way to iterate through multidimensional space? Gayathri J <usethisid2014@gmail.com> - 2014-08-06 17:43 +0530
    Re: Pythonic way to iterate through multidimensional space? Peter Otten <__peter__@web.de> - 2014-08-06 14:39 +0200
    Re: Pythonic way to iterate through multidimensional space? Gayathri J <usethisid2014@gmail.com> - 2014-08-06 18:57 +0530

#75750 — Pythonic way to iterate through multidimensional space?

I need to evaluate a complicated function over a multidimensional space
as part of an optimization problem.  This is a somewhat general problem
in which the number of dimensions and the function being evaluated can
vary from problem to problem.

I've got a working version (with loads of conditionals, and it only works
to #dimensions <= 10), but I'd like something simpler and clearer and
less hard-coded.

I've web-searched for some plausible method, but haven't found anything
"nice".  Any recommendations where I should look, or what technique should
be used?

TIA!

[toc] | [next] | [standalone]

#75753

Frank Miles wrote:

> I need to evaluate a complicated function over a multidimensional space
> as part of an optimization problem.  This is a somewhat general problem
> in which the number of dimensions and the function being evaluated can
> vary from problem to problem.
> 
> I've got a working version (with loads of conditionals, and it only works
> to #dimensions <= 10), but I'd like something simpler and clearer and
> less hard-coded.
> 
> I've web-searched for some plausible method, but haven't found anything
> "nice".  Any recommendations where I should look, or what technique should
> be used?

Not sure this is what you want, but if you have nested for loops -- these 
can be replaced with itertools.product():

>>> a = [1, 2]
>>> b = [10, 20, 30]
>>> c = [100]
>>> for x in a:
...     for y in b:
...         for z in c:
...             print(x, y, z)
... 
1 10 100
1 20 100
1 30 100
2 10 100
2 20 100
2 30 100
>>> for xyz in product(a, b, c):
...     print(*xyz)
... 
1 10 100
1 20 100
1 30 100
2 10 100
2 20 100
2 30 100

[toc] | [prev] | [next] | [standalone]

#75756

On Tue, 05 Aug 2014 20:06:05 +0000, Frank Miles wrote:

> I need to evaluate a complicated function over a multidimensional space
> as part of an optimization problem.  This is a somewhat general problem
> in which the number of dimensions and the function being evaluated can
> vary from problem to problem.
> 
> I've got a working version (with loads of conditionals, and it only works
> to #dimensions <= 10), but I'd like something simpler and clearer and
> less hard-coded.
> 
> I've web-searched for some plausible method, but haven't found anything
> "nice".  Any recommendations where I should look, or what technique should
> be used?
> 
> TIA!

Ahhhh.... should have waited.  The answer: itertools.product!  very nice

[toc] | [prev] | [next] | [standalone]

#75780

[Multipart message — attachments visible in raw view] — view raw

Dear Peter

Below is the code I tried to check if itertools.product() was faster than
normal nested loops...

they arent! arent they supposed to be...or am i making a mistake? any idea?


*############################################################*

*# -*- coding: utf-8 -*-*
*import numpy as np*
*import time*
*from itertools import product,repeat*
*def main():*
*    # N - size of grid*
*    # nvel - number of velocities*
*    # times - number of times to run the functions*
*    N=256*
*    times=3*
*    f=np.random.rand(N,N,N)*

*    # using loops*
*    print "normal nested loop"*
*    python_dot_loop1(f,times,N)*

*    print "nested loop using itertools.product()"*
*    python_dot_loop2(f,times,N)*

*def python_dot_loop1(f,times,N):*
*    for t in range(times):*
*         t1=time.time()*
*         for i in range(N):*
*             for j in range(N):*
*                   for k in range(N):*
*                       f[i,j,k] = 0.0*
*         print "python dot loop " + str(time.time()-t1)*

*def python_dot_loop2(f,times,N):*
*    rangeN=range(N)*
*    for t in range(times):*
*         t1=time.time()*
*         for i,j,k in product(rangeN,repeat=3):*
*             f[i,j,k]=0.0*
*         print "python dot loop " + str(time.time()-t1)*


*if __name__=='__main__':*
*    main()*
*############################################################*

[toc] | [prev] | [next] | [standalone]

#75781

On Wed, Aug 6, 2014 at 3:34 PM, Gayathri J <usethisid2014@gmail.com> wrote:
> Below is the code I tried to check if itertools.product() was faster than
> normal nested loops...
>
> they arent! arent they supposed to be...or am i making a mistake? any idea?

Don't worry about what's faster. That almost never matters. Worry,
instead, about how you would code it if you can't be sure how many
dimensions there'll be until run time (which the OP said can happen).
With product(), you can give it a variable number of arguments (eg
with *args notation), but with loops, you'd need to compile up some
code with that many nested loops - or at best, something where you cap
the number of loops and thus dimensions, and have a bunch of them
iterate over a single iterable.

ChrisA

[toc] | [prev] | [next] | [standalone]

#75783

On 06/08/2014 06:34, Gayathri J wrote:
> Dear Peter
>
> Below is the code I tried to check if itertools.product() was faster
> than normal nested loops...
>
> they arent! arent they supposed to be...or am i making a mistake? any idea?
> *
> *
> *############################################################
> *
> *# -*- coding: utf-8 -*-
> *
> *import numpy as np*
> *import time*
> *from itertools import product,repeat*
> *def main():*
> *    # N - size of grid*
> *    # nvel - number of velocities*
> *    # times - number of times to run the functions*
> *    N=256*
> *    times=3*
> *    f=np.random.rand(N,N,N)*
> **
> *    # using loops*
> *    print "normal nested loop"*
> *    python_dot_loop1(f,times,N)*
> *
> *
> *    print "nested loop using itertools.product()"*
> *    python_dot_loop2(f,times,N)*
> *
> *
> *def python_dot_loop1(f,times,N):*
> *    for t in range(times):*
> *         t1=time.time()*
> *         for i in range(N):*
> *             for j in range(N):*
> *                   for k in range(N):*
> *                       f[i,j,k] = 0.0*
> *         print "python dot loop " + str(time.time()-t1)*
> **
> *def python_dot_loop2(f,times,N):*
> *    rangeN=range(N)*
> *    for t in range(times):*
> *         t1=time.time()*
> *         for i,j,k in product(rangeN,repeat=3):*
> *             f[i,j,k]=0.0*
> *         print "python dot loop " + str(time.time()-t1)*
> *
> *
> *
> *
> *if __name__=='__main__':*
> *    main()*
> *############################################################*
>
>

Who cares, well not I for one?  Give me slower but accurate code over 
faster but inaccurate code any day of the week?

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#75784

Gayathri J wrote:

> Dear Peter
> 
> Below is the code I tried to check if itertools.product() was faster than
> normal nested loops...
> 
> they arent! arent they supposed to be...

I wouldn't have expected product() to be significantly faster, but neither 
did I expect it to be slower.

> or am i making a mistake? any
> idea?

For your testcase you can shave off a small amount by avoiding the tuple-
unpacking

for t in product(...):
    f[t] = 0.0

but that may be cheating, and the effect isn't big.

When you are working with numpy there may be specialised approaches, but

    f[:,:,:] = 0.0

is definitely cheating I suppose...

[toc] | [prev] | [next] | [standalone]

#75800

On 2014-08-06 11:04, Gayathri J wrote:
> Below is the code I tried to check if itertools.product() was
> faster than normal nested loops...
> 
> they arent! arent they supposed to be...or am i making a mistake?

I believe something like this was discussed a while ago and there was
a faster-but-uglier solution so you might want to consult this thread:

https://mail.python.org/pipermail/python-list/2008-January/516109.html

I believe this may have taken place before itertools.product() came
into existence.

-tkc

[toc] | [prev] | [next] | [standalone]

#75785

You might check numpy it is really powerful tool for working with multi 
dimensional arrays:

ex.
 >>> a = arange(81).reshape(3,3,3,3)
 >>> a

array([[[[ 0,  1,  2],
          [ 3,  4,  5],
          [ 6,  7,  8]],

         [[ 9, 10, 11],
          [12, 13, 14],
          [15, 16, 17]],

         [[18, 19, 20],
          [21, 22, 23],
          [24, 25, 26]]],


        [[[27, 28, 29],
          [30, 31, 32],
          [33, 34, 35]],

         [[36, 37, 38],
          [39, 40, 41],
          [42, 43, 44]],

         [[45, 46, 47],
          [48, 49, 50],
          [51, 52, 53]]],


        [[[54, 55, 56],
          [57, 58, 59],
          [60, 61, 62]],

         [[63, 64, 65],
          [66, 67, 68],
          [69, 70, 71]],

         [[72, 73, 74],
          [75, 76, 77],
          [78, 79, 80]]]])

 >>> f = a.flat
 >>> for i in f:
...    print(i)
0
1
2
..
98
99

cheers
Wojciech



On 05/08/14 21:06, Frank Miles wrote:
> I need to evaluate a complicated function over a multidimensional space
> as part of an optimization problem.  This is a somewhat general problem
> in which the number of dimensions and the function being evaluated can
> vary from problem to problem.
>
> I've got a working version (with loads of conditionals, and it only works
> to #dimensions <= 10), but I'd like something simpler and clearer and
> less hard-coded.
>
> I've web-searched for some plausible method, but haven't found anything
> "nice".  Any recommendations where I should look, or what technique should
> be used?
>
> TIA!

[toc] | [prev] | [next] | [standalone]

#75794

[Multipart message — attachments visible in raw view] — view raw

Dear Peter

Yes the f[t] or f[:,:,:] might give  a marginal increase, but then i need
to do further operations  using the indices, in which case this wouldnt help


Dear Wojciech

np.flat() works if u dont care about the indices and only the matrix/array
values matter.
but if the <i,j,k> matters, flatten wouldnt work



On Wed, Aug 6, 2014 at 1:34 PM, Wojciech Giel <wojtekgiel@gmail.com> wrote:

> You might check numpy it is really powerful tool for working with multi
> dimensional arrays:
>
> ex.
> >>> a = arange(81).reshape(3,3,3,3)
> >>> a
>
> array([[[[ 0,  1,  2],
>          [ 3,  4,  5],
>          [ 6,  7,  8]],
>
>         [[ 9, 10, 11],
>          [12, 13, 14],
>          [15, 16, 17]],
>
>         [[18, 19, 20],
>          [21, 22, 23],
>          [24, 25, 26]]],
>
>
>        [[[27, 28, 29],
>          [30, 31, 32],
>          [33, 34, 35]],
>
>         [[36, 37, 38],
>          [39, 40, 41],
>          [42, 43, 44]],
>
>         [[45, 46, 47],
>          [48, 49, 50],
>          [51, 52, 53]]],
>
>
>        [[[54, 55, 56],
>          [57, 58, 59],
>          [60, 61, 62]],
>
>         [[63, 64, 65],
>          [66, 67, 68],
>          [69, 70, 71]],
>
>         [[72, 73, 74],
>          [75, 76, 77],
>          [78, 79, 80]]]])
>
> >>> f = a.flat
> >>> for i in f:
> ...    print(i)
> 0
> 1
> 2
> ..
> 98
> 99
>
> cheers
> Wojciech
>
>
>
>
> On 05/08/14 21:06, Frank Miles wrote:
>
>> I need to evaluate a complicated function over a multidimensional space
>> as part of an optimization problem.  This is a somewhat general problem
>> in which the number of dimensions and the function being evaluated can
>> vary from problem to problem.
>>
>> I've got a working version (with loads of conditionals, and it only works
>> to #dimensions <= 10), but I'd like something simpler and clearer and
>> less hard-coded.
>>
>> I've web-searched for some plausible method, but haven't found anything
>> "nice".  Any recommendations where I should look, or what technique should
>> be used?
>>
>> TIA!
>>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#75796

Gayathri J wrote:

> Dear Peter
> 
> Yes the f[t] or f[:,:,:] might give  a marginal increase, 

The speedup compared itertools.product() is significant:

$ python -m timeit -s 'from itertools import product; from numpy.random 
import rand; N = 100; a = rand(N, N, N); r = range(N)' 'for x in product(r, 
repeat=3): a[x] = 0.0'
10 loops, best of 3: 290 msec per loop

$ python -m timeit -s 'from itertools import product; from numpy.random 
import rand; N = 100; a = rand(N, N, N); r = range(N)' 'a[:,:,:] = 0.0'
100 loops, best of 3: 3.58 msec per loop

But normally you'd just make a new array with numpy.zeros().

> but then i need
> to do further operations  using the indices, in which case this wouldnt
> help

Which is expected and also the crux of such micro-benchmarks. They distract 
from big picture.

[toc] | [prev] | [next] | [standalone]

#75801

[Multipart message — attachments visible in raw view] — view raw

Dear Peter

thanks . But thats what I was trying to say

just taking them to zero by f[:,:,:] = 0.0 or using np.zeros is surely
going to give me a time gain...
but my example of using the itertools.product() and doing f[x] =0.0 is just
to compare the looping timing with the traditional nested loops and not to
distract us to the operation done inside the loop.

right?


On Wed, Aug 6, 2014 at 6:09 PM, Peter Otten <__peter__@web.de> wrote:

> Gayathri J wrote:
>
> > Dear Peter
> >
> > Yes the f[t] or f[:,:,:] might give  a marginal increase,
>
> The speedup compared itertools.product() is significant:
>
> $ python -m timeit -s 'from itertools import product; from numpy.random
> import rand; N = 100; a = rand(N, N, N); r = range(N)' 'for x in product(r,
> repeat=3): a[x] = 0.0'
> 10 loops, best of 3: 290 msec per loop
>
> $ python -m timeit -s 'from itertools import product; from numpy.random
> import rand; N = 100; a = rand(N, N, N); r = range(N)' 'a[:,:,:] = 0.0'
> 100 loops, best of 3: 3.58 msec per loop
>
> But normally you'd just make a new array with numpy.zeros().
>
> > but then i need
> > to do further operations  using the indices, in which case this wouldnt
> > help
>
> Which is expected and also the crux of such micro-benchmarks. They distract
> from big picture.
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web