Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #82413 > unrolled thread

list comprehension return a list and sum over in loop

Back to article view | Back to comp.lang.python

Contents

  list comprehension return a list and sum over in loop KK Sasa <genwei007@gmail.com> - 2014-12-11 22:22 -0800
    Re: list comprehension return a list and sum over in loop Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-12-12 07:17 +0000
      Re: list comprehension return a list and sum over in loop KK Sasa <genwei007@gmail.com> - 2014-12-12 00:30 -0800
        Re: list comprehension return a list and sum over in loop Christian Gollwitzer <auriocus@gmx.de> - 2014-12-12 10:09 +0100
        Re: list comprehension return a list and sum over in loop Peter Otten <__peter__@web.de> - 2014-12-12 10:13 +0100
          Re: list comprehension return a list and sum over in loop KK Sasa <genwei007@gmail.com> - 2014-12-12 01:45 -0800
            Re: list comprehension return a list and sum over in loop Jussi Piitulainen <jpiitula@ling.helsinki.fi> - 2014-12-12 13:12 +0200
              Re: list comprehension return a list and sum over in loop Peter Otten <__peter__@web.de> - 2014-12-12 13:32 +0100
                Re: list comprehension return a list and sum over in loop KK Sasa <genwei007@gmail.com> - 2014-12-12 05:09 -0800
                  Re: list comprehension return a list and sum over in loop Peter Otten <__peter__@web.de> - 2014-12-12 17:08 +0100
                    Re: list comprehension return a list and sum over in loop KK Sasa <genwei007@gmail.com> - 2014-12-12 19:04 -0800
                      Re: list comprehension return a list and sum over in loop Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-12-13 11:27 +0000
              Re: list comprehension return a list and sum over in loop KK Sasa <genwei007@gmail.com> - 2014-12-12 05:03 -0800
    Re: list comprehension return a list and sum over in loop Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-12-12 20:46 +1100
    Re: list comprehension return a list and sum over in loop Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2014-12-12 16:47 +0000
    Re: list comprehension return a list and sum over in loop pecore@pascolo.net - 2014-12-13 13:38 +0100

#82413 — list comprehension return a list and sum over in loop

Hi there,

The list comprehension is results = [d2(t[k]) for k in xrange(1000)], where d2 is a function returning a list, say [x1,x2,x3,x4] for one example. So "results" is a list consisting of 1000 lists, each of length four. Here, what I want to get is the sum of 1000 lists, and then the result is a list of length four. Is there any efficient way to do this? Because I found it is slow in my case. I tried sum(d2(t[k]) for k in xrange(1000)), but it returned error: TypeError: unsupported operand type(s) for +: 'int' and 'list'. Thanks.

[toc] | [next] | [standalone]

#82415

On 12/12/2014 06:22, KK Sasa wrote:
> Hi there,
>
> The list comprehension is results = [d2(t[k]) for k in xrange(1000)], where d2 is a function returning a list, say [x1,x2,x3,x4] for one example. So "results" is a list consisting of 1000 lists, each of length four. Here, what I want to get is the sum of 1000 lists, and then the result is a list of length four. Is there any efficient way to do this? Because I found it is slow in my case. I tried sum(d2(t[k]) for k in xrange(1000)), but it returned error: TypeError: unsupported operand type(s) for +: 'int' and 'list'. Thanks.
>

I think you need something like this 
http://stackoverflow.com/questions/19339/a-transpose-unzip-function-in-python-inverse-of-zip

I'll let you add the finishing touches if I'm correct :)

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#82425

Mark Lawrence於 2014年12月12日星期五UTC+8下午3時17分43秒寫道：
> On 12/12/2014 06:22, KK Sasa wrote:
> > Hi there,
> >
> > The list comprehension is results = [d2(t[k]) for k in xrange(1000)], where d2 is a function returning a list, say [x1,x2,x3,x4] for one example. So "results" is a list consisting of 1000 lists, each of length four. Here, what I want to get is the sum of 1000 lists, and then the result is a list of length four. Is there any efficient way to do this? Because I found it is slow in my case. I tried sum(d2(t[k]) for k in xrange(1000)), but it returned error: TypeError: unsupported operand type(s) for +: 'int' and 'list'. Thanks.
> >
> 
> I think you need something like this 
> http://stackoverflow.com/questions/19339/a-transpose-unzip-function-in-python-inverse-of-zip
> 
> I'll let you add the finishing touches if I'm correct :)
> 
> -- 
> My fellow Pythonistas, ask not what our language can do for you, ask
> what you can do for our language.
> 
> Mark Lawrence

Hi Mark and Yotam,
  Thanks for kind reply. I think I didn't make my problem clear enough. The slow part is "[d2(t[k]) for k in xrange(1000)]". In addition, I don't need to construct a list of 1000 lists inside, but my aim is to get the sum of all "d2(t[k])". I wonder if there is any method to sum up efficiently.

[toc] | [prev] | [next] | [standalone]

#82428

Am 12.12.14 09:30, schrieb KK Sasa:
> Mark Lawrence於 2014年12月12日星期五UTC+8下午3時17分43秒寫道：
> Hi Mark and Yotam, Thanks for kind reply. I think I didn't make my
> problem clear enough. The slow part is "[d2(t[k]) for k in
> xrange(1000)]". In addition, I don't need to construct a list of 1000
> lists inside, but my aim is to get the sum of all "d2(t[k])". I
> wonder if there is any method to sum up efficiently.

Not sure I understand what you need, but it seems that NumPy would be a 
more efficient method. numpy.sum can sum elements along every dimensions 
of a higher-dimensional matrix, and in general NumPy stores the elements 
in native doubles instead of Python lists, which improves both 
performance and memory usage
	
	Christian

[toc] | [prev] | [next] | [standalone]

#82429

KK Sasa wrote:

> Mark Lawrence於 2014年12月12日星期五UTC+8下午3時17分43秒寫道：
>> On 12/12/2014 06:22, KK Sasa wrote:
>> > Hi there,
>> >
>> > The list comprehension is results = [d2(t[k]) for k in xrange(1000)],
>> > where d2 is a function returning a list, say [x1,x2,x3,x4] for one
>> > example. So "results" is a list consisting of 1000 lists, each of
>> > length four. Here, what I want to get is the sum of 1000 lists, and
>> > then the result is a list of length four. Is there any efficient way to
>> > do this? Because I found it is slow in my case. I tried sum(d2(t[k])
>> > for k in xrange(1000)), but it returned error: TypeError: unsupported
>> > operand type(s) for +: 'int' and 'list'. Thanks.
>> >
>> 
>> I think you need something like this
>> http://stackoverflow.com/questions/19339/a-transpose-unzip-function-in-python-inverse-of-zip
>> 
>> I'll let you add the finishing touches if I'm correct :)
>> 
>> --
>> My fellow Pythonistas, ask not what our language can do for you, ask
>> what you can do for our language.
>> 
>> Mark Lawrence
> 
> Hi Mark and Yotam,
>   Thanks for kind reply. I think I didn't make my problem clear enough.
>   The slow part is "[d2(t[k]) for k in xrange(1000)]". In addition, I
>   don't need to construct a list of 1000 lists inside, but my aim is to
>   get the sum of all "d2(t[k])". I wonder if there is any method to sum up
>   efficiently.

If that is slow the culprit is probably the d2() function. If so 

results = [0] * 4
for k in xrange(1000):
    for i, v in enumerate(d2(t[k])):
        results[i] += v

won't help. Can you tell us what's inside d2()?

[toc] | [prev] | [next] | [standalone]

#82432

Peter Otten於 2014年12月12日星期五UTC+8下午5時13分58秒寫道：
> KK Sasa wrote:
> 
> > Mark Lawrence於 2014年12月12日星期五UTC+8下午3時17分43秒寫道：
> >> On 12/12/2014 06:22, KK Sasa wrote:
> >> > Hi there,
> >> >
> >> > The list comprehension is results = [d2(t[k]) for k in xrange(1000)],
> >> > where d2 is a function returning a list, say [x1,x2,x3,x4] for one
> >> > example. So "results" is a list consisting of 1000 lists, each of
> >> > length four. Here, what I want to get is the sum of 1000 lists, and
> >> > then the result is a list of length four. Is there any efficient way to
> >> > do this? Because I found it is slow in my case. I tried sum(d2(t[k])
> >> > for k in xrange(1000)), but it returned error: TypeError: unsupported
> >> > operand type(s) for +: 'int' and 'list'. Thanks.
> >> >
> >> 
> >> I think you need something like this
> >> http://stackoverflow.com/questions/19339/a-transpose-unzip-function-in-python-inverse-of-zip
> >> 
> >> I'll let you add the finishing touches if I'm correct :)
> >> 
> >> --
> >> My fellow Pythonistas, ask not what our language can do for you, ask
> >> what you can do for our language.
> >> 
> >> Mark Lawrence
> > 
> > Hi Mark and Yotam,
> >   Thanks for kind reply. I think I didn't make my problem clear enough.
> >   The slow part is "[d2(t[k]) for k in xrange(1000)]". In addition, I
> >   don't need to construct a list of 1000 lists inside, but my aim is to
> >   get the sum of all "d2(t[k])". I wonder if there is any method to sum up
> >   efficiently.
> 
> If that is slow the culprit is probably the d2() function. If so 
> 
> results = [0] * 4
> for k in xrange(1000):
>     for i, v in enumerate(d2(t[k])):
>         results[i] += v
> 
> won't help. Can you tell us what's inside d2()?

Thanks for reply, Christian and Peter. Actually, the d2() is the Hessian function of a simple function (derived by using "ad" package, http://pythonhosted.org//ad/). The package hasn't supported Numpy array so far. And I am still not how to take a look inside d2(). Because I have to use d2() in a heavy way for iterative algorithm, the efficiency is a issue in my case. Following is an example.

import scipy
from scipy import stats
import numpy
from ad import adnumber
from ad.admath import *
from ad import jacobian
from ad import gh  # the gradient and hessian functions generator
from ad import *
import time
people = 1000
range_people = xrange(people)
dim0 = 2; mean0 = [0,0]; cov0 = [[1,0],[0,1]]
seed([1])
t = stats.multivariate_normal.rvs(mean0,cov0,people)
t = t.reshape(people,dim0)
t = t.tolist() # back to list
x = [0, 0, 1, 2]
point = 2
def p(x,t,point,z,obs):    
    d = x[0]
    tau = [0]+[x[1:point]] 
    a = x[point:len(x)]
    at = sum(i*j for i, j in zip(a, t))
    nu = [exp(z[k]*(at-d)-sum(tau[k])) for k in xrange(point)]
    de = sum(nu, axis=0)
    probability = [nu[k]/de for k in xrange(point)]
    return probability[obs]

d1, d2 = gh(p)    
tStart = time.time()
z = range(point)
re = [d2(x,t[k],2,z,1) for k in range_people]
tEnd = time.time()
print "It cost %f sec" % (tEnd - tStart)

[toc] | [prev] | [next] | [standalone]

#82436

KK Sasa writes:

> def p(x,t,point,z,obs):    
>     d = x[0]
>     tau = [0]+[x[1:point]] 
>     a = x[point:len(x)]
>     at = sum(i*j for i, j in zip(a, t))
>     nu = [exp(z[k]*(at-d)-sum(tau[k])) for k in xrange(point)]
>     de = sum(nu, axis=0)
>     probability = [nu[k]/de for k in xrange(point)]
>     return probability[obs]

I must be blind, but this looks like computing a whole probability
distribution and then throwing almost all of it away.

Can't this just return nu[obs]/de?

The expression for tau also seems weird to me. Isn't it equivalent to
[0, x[1:point]], a two-element list with the second element a list?
How can sum(tau[k]) work at all then, for any k > 1?

[toc] | [prev] | [next] | [standalone]

#82439

Jussi Piitulainen wrote:

> KK Sasa writes:
> 
>> def p(x,t,point,z,obs):
>>     d = x[0]
>>     tau = [0]+[x[1:point]]
>>     a = x[point:len(x)]
>>     at = sum(i*j for i, j in zip(a, t))
>>     nu = [exp(z[k]*(at-d)-sum(tau[k])) for k in xrange(point)]
>>     de = sum(nu, axis=0)
>>     probability = [nu[k]/de for k in xrange(point)]
>>     return probability[obs]
> 
> I must be blind, but this looks like computing a whole probability
> distribution and then throwing almost all of it away.
> 
> Can't this just return nu[obs]/de?
> 
> The expression for tau also seems weird to me. Isn't it equivalent to
> [0, x[1:point]], a two-element list with the second element a list?
> How can sum(tau[k]) work at all then, for any k > 1?

Also, after adding 

from numpy.random import seed

to the code I run into the next problem:

Traceback (most recent call last):
  File "hessian.py", line 34, in <module>
    re = [d2(x,t[k],2,z,1) for k in range_people]
  File "/home/petto/.local/lib/python2.7/site-packages/ad/__init__.py", line 
1114, in hess
    return func(xa, *args).hessian([xa])
  File "hessian.py", line 26, in p
    nu = [exp(z[k]*(at-d)-sum(tau[k])) for k in xrange(point)]
TypeError: 'int' object is not iterable

OP: Optimizations only make sense when you can start from a known-good base. 
You should sort out the logic of your problem (we probably can't help you 
with that), then fix the bugs in your code and only then revisit the "speed 
issue". 

[toc] | [prev] | [next] | [standalone]

#82444

Peter Otten於 2014年12月12日星期五UTC+8下午8時32分55秒寫道：
> Jussi Piitulainen wrote:
> 
> > KK Sasa writes:
> > 
> >> def p(x,t,point,z,obs):
> >>     d = x[0]
> >>     tau = [0]+[x[1:point]]
> >>     a = x[point:len(x)]
> >>     at = sum(i*j for i, j in zip(a, t))
> >>     nu = [exp(z[k]*(at-d)-sum(tau[k])) for k in xrange(point)]
> >>     de = sum(nu, axis=0)
> >>     probability = [nu[k]/de for k in xrange(point)]
> >>     return probability[obs]
> > 
> > I must be blind, but this looks like computing a whole probability
> > distribution and then throwing almost all of it away.
> > 
> > Can't this just return nu[obs]/de?
> > 
> > The expression for tau also seems weird to me. Isn't it equivalent to
> > [0, x[1:point]], a two-element list with the second element a list?
> > How can sum(tau[k]) work at all then, for any k > 1?
> 
> Also, after adding 
> 
> from numpy.random import seed
> 
> to the code I run into the next problem:
> 
> Traceback (most recent call last):
>   File "hessian.py", line 34, in <module>
>     re = [d2(x,t[k],2,z,1) for k in range_people]
>   File "/home/petto/.local/lib/python2.7/site-packages/ad/__init__.py", line 
> 1114, in hess
>     return func(xa, *args).hessian([xa])
>   File "hessian.py", line 26, in p
>     nu = [exp(z[k]*(at-d)-sum(tau[k])) for k in xrange(point)]
> TypeError: 'int' object is not iterable
> 
> OP: Optimizations only make sense when you can start from a known-good base. 
> You should sort out the logic of your problem (we probably can't help you 
> with that), then fix the bugs in your code and only then revisit the "speed 
> issue".

I have no idea why you added "from numpy.random import seed" to this syntax. Even I added that, my python didn't complain any bugs inside. Maybe your ad package was not be installed correctly?

[toc] | [prev] | [next] | [standalone]

#82457

KK Sasa wrote:

> Peter Otten於 2014年12月12日星期五UTC+8下午8時32分55秒寫道：
>> Jussi Piitulainen wrote:
>> 
>> > KK Sasa writes:
>> > 
>> >> def p(x,t,point,z,obs):
>> >>     d = x[0]
>> >>     tau = [0]+[x[1:point]]
>> >>     a = x[point:len(x)]
>> >>     at = sum(i*j for i, j in zip(a, t))
>> >>     nu = [exp(z[k]*(at-d)-sum(tau[k])) for k in xrange(point)]
>> >>     de = sum(nu, axis=0)
>> >>     probability = [nu[k]/de for k in xrange(point)]
>> >>     return probability[obs]
>> > 
>> > I must be blind, but this looks like computing a whole probability
>> > distribution and then throwing almost all of it away.
>> > 
>> > Can't this just return nu[obs]/de?
>> > 
>> > The expression for tau also seems weird to me. Isn't it equivalent to
>> > [0, x[1:point]], a two-element list with the second element a list?
>> > How can sum(tau[k]) work at all then, for any k > 1?
>> 
>> Also, after adding
>> 
>> from numpy.random import seed
>> 
>> to the code I run into the next problem:
>> 
>> Traceback (most recent call last):
>>   File "hessian.py", line 34, in <module>
>>     re = [d2(x,t[k],2,z,1) for k in range_people]
>>   File "/home/petto/.local/lib/python2.7/site-packages/ad/__init__.py",
>>   line
>> 1114, in hess
>>     return func(xa, *args).hessian([xa])
>>   File "hessian.py", line 26, in p
>>     nu = [exp(z[k]*(at-d)-sum(tau[k])) for k in xrange(point)]
>> TypeError: 'int' object is not iterable
>> 
>> OP: Optimizations only make sense when you can start from a known-good
>> base. You should sort out the logic of your problem (we probably can't
>> help you with that), then fix the bugs in your code and only then revisit
>> the "speed issue".
> 
> I have no idea why you added "from numpy.random import seed" to this
> syntax. Even I added that, my python didn't complain any bugs inside.
> Maybe your ad package was not be installed correctly?

It may be a version issue. I'm using

>>> import ad, numpy
>>> ad.__version__
'1.2.2'
>>> numpy.__version__
'1.9.1'

Or are you using an environment that does some imports automatically?

[toc] | [prev] | [next] | [standalone]

#82472

Sorry, i should say I'm using pythonxy, maybe it imports other things. 

[toc] | [prev] | [next] | [standalone]

#82475

On 13/12/2014 03:04, KK Sasa wrote:
> Sorry, i should say I'm using pythonxy, maybe it imports other things.
>

That is good to know but without any context it's rather difficult to 
relate it to anything.  Some people may have photographic memories and 
so remember everything that's been said in a thread, that certainly 
doesn't apply to me, and right now I'm just too lazy to go back and find 
out what this relates to :)

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#82442

Jussi Piitulainen於 2014年12月12日星期五UTC+8下午7時12分39秒寫道：
> KK Sasa writes:
> 
> > def p(x,t,point,z,obs):    
> >     d = x[0]
> >     tau = [0]+[x[1:point]] 
> >     a = x[point:len(x)]
> >     at = sum(i*j for i, j in zip(a, t))
> >     nu = [exp(z[k]*(at-d)-sum(tau[k])) for k in xrange(point)]
> >     de = sum(nu, axis=0)
> >     probability = [nu[k]/de for k in xrange(point)]
> >     return probability[obs]
> 
> I must be blind, but this looks like computing a whole probability
> distribution and then throwing almost all of it away.
> 
> Can't this just return nu[obs]/de?
> 
> The expression for tau also seems weird to me. Isn't it equivalent to
> [0, x[1:point]], a two-element list with the second element a list?
> How can sum(tau[k]) work at all then, for any k > 1?

This is just a probability of binary response. Not continuous one. Tau is a parameter and just have two in this case.

[toc] | [prev] | [next] | [standalone]

#82433

KK Sasa wrote:

> Hi there,
> 
> The list comprehension is results = [d2(t[k]) for k in xrange(1000)],
> where d2 is a function returning a list, say [x1,x2,x3,x4] for one
> example. So "results" is a list consisting of 1000 lists, each of length
> four. Here, what I want to get is the sum of 1000 lists, and then the
> result is a list of length four. Is there any efficient way to do this?
> Because I found it is slow in my case. I tried sum(d2(t[k]) for k in
> xrange(1000)), but it returned error: TypeError: unsupported operand
> type(s) for +: 'int' and 'list'. Thanks.

That's because sum() defaults to adding with a default value of 0:

py> sum([[1, 2], [3, 4]], 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'list'
py> sum([[1, 2], [3, 4]], [])
[1, 2, 3, 4]


But don't do that! It will be slow.


I don't completely understand your requirements. You should show some
typical data, and the expected result. The business with the xrange and
t[k] and even d2() is probably irrelevant. The important part is summing
the lists. That could mean either of these two things:

results = [ [1, 2, 3, 4], 
            [5, 6, 7, 8],
            [9, 10, 11, 12]]

sum(results)
--> gives [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]

The way to do this efficiently is:

results = []
for sublist in [d2(t[k]) for k in xrange(1000)]:
    results.extend(sublist)


Or perhaps you mean this:


results = [ [1, 2, 3, 4], 
            [5, 6, 7, 8],
            [9, 10, 11, 12]]

sum(results)
--> gives [15, 18, 21, 24]


I expect that the fastest, most efficient way to do this will be with the
third-party library numpy. But in pure Python, you can do this:

def add(alist, blist):
    if len(blist) != len(alist): raise ValueError
    for i, x in blist:
        alist[i] += x


results = [0]*4  # Like [0,0,0,0]
for sublist in [d2(t[k]) for k in xrange(1000)]:
    add(results, sublist)





-- 
Steven

[toc] | [prev] | [next] | [standalone]

#82459

On 12 December 2014 at 06:22, KK Sasa <genwei007@gmail.com> wrote:
> Hi there,
>
> The list comprehension is results = [d2(t[k]) for k in xrange(1000)], where d2 is a function returning a list, say [x1,x2,x3,x4] for one example. So "results" is a list consisting of 1000 lists, each of length four. Here, what I want to get is the sum of 1000 lists, and then the result is a list of length four. Is there any efficient way to do this? Because I found it is slow in my case. I tried sum(d2(t[k]) for k in xrange(1000)), but it returned error: TypeError: unsupported operand type(s) for +: 'int' and 'list'. Thanks.

Use numpy.sum:

>>> import numpy as np
>>> a = [[1, 1, 1, 1], [2, 2, 2, 2], [3, 3, 3, 3]]
>>> np.sum(a, 0)
array([6, 6, 6, 6])
>>> np.sum(a, 1)
array([ 4,  8, 12])

The second argument to numpy.sum is the dimension along which you want
to sum e.g. rows or columns.


Oscar

[toc] | [prev] | [next] | [standalone]

#82477

KK Sasa <genwei007@gmail.com> writes:

> Hi there,
>
> The list comprehension is results = [d2(t[k]) for k in
> xrange(1000)], where d2 is a function returning a list, say
> [x1,x2,x3,x4] for one example. So "results" is a list consisting of
> 1000 lists, each of length four. Here, what I want to get is the sum
> of 1000 lists, and then the result is a list of length four. Is
> there any efficient way to do this? Because I found it is slow in my
> case. I tried sum(d2(t[k]) for k in xrange(1000)), but it returned
> error: TypeError: unsupported operand type(s) for +: 'int' and
> 'list'. Thanks.

Why didn't you  follow Mark Lawrence's advice? 

In your problem, results is a list of N sublists, each containing
exactly four numerical values,

Let's try with N=2

In [36]: results = [d2(t[k]) for k in range(2)]
In [37]: print results
[[1, 2, 3, 4], [5, 6, 7, 8]]

Let's try the obvious method to sum

In [38]: [sum(el) for el in results]
Out[38]: [10, 26]

not what you're looking for, but what if we had 

In [39]: [sum(el) for el in zip(*results)]
Out[39]: [6, 8, 10, 12]

correct.

BTW, as you're using the scientific stack the answer of Christian
Gollwitzer is the more appropriate.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web