Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #18462

Large list in memory slows Python

Show all headers | View raw

Hi.

I am experiencing a puzzling problem with both Python 2.4 and Python
2.6 on CentOS 5. I'm looking for an explanation of the problem and
possible solutions. Here is what I did:

Python 2.4.3 (#1, Sep 21 2011, 19:55:41)
IPython 0.8.4 -- An enhanced Interactive Python.

In [1]: def test():
   ...:     return [(i,) for i in range(10**6)]

In [2]: %time x = test()
CPU times: user 0.82 s, sys: 0.04 s, total: 0.86 s
Wall time: 0.86 s

In [4]: big_list = range(50 * 10**6)

In [5]: %time y = test()
CPU times: user 9.11 s, sys: 0.03 s, total: 9.14 s
Wall time: 9.15 s

As you can see, after creating a list of 50 million integers, creating
the same list of 1 million tuples takes about 10 times longer than the
first time.

I ran these tests on a machine with 144GB of memory and it is not
swapping. Before creating the big list of integers, IPython used 111MB
of memory; After the creation, it used 1664MB of memory.

Thanks for your time.
Cheers.

-- 
Benoit Thiell
The SAO/NASA Astrophysics Data System
http://adswww.harvard.edu/

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Large list in memory slows Python Benoit Thiell <bthiell@cfa.harvard.edu> - 2012-01-03 17:38 -0500

csiph-web