Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #50825 > unrolled thread

Why on CentOS, python consumes too much memory ?

Back to article view | Back to comp.lang.python

Contents

  Why on CentOS, python consumes too much memory ? snowingbear@gmail.com - 2013-07-17 21:45 -0700
    Re: Why on CentOS, python consumes too much memory ? William Bai <snowingbear@gmail.com> - 2013-07-18 02:13 -0700
      Re: Why on CentOS, python consumes too much memory ? Michael Torrie <torriem@gmail.com> - 2013-07-18 07:51 -0600
    Re: Why on CentOS, python consumes too much memory ? Antoine Pitrou <solipsis@pitrou.net> - 2013-07-18 09:21 +0000

#50825 — Why on CentOS, python consumes too much memory ?

Hi:

   Previously, we found that our python scripts consume too much memory. So I use python's resource module to restrict RLIMIT_AS's soft limit and hard limit to 200M.
On my RHEL5.3(i386)+python2.6.2, it works OK. But on CentOS 6.2(x86_64)+python2.6.6, it reports memory error(exceeding 200M).

    And I tested with a very small script, and result is out of my expect, it still use too much memory on my CentOS 6.2 python:
import time
time.sleep(200)

    I use guppy and memory_profiler to see the memory usage and see that python objects just use about 6M memory both on RHEL5.3(i386)+python2.6 and CentOS 6.2(x86_64)+python2.6. But when I cat /proc/<pid>/status. I found that though VmRss is not very large on both machines. But the VmSize on CentOS 6.2(x86_64)+python2.6 is 140M-180M, while on my RHEL5.3+python2.6, the VmSize is just 6M. And I tested on CentOS 5.7(x86_64)+python2.4.3, the VmSize is 70M.

     I could understand that 64 bit machines will occupy more virtual memory than that on 32 bit, because the length of some types are not the same. But I don't know why they differs so greatly(6M to 180M), Or is this only caused by that python2.6 on CentOS 6.2's memory allocation is different from python's default one? Could you kindly give me some clues? Thank you very much.

Best Regards
William

[toc] | [next] | [standalone]

#50850

I found that it was caused by not by python but by /usr/lib/locale/locale-archive, the same problem as that described in
http://illiterat.livejournal.com/4615.html.

William

在 2013年7月18日星期四UTC+8下午12时45分01秒，William Bai写道：
> Hi:
> 
> 
> 
>    Previously, we found that our python scripts consume too much memory. So I use python's resource module to restrict RLIMIT_AS's soft limit and hard limit to 200M.
> 
> On my RHEL5.3(i386)+python2.6.2, it works OK. But on CentOS 6.2(x86_64)+python2.6.6, it reports memory error(exceeding 200M).
> 
> 
> 
>     And I tested with a very small script, and result is out of my expect, it still use too much memory on my CentOS 6.2 python:
> 
> import time
> 
> time.sleep(200)
> 
> 
> 
>     I use guppy and memory_profiler to see the memory usage and see that python objects just use about 6M memory both on RHEL5.3(i386)+python2.6 and CentOS 6.2(x86_64)+python2.6. But when I cat /proc/<pid>/status. I found that though VmRss is not very large on both machines. But the VmSize on CentOS 6.2(x86_64)+python2.6 is 140M-180M, while on my RHEL5.3+python2.6, the VmSize is just 6M. And I tested on CentOS 5.7(x86_64)+python2.4.3, the VmSize is 70M.
> 
> 
> 
>      I could understand that 64 bit machines will occupy more virtual memory than that on 32 bit, because the length of some types are not the same. But I don't know why they differs so greatly(6M to 180M), Or is this only caused by that python2.6 on CentOS 6.2's memory allocation is different from python's default one? Could you kindly give me some clues? Thank you very much.
> 
> 
> 
> Best Regards
> 
> William

[toc] | [prev] | [next] | [standalone]

#50857

On 07/18/2013 03:13 AM, William Bai wrote:
> I found that it was caused by not by python but by
> /usr/lib/locale/locale-archive, the same problem as that described
> in http://illiterat.livejournal.com/4615.html.

Too funny.  So in other words there isn't a problem at all.  What you
thought was RAM usage was really just a memory-mapped file.  That's why
Antoine said that using VSS to determine memory usage is not valid at
all in determining memory footprint.  VSS shows all kinds of things from
shared libraries to locale archives that have zero impact on RAM usage.
 Every app will show at least the size of glibc in its VSS number.

What is "too much memory" anyway?  What do you mean by "consume too much
memory?"  Now if your script has a resource leak and is long-running,
then that's a problem, but the solution is to fix your resource leak,
not have the OS kill your app when it exceeds the RLIMIT.

[toc] | [prev] | [next] | [standalone]

#50851

<snowingbear <at> gmail.com> writes:
> 
> Hi:
> 
>    Previously, we found that our python scripts consume too much memory.
So I use 
python's resource module to
> restrict RLIMIT_AS's soft limit and hard limit to 200M.
> On my RHEL5.3(i386)+python2.6.2, it works OK. But on CentOS 
6.2(x86_64)+python2.6.6, it reports memory
> error(exceeding 200M).

Take a look at http://www.selenic.com/smem/ for accurate measurement of
actual memory consumption under Linux. Virtual memory size is generally
useless for this purpose.

Regards

Antoine.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web