Path: csiph.com!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder2.enfer-du-nord.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <sjo.india@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.001
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'algorithm': 0.03; 'output': 0.04; 'sys': 0.05; 'python': 0.09; 'grid': 0.09; 'internally': 0.09; 'normally,': 0.09; 'res': 0.09; 'to:addr:comp.lang.python': 0.09; 'cc:addr:python-list': 0.10; '2.7.3': 0.16; '4gb': 0.16; 'correlation': 0.16; 'numpy': 0.16; 'scipy,': 0.16; 'implementing': 0.17; 'memory': 0.18; 'math': 0.20; 'define': 0.20; 'suggested': 0.20; 'bit': 0.21; 'import': 0.21; 'fairly': 0.21; 'people,': 0.21; 'cc:2**0': 0.23; "haven't": 0.23; 'cc:no real name:2**0': 0.24; 'command': 0.24; 'cc:addr:python.org': 0.25; 'header:In-Reply-To:1': 0.25; 'header :User-Agent:1': 0.26; 'have,': 0.27; 'run': 0.28; 'arrays': 0.29; 'points': 0.29; 'that.': 0.30; 'checked': 0.30; 'file': 0.32; 'computer.': 0.32; 'running': 0.32; 'ram': 0.33; 'ubuntu': 0.33; 'version': 0.34; 'received:google.com': 0.34; 'done': 0.34; 'list': 0.35; 'received:209.85.220': 0.35; 'received:209.85': 0.35; 'but': 0.36; 'loaded': 0.36; 'does': 0.37; 'listing': 0.37; 'received:209': 0.37; 'data': 0.37; 'subject:: ': 0.38; 'store': 0.38; 'some': 0.38; 'sure': 0.38; 'time,': 0.62; 'here': 0.65; '140': 0.84; '180': 0.84; 'latitude': 0.84; 'longitude': 0.84
X-Received: by 10.50.10.138 with SMTP id i10mr1710457igb.2.1361236412572; Mon, 18 Feb 2013 17:13:32 -0800 (PST)
Newsgroups: comp.lang.python
Date: Mon, 18 Feb 2013 17:13:32 -0800 (PST)
In-Reply-To: <mailman.1983.1361232670.2939.python-list@python.org>
Complaints-To: groups-abuse@google.com
Injection-Info: glegroupsg2000goo.googlegroups.com; posting-host=117.195.176.151; posting-account=OK8l-QoAAABUSmU-80MsG3xZmK8hhJMB
References: <acd5afba-fa4f-4736-b941-778c646e20bf@googlegroups.com> <mailman.1983.1361232670.2939.python-list@python.org>
User-Agent: G2/1.0
X-Google-Web-Client: true
X-Google-IP: 117.195.176.151
MIME-Version: 1.0
Subject: Re: memory management
From: Sudheer Joseph <sjo.india@gmail.com>
To: comp.lang.python@googlegroups.com
Content-Type: text/plain; charset=ISO-8859-1
Cc: python-list@python.org
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Message-ID: <mailman.1988.1361236422.2939.python-list@python.org>
Lines: 62
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1361236422 news.xs4all.nl 6947 [2001:888:2000:d::a6]:41155
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:39153

> Python version and OS please.  And is the Python 32bit or 64bit?  How 
> 
> much RAM does the computer have, and how big are the swapfiles ?
> 
Python 2.7.3
ubuntu 12.04 64 bit
4GB RAM
> 
> "Fairly big" is fairly vague.  To some people, a list with 100k members 
> 
> is huge, but not to a modern computer.
I have a data loaded to memory from netcdf file which is 2091*140*180 grid points (2091 time, 140 latitude 180 longitude) apart from this I define a 2 3d arrays r3d and lags3d to store the output for writing out to netcdf file after completion. 
> 
> 
> How have you checked whether it's running out of memory?  Have you run 
> 
> 'top' on it?  Or is that just a guess?

I have not done this but the speed (assessed from the listing of grid i and j) get stopped after j=6 ie after running 6 longitude grids)
>
Will check the top as you suggested

Here is the result of top it used about 3gB memory

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 3069 sjo       20   0 3636m 3.0g 2504 D    3 78.7   3:07.44 python  
> 
> I haven't used numpy, scipy, nor matplotlib, and it's been a long time 
> 
> since I did correlations.  But are you sure you're not just implementing 
> 
> an O(n**3) algorithm or something, and it's just extremely slow?
> 
Correlation do not involve such computation normally, I am not sure if internally python does some thing like that.
with best regards,
Sudheer
> 
> 
> 
> > from mpl_toolkits.basemap import Basemap as bm, shiftgrid, cm
> 
> > import numpy as np
> 
> > import matplotlib.pyplot as plt
> 
> > from netCDF4 import Dataset
> 
> > from math import pow, sqrt
> 
> > import sys
> 
> > from scipy.stats import t
> 
> 
> 
>   <snip>
> 
> 
> 
> -- 
> 
> DaveA