Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!ecngs!feeder2.ecngs.de!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <davea@davea.name>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.009
X-Spam-Evidence: '*H*': 0.98; '*S*': 0.00; 'algorithm': 0.03; 'sys': 0.05; 'purpose.': 0.07; 'python': 0.09; 'compute': 0.09; 'suggest': 0.11; 'resulting': 0.13; 'correlation': 0.16; 'hangs': 0.16; 'numpy': 0.16; 'scipy,': 0.16; 'storing': 0.16; 'wrote:': 0.17; 'implementing': 0.17; 'memory': 0.18; 'math': 0.20; 'trying': 0.21; 'import': 0.21; 'fairly': 0.21; 'people,': 0.21; "haven't": 0.23; 'header:In-Reply-To:1': 0.25; 'header:User- Agent:1': 0.26; 'wrote': 0.26; 'am,': 0.27; 'implemented': 0.27; 'have,': 0.27; 'run': 0.28; 'piece': 0.29; 'array': 0.29; 'checked': 0.30; 'code': 0.31; 'file': 0.32; 'anybody': 0.32; 'computer.': 0.32; 'running': 0.32; 'ram': 0.33; 'handle': 0.33; 'to:addr:python-list': 0.33; 'hi,': 0.33; 'version': 0.34; 'done': 0.34; 'list': 0.35; 'but': 0.36; 'does': 0.37; 'data': 0.37; 'subject:: ': 0.38; 'some': 0.38; 'sure': 0.38; 'to:addr:python.org': 0.39; 'received:192': 0.39; 'received:192.168': 0.40; 'save': 0.61; 'situation': 0.62; 'between': 0.63; 'series': 0.63; 'due': 0.66; 'finally': 0.66; 'received:74.208': 0.71; 'saving': 0.72; 'dimensional': 0.84; 'received:74.208.4.194': 0.84; 'apparent': 0.91
Date: Mon, 18 Feb 2013 19:10:55 -0500
From: Dave Angel <davea@davea.name>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130106 Thunderbird/17.0.2
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: memory management
References: <acd5afba-fa4f-4736-b941-778c646e20bf@googlegroups.com>
In-Reply-To: <acd5afba-fa4f-4736-b941-778c646e20bf@googlegroups.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Provags-ID: V02:K0:SPF3GQt1q7EkTThs7Mw5a11h4K1x1Zk8wm6tmr5qz2L T7jpKl7nQcWq1+jhaTUbFbW/ZaRodW4GThHg40t8F9wW3xjE7q vmJnixfBR+PbX/5/MNbzUV9a2SNrKzwKx1yxDWVDwyeLNc1GCU J+qHQB0GeVmMz2Uhh37rYnlN6aKxhCHrrt1F4HMQe58cmoWiyR 6a7z6YL8EPcEsWWDRpi8XDcQCDvy4QNVwEH9WxFVmvBTO0UHFk 1iUh750GlLilVRQwdE8CE6VwkDVP8UxfI9s3qGcEcgduC5SB52 em3ULhTpnU7PkzszTZe3soOVK/6K8GJLylkgNMDu3ufIieUAg= =
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.1983.1361232670.2939.python-list@python.org>
Lines: 31
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1361232670 news.xs4all.nl 6913 [2001:888:2000:d::a6]:38890
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:39143

On 02/18/2013 10:29 AM, Sudheer Joseph wrote:
> HI,
>          I have been trying to compute cross correlation between a time series at a location f(1) and the timeseries of spatial data f(XYT) and saving the resulting correlation coefficients and lags in a 3 dimensional array which is of fairly big size. Though the code I made for this purpose works up to few iterations then it hangs due to apparent memory crunch. Can anybody suggest a better way to handle this situation so that the computation and data storing can be done with out hangups. Finally I intend to save the data as netcdf file which is not implemented as of now. Below is the piece of code I wrote for this purpose.
>

Python version and OS please.  And is the Python 32bit or 64bit?  How 
much RAM does the computer have, and how big are the swapfiles ?

"Fairly big" is fairly vague.  To some people, a list with 100k members 
is huge, but not to a modern computer.

How have you checked whether it's running out of memory?  Have you run 
'top' on it?  Or is that just a guess?

I haven't used numpy, scipy, nor matplotlib, and it's been a long time 
since I did correlations.  But are you sure you're not just implementing 
an O(n**3) algorithm or something, and it's just extremely slow?


> from mpl_toolkits.basemap import Basemap as bm, shiftgrid, cm
> import numpy as np
> import matplotlib.pyplot as plt
> from netCDF4 import Dataset
> from math import pow, sqrt
> import sys
> from scipy.stats import t

  <snip>

-- 
DaveA