memory management

X-Received	by 10.66.72.3 with SMTP id z3mr2392586pau.40.1361201353529; Mon, 18 Feb 2013 07:29:13 -0800 (PST)
X-Received	by 10.50.150.180 with SMTP id uj20mr1419265igb.7.1361201353299; Mon, 18 Feb 2013 07:29:13 -0800 (PST)
Path	csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.glorb.com!usenet.stanford.edu!su1no13111658pbb.0!news-out.google.com!ov8ni9pbb.1!nntp.google.com!su1no13111655pbb.0!postnews.google.com!glegroupsg2000goo.googlegroups.com!not-for-mail
Newsgroups	comp.lang.python
Date	Mon, 18 Feb 2013 07:29:13 -0800 (PST)
Complaints-To	groups-abuse@google.com
Injection-Info	glegroupsg2000goo.googlegroups.com; posting-host=117.195.165.123; posting-account=OK8l-QoAAABUSmU-80MsG3xZmK8hhJMB
NNTP-Posting-Host	117.195.165.123
User-Agent	G2/1.0
MIME-Version	1.0
Message-ID	<acd5afba-fa4f-4736-b941-778c646e20bf@googlegroups.com> (permalink)
Subject	memory management
From	Sudheer Joseph <sjo.india@gmail.com>
Injection-Date	Mon, 18 Feb 2013 15:29:13 +0000
Content-Type	text/plain; charset=ISO-8859-1
Content-Transfer-Encoding	quoted-printable
Xref	csiph.com comp.lang.python:39104

Show key headers only | View raw

HI,
        I have been trying to compute cross correlation between a time series at a location f(1) and the timeseries of spatial data f(XYT) and saving the resulting correlation coefficients and lags in a 3 dimensional array which is of fairly big size. Though the code I made for this purpose works up to few iterations then it hangs due to apparent memory crunch. Can anybody suggest a better way to handle this situation so that the computation and data storing can be done with out hangups. Finally I intend to save the data as netcdf file which is not implemented as of now. Below is the piece of code I wrote for this purpose.

from mpl_toolkits.basemap import Basemap as bm, shiftgrid, cm
import numpy as np
import matplotlib.pyplot as plt
from netCDF4 import Dataset
from math import pow, sqrt
import sys
from scipy.stats import t
indep=120
nlags=365
ncin = Dataset('qu_ru.nc', 'r')
lons = ncin.variables['LON421_600'][:]
lats = ncin.variables['LAT81_220'][:]
dep = ncin.variables['DEPTH1_29'][:]
adep=(dep==indep).nonzero()
didx=int(adep[0])
qu = ncin.variables['qu'][:,:,:]
#qv = ncin.variables['QV'][0,:,:]
ru = ncin.variables['ru'][:,didx,0,0]
ncin.close()
fig = plt.figure()
ax = fig.add_axes([0.1,0.1,0.8,0.8])
# use major and minor sphere radii from WGS84 ellipsoid.
m = bm(projection='cyl', llcrnrlon=30, llcrnrlat=-40,urcrnrlon=120, urcrnrlat=30)
# transform to nx x ny regularly spaced 5km native projection grid
nx = int((m.xmax-m.xmin))+1; ny = int((m.ymax-m.ymin)+1)
q=ru[1:2190]
qmean=np.mean(q)
qstd=np.std(q)
qnorm=(q-qmean)/qstd
lags3d=np.arange(731*140*180).reshape(731,140,180)
r3d=np.arange(731*140*180).reshape(731,140,180)
for i in np.arange(len(lons)):
   for j in np.arange(len(lats)):
      print i,j
      p=qu[1:2190,j,i].squeeze()
      p.shape
      pmean=np.mean(p)
      pstd=np.std(p)
      pnorm=(p-pmean)/pstd
      n=len(p)
#      fg=plt.figure()
      c=plt.xcorr(p,q,usevlines=True,maxlags=nlags,normed=True,lw=2)
      acp=plt.acorr(p,usevlines=True,maxlags=nlags,normed=True,lw=2)
      acq=plt.acorr(q,usevlines=True,maxlags=nlags,normed=True,lw=2)
      acp[1][nlags]=0
      acq[1][nlags]=0
      lags=c[0]
      r=c[1]
      lags3d[:,j,i]=lags
      r3d[:,j,i]=r

Back to comp.lang.python | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

memory management Sudheer Joseph <sjo.india@gmail.com> - 2013-02-18 07:29 -0800
  Re: memory management Dave Angel <davea@davea.name> - 2013-02-18 19:10 -0500
    Re: memory management Sudheer Joseph <sjo.india@gmail.com> - 2013-02-18 17:13 -0800
    Re: memory management Sudheer Joseph <sjo.india@gmail.com> - 2013-02-18 17:13 -0800

csiph-web