Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.soft-sys.math.mathematica > #10320

Re: Memory Blowup Issues

From Szabolcs <szhorvat@gmail.com>
Newsgroups comp.soft-sys.math.mathematica
Subject Re: Memory Blowup Issues
Date 2012-06-02 09:48 +0000
Organization Steven M. Christensen and Associates, Inc and MathTensor, Inc.
Message-ID <jqcnh2$174$1@smc.vnet.net> (permalink)
References <jqa1h7$bj7$1@smc.vnet.net>

Show all headers | View raw


On Friday, 1 June 2012 11:20:39 UTC+2, Kevin J. McCann  wrote:
> I am dealing with some good sized data sets from satellite data. These
> data are stored in hdf format. The problem is that I process a sequence
> of these data sets by Importing and then processing; however, it seems
> that the memory usage grows with each Import, even if I Remove the data.
>
> Here is a somewhat simplified example:
>
> MemoryInUse[] (* some data already read in *)
> 2781549920
>
> Data=Import[fileName,"Data"]; (* import a new data set *)
> MemoryInUse[]
> 3302168936
>
> Data=Import[fileName,"Data"];(* import  it again *)
> MemoryInUse[]
> 3822781920
>
> Remove[Data] (* this has no effect on the memory usage *)
> MemoryInUse[]
> 3822787840
>
> As you can see, I will shortly run out of memory. Any ideas?
>

Hello Kevin,

The most likely cause of this behaviour is that the intermediate results are being remembered by Mathematica through the Out symbol.  You probably know that you can refer to previous results using %, %%, %%% or Out[10], Out[11], etc. but you may not have realized that Out will get assigned a new value even if the last input ended in a semicolon.

Example:

In[1]:= data=RandomReal[1,10000]; (* output suppressed using semicolon *)

In[2]:= Length[%] (* note that % will have the same value as data *)

The solution is to set $HistoryLength = 0 before you start working, to prevent the system from remembering previous results.

Please see these articles as well:

http://mathematica.stackexchange.com/questions/3376/old-values-are-not-freed-garbage-collected-when-you-re-evaluate-an-assignment

http://mathematica.stackexchange.com/questions/4322/memory-not-freed-after-running-clear-when-using-a-table

Szabolcs

Back to comp.soft-sys.math.mathematica | Previous | Next — Previous in thread | Find similar | Unroll thread


Thread

Memory Blowup Issues "Kevin J. McCann" <kjm@KevinMcCann.com> - 2012-06-01 09:20 +0000
  Re: Memory Blowup Issues David Reiss <dbreiss@gmail.com> - 2012-06-02 09:46 +0000
  Re: Memory Blowup Issues Szabolcs <szhorvat@gmail.com> - 2012-06-02 09:48 +0000

csiph-web