Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.soft-sys.math.mathematica > #10337 > unrolled thread

Re: Memory Blowup Issues

Started byRalph Dratman <ralph.dratman@gmail.com>
First post2012-06-03 09:03 +0000
Last post2012-06-07 09:19 +0000
Articles 4 — 4 participants

Back to article view | Back to comp.soft-sys.math.mathematica

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.


Contents

  Re: Memory Blowup Issues Ralph Dratman <ralph.dratman@gmail.com> - 2012-06-03 09:03 +0000
    Re: Memory Blowup Issues Szabolcs Horvát <szhorvat@gmail.com> - 2012-06-05 08:51 +0000
    Re: Memory Blowup Issues David Bailey <dave@removedbailey.co.uk> - 2012-06-06 08:50 +0000
    Re: Memory Blowup Issues W Craig Carter <ccarter@mit.edu> - 2012-06-07 09:19 +0000

#10337 — Re: Memory Blowup Issues

FromRalph Dratman <ralph.dratman@gmail.com>
Date2012-06-03 09:03 +0000
SubjectRe: Memory Blowup Issues
Message-ID<jqf98n$a65$1@smc.vnet.net>
Kevin,

Thank you for passing along those tips. I usually put $HistoryLength=1
and also turn off "Enable Notebook History Tracking" in the
Preferences dialog -- though I'm not sure the latter makes any
difference.

I just want to re-emphasize how egregious I think the situation with
crashing really is. It is unacceptable. I hope everyone is making that
absolutely clear to WRI. How can we continue to recommend a program
which often crashes? It defies common sense.

Ralph


On Sat, Jun 2, 2012 at 9:45 AM, Kevin J. McCann <kjm@kevinmccann.com> wrote:
> I already have 12Gb; so, that is not a solution. Others have suggested
> $HistoryLength = 0, which works. On the other hand, mvary = . did not change
> the amount of memory in use at all.
>
> I guess acceptability of the memory/crash problem depends on your point of
> view. If indeed the ability to access previous outputs is important to the
> way you work (it is not to me), then I can see why it needs to be there even
> though it could lead to a crash in extreme cases like mine. However, the
> ability to limit this with $HistoryLength gives flexibility to the user.
>
> I do not make use of the whole In[] Out[] thing, and, in fact, turn those
> labels off in all my notebooks.
>
> Kevin
>
> On 6/2/2012 5:48 AM, Ralph Dratman wrote:
>>
>> Solution: install more memory in your computer?
>>
>> That answer is sort of a wisecrack, but in real life, my Mac Mini now
>> has 8 GB of RAM for the benefit of Mathematica -- yet the program
>> still somehow manages to fill up memory and crash. That is why I keep
>> glancing at the desktop memory monitor. If memory is almost full, I
>> can usually stop Mathematica before things get really serious.
>>
>> That said, my experience has been that disposing of variables (using
>> myvar=.) does help quite a bit. But I am not reading files very often.
>> In your case, maybe something about reading a file is making it harder
>> to free the memory used.
>>
>> By the way, in case you might be thinking that this whole crash
>> problem is pretty unacceptable behavior for an app that sells for a
>> lot of money in 2012.... yeah, you're right, it is.
>>
>> Ralph
>>
>>
>>
>> On Fri, Jun 1, 2012 at 5:18 AM, Kevin J. McCann<kjm@kevinmccann.com>
>>  wrote:
>>>
>>> I am dealing with some good sized data sets from satellite data. These
>>> data are stored in hdf format. The problem is that I process a sequence
>>> of these data sets by Importing and then processing; however, it seems
>>> that the memory usage grows with each Import, even if I Remove the data.
>>>
>>> Here is a somewhat simplified example:
>>>
>>> MemoryInUse[] (* some data already read in *)
>>> 2781549920
>>>
>>> Data=Import[fileName,"Data"]; (* import a new data set *)
>>> MemoryInUse[]
>>> 3302168936
>>>
>>> Data=Import[fileName,"Data"];(* import  it again *)
>>> MemoryInUse[]
>>> 3822781920
>>>
>>> Remove[Data] (* this has no effect on the memory usage *)
>>> MemoryInUse[]
>>> 3822787840
>>>
>>> As you can see, I will shortly run out of memory. Any ideas?
>>>
>>> Thanks,
>>>
>>> Kevin
>>>
>>
>

[toc] | [next] | [standalone]


#10356

FromSzabolcs Horvát <szhorvat@gmail.com>
Date2012-06-05 08:51 +0000
Message-ID<jqkhal$4jv$1@smc.vnet.net>
In reply to#10337
On 2012.06.03. 11:03, Ralph Dratman wrote:
> Kevin,
>
> Thank you for passing along those tips. I usually put $HistoryLength=1
> and also turn off "Enable Notebook History Tracking" in the
> Preferences dialog -- though I'm not sure the latter makes any
> difference.
>

Hello Ralph,

"Notebook History Tracking" only affects what is saved with the notebook 
you are working with, but it does not affect how the kernel works or 
what it keeps in memory (i.e. doesn't appreciably affect memory use). 
This option will simply record the times at which who have created or 
edited each cell.  If you open a notebook file with a text editor, 
you'll see this information in the CellChangeTimes options.

Turning it off will slightly reduce notebook sizes and makes version 
tracking much more convenient.

-- 
Szabolcs Horvát
Visit Mathematica.SE:  http://mathematica.stackexchange.com/

[toc] | [prev] | [next] | [standalone]


#10362

FromDavid Bailey <dave@removedbailey.co.uk>
Date2012-06-06 08:50 +0000
Message-ID<jqn5jr$hdi$1@smc.vnet.net>
In reply to#10337
On 03/06/2012 10:03, Ralph Dratman wrote:
> Kevin,
>
> Thank you for passing along those tips. I usually put $HistoryLength=1
> and also turn off "Enable Notebook History Tracking" in the
> Preferences dialog -- though I'm not sure the latter makes any
> difference.
>
> I just want to re-emphasize how egregious I think the situation with
> crashing really is. It is unacceptable. I hope everyone is making that
> absolutely clear to WRI. How can we continue to recommend a program
> which often crashes? It defies common sense.
>


To be fair to WRI, I think the problem is that you have a very complex 
piece of software that is partly programmed by users. The kernel does 
put out a message if it runs out of memory, so it does detect the 
problem (at least under Windows!), but leaving the software in a well 
defined and useful state is another matter. The FrontEnd should remain 
running.

It is also possible to run calculations using MemoryConstrained.

Assuming the data you are reading in  is an array or real numbers (or of 
integers), you should check if Import returns a packed array, and pack 
it if necessary - this will save a lot of space.

David Bailey
http://www.dbaileyconsultancy.co.uk

[toc] | [prev] | [next] | [standalone]


#10370

FromW Craig Carter <ccarter@mit.edu>
Date2012-06-07 09:19 +0000
Message-ID<jqprmg$2cm$1@smc.vnet.net>
In reply to#10337
Hello,

I find the following function to be a useful way to extract information 
for memory management:

checkMemory[]:=Grid@Sort[{ByteCount[ToExpression[#]], HoldForm[#]} & 
/@ Names["Global`*"], (First[#1] > First[#2]) &]

This returns a list of symbols ranked by their ByteCount.  However, I am 
not sure if evaluating checkMemory causes a momentary doubling of memory 
for each evaluation of ByteCount=85.

I thought I would pass this along for anyone who might be hunting for 
such a thing=85


W Craig Carter
Professor of Materials Science, MIT



On Jun 6, , at Wed Jun 6, 12 @4:52 AM, David Bailey wrote:

> On 03/06/2012 10:03, Ralph Dratman wrote:
>> Kevin,
>>
>> Thank you for passing along those tips. I usually put $HistoryLength=1
>> and also turn off "Enable Notebook History Tracking" in the
>> Preferences dialog -- though I'm not sure the latter makes any
>> difference.
>>
>> I just want to re-emphasize how egregious I think the situation with
>> crashing really is. It is unacceptable. I hope everyone is making that
>> absolutely clear to WRI. How can we continue to recommend a program
>> which often crashes? It defies common sense.
>>
>
>
> To be fair to WRI, I think the problem is that you have a very complex
> piece of software that is partly programmed by users. The kernel does
> put out a message if it runs out of memory, so it does detect the
> problem (at least under Windows!), but leaving the software in a well
> defined and useful state is another matter. The FrontEnd should remain
> running.
>
> It is also possible to run calculations using MemoryConstrained.
>
> Assuming the data you are reading in  is an array or real numbers (or of
> integers), you should check if Import returns a packed array, and pack
> it if necessary - this will save a lot of space.
>
> David Bailey
> http://www.dbaileyconsultancy.co.uk
>

[toc] | [prev] | [standalone]


Back to top | Article view | comp.soft-sys.math.mathematica


csiph-web