Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.soft-sys.math.mathematica > #14436 > unrolled thread

Simple program leaking memory - why?

Started bygredner@gmail.com
First post2013-05-07 07:53 +0000
Last post2013-05-13 07:47 +0000
Articles 5 — 4 participants

Back to article view | Back to comp.soft-sys.math.mathematica


Contents

  Simple program leaking memory - why? gredner@gmail.com - 2013-05-07 07:53 +0000
    Re: Simple program leaking memory - why? Joe Gwinn <joegwinn@comcast.net> - 2013-05-08 08:10 +0000
      Re: Simple program leaking memory - why? Gabriel Redner <gredner@gmail.com> - 2013-05-09 07:56 +0000
        Re: Simple program leaking memory - why? Joe Gwinn <joegwinn@comcast.net> - 2013-05-11 06:27 +0000
    Re: Simple program leaking memory - why? Roland Franzius <roland.franzius@uos.de> - 2013-05-13 07:47 +0000

#14436 — Simple program leaking memory - why?

Fromgredner@gmail.com
Date2013-05-07 07:53 +0000
SubjectSimple program leaking memory - why?
Message-ID<kmabu3$qdj$1@smc.vnet.net>
Hi folks,

I'm fighting an issue in which a simple Mathematica program to export a bunch of plots is leaking a huge amount of memory - enough to exhaust the 12G available on my machine.  The problem is, it looks like the program should run in bounded memory!  A simplified version of my code which demonstrates the problem follows:

====================
n = 10000;
(* just some bogus data *)
makeData[_] := Table[{{x, y} = RandomReal[{-3, 3}, {2}], {y, -x}}, {200}];
data = Array[makeData, n];
ParallelMap[
  Export[
    "movie-" <> IntegerString[#, 10, 4] <> ".png",
    ListVectorPlot[data[[#]]]] &,
  Range[1, n]];
====================

The problem occurs whether the loop is Map or Do, Parallel- or not.  It occurs in Mathematica 8 and 9, both under Linux.

The amount of memory leaked per frame is considerable - approximately .6 MB in my tests.  I don't see where it could be going, unless Export is caching every byte which passes through it!

I've also asked this question on stackexchange [1] but so far nobody has a clue.

Thanks,
-Gabe

[1] http://mathematica.stackexchange.com/questions/24461/why-does-this-simple-program-leak-memory

[toc] | [next] | [standalone]


#14451

FromJoe Gwinn <joegwinn@comcast.net>
Date2013-05-08 08:10 +0000
Message-ID<kmd1ag$4km$1@smc.vnet.net>
In reply to#14436
In article <kmabu3$qdj$1@smc.vnet.net>, <gredner@gmail.com> wrote:

> Hi folks,
> 
> I'm fighting an issue in which a simple Mathematica program to export a bunch
> of plots is leaking a huge amount of memory - enough to exhaust the 12G
> available on my machine.  The problem is, it looks like the program should
> run in bounded memory!  A simplified version of my code which demonstrates
> the problem follows:
> 
> ====================
> n = 10000;
> (* just some bogus data *)
> makeData[_] := Table[{{x, y} = RandomReal[{-3, 3}, {2}], {y, -x}}, {200}];
> data = Array[makeData, n];
> ParallelMap[
>   Export[
>     "movie-" <> IntegerString[#, 10, 4] <> ".png",
>     ListVectorPlot[data[[#]]]] &,
>   Range[1, n]];
> ====================
> 
> The problem occurs whether the loop is Map or Do, Parallel- or not.  It
> occurs in Mathematica 8 and 9, both under Linux.
> 
> The amount of memory leaked per frame is considerable - approximately .6 MB
> in my tests.  I don't see where it could be going, unless Export is caching
> every byte which passes through it!
> 
> I've also asked this question on stackexchange [1] but so far nobody has a
> clue.
> 
> Thanks,
> -Gabe
> 
> [1] http://mathematica.stackexchange.com/questions/24461/why-does-this-simple-program-leak-memory
> 

Mathematica keeps all In and Out cells, and this can look like a leak. 
Set $HistoryLength to zero and see if that helps.

There was a thread on this in September 2012 titled "Kernel exits (for
lack of memory)" or just "Kernel exits".

Joe Gwinn

[toc] | [prev] | [next] | [standalone]


#14457

FromGabriel Redner <gredner@gmail.com>
Date2013-05-09 07:56 +0000
Message-ID<kmfks5$bqt$1@smc.vnet.net>
In reply to#14451
Hi Joe,

I don't see how this could be the issue.  All of the statements have semicolons after them, and in any case I've tried changing $HistoryLength with no effect.

Thanks,
-Gabe

[toc] | [prev] | [next] | [standalone]


#14467

FromJoe Gwinn <joegwinn@comcast.net>
Date2013-05-11 06:27 +0000
Message-ID<kmkobs$pbk$1@smc.vnet.net>
In reply to#14457
Gabe,

In article <kmfks5$bqt$1@smc.vnet.net>, Gabriel Redner
<gredner@gmail.com> wrote:

> Hi Joe,
> 
> I don't see how this could be the issue.  All of the statements have
> semicolons after them, and in any case I've tried changing $HistoryLength
> with no effect.

The semicolons prevent printing, not generation.

If you did $HistoryLength=0 early in the computations, with a clean
start (exit Mathematica and launch Mathematica) and the problem persists, then it is
probably some other cause.

The most general way to figure out what's going on is to divide the
notebook into two notebooks (communicating via files if necessary) and
see which half leaks.  Do this recursively.  If neither half leaks, the
cause is something in or near the cut point.  This approach is modeled
on binary search, and is very fast albeit manual.

Joe Gwinn

[toc] | [prev] | [next] | [standalone]


#14474

FromRoland Franzius <roland.franzius@uos.de>
Date2013-05-13 07:47 +0000
Message-ID<kmq5r8$ahp$1@smc.vnet.net>
In reply to#14436
Am 07.05.2013 09:53, schrieb gredner@gmail.com:
> Hi folks,
>
> I'm fighting an issue in which a simple Mathematica program to export a bunch of plots is leaking a huge amount of memory - enough to exhaust the 12G available on my machine.  The problem is, it looks like the program should run in bounded memory!  A simplified version of my code which demonstrates the problem follows:
>

FileNames["*.png"]
DeleteFiles["*.png"]

<<VectorFieldPlots`

> ====================
> n = 10000;
> (* just some bogus data *)
> makeData[_] := Table[{{x, y} = RandomReal[{-3, 3}, {2}], {y, -x}}, {200}];
> data = Array[makeData, n];
> ParallelMap[
>    Export[
>      "movie-" <> IntegerString[#, 10, 4] <> ".png",

strg="movie-" <> IntegerString[#, 10, 4] <> ".png";Print[strg];strg,

>      ListVectorPlot[data[[#]]]] &,

        ListVectorFieldPlot

>    Range[1, n]];
> ====================
>
> The problem occurs whether the loop is Map or Do, Parallel- or not.  It occurs in Mathematica 8 and 9, both under Linux.
>
> The amount of memory leaked per frame is considerable - approximately .6 MB in my tests.  I don't see where it could be going, unless Export is caching every byte which passes through it!
>
> I've also asked this question on stackexchange [1] but so far nobody has a clue.
>
> Thanks,
> -Gabe
>
> [1] http://mathematica.stackexchange.com/questions/24461/why-does-this-simple-program-leak-memory
>

I can't comment on Linux.

In Windows 7 the Windows Explorer interrupts, after generation of about 
1000 files, are stalling suddenly. The Task Manager is still working, 
but only on free parts of the screen. Overlapping windows will be cut in 
two.

You even cannot open  the %user%\Document directory in Windows Explorer, 
so I suspect the file name writing interface to the NTFS system  has 
generated an infinite loop on the medium or its map in memory. Possibly 
there may exist an 1024-overflow error in the file stack interface 
somewhere.

In Windows 7, one hase to delete the whole bunch of .png files using 
DeleteFiles[] in Mathematica or a del in a cmd window.

Since the problem seems to occur on two completely different systems 
there will be an error in the Link library of MathLink, Jlink or 
whatsoever.

-- 

Roland Franzius


[toc] | [prev] | [standalone]


Back to top | Article view | comp.soft-sys.math.mathematica


csiph-web