Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.soft-sys.math.mathematica > #14436 > unrolled thread
| Started by | gredner@gmail.com |
|---|---|
| First post | 2013-05-07 07:53 +0000 |
| Last post | 2013-05-13 07:47 +0000 |
| Articles | 5 — 4 participants |
Back to article view | Back to comp.soft-sys.math.mathematica
Simple program leaking memory - why? gredner@gmail.com - 2013-05-07 07:53 +0000
Re: Simple program leaking memory - why? Joe Gwinn <joegwinn@comcast.net> - 2013-05-08 08:10 +0000
Re: Simple program leaking memory - why? Gabriel Redner <gredner@gmail.com> - 2013-05-09 07:56 +0000
Re: Simple program leaking memory - why? Joe Gwinn <joegwinn@comcast.net> - 2013-05-11 06:27 +0000
Re: Simple program leaking memory - why? Roland Franzius <roland.franzius@uos.de> - 2013-05-13 07:47 +0000
| From | gredner@gmail.com |
|---|---|
| Date | 2013-05-07 07:53 +0000 |
| Subject | Simple program leaking memory - why? |
| Message-ID | <kmabu3$qdj$1@smc.vnet.net> |
Hi folks,
I'm fighting an issue in which a simple Mathematica program to export a bunch of plots is leaking a huge amount of memory - enough to exhaust the 12G available on my machine. The problem is, it looks like the program should run in bounded memory! A simplified version of my code which demonstrates the problem follows:
====================
n = 10000;
(* just some bogus data *)
makeData[_] := Table[{{x, y} = RandomReal[{-3, 3}, {2}], {y, -x}}, {200}];
data = Array[makeData, n];
ParallelMap[
Export[
"movie-" <> IntegerString[#, 10, 4] <> ".png",
ListVectorPlot[data[[#]]]] &,
Range[1, n]];
====================
The problem occurs whether the loop is Map or Do, Parallel- or not. It occurs in Mathematica 8 and 9, both under Linux.
The amount of memory leaked per frame is considerable - approximately .6 MB in my tests. I don't see where it could be going, unless Export is caching every byte which passes through it!
I've also asked this question on stackexchange [1] but so far nobody has a clue.
Thanks,
-Gabe
[1] http://mathematica.stackexchange.com/questions/24461/why-does-this-simple-program-leak-memory
[toc] | [next] | [standalone]
| From | Joe Gwinn <joegwinn@comcast.net> |
|---|---|
| Date | 2013-05-08 08:10 +0000 |
| Message-ID | <kmd1ag$4km$1@smc.vnet.net> |
| In reply to | #14436 |
In article <kmabu3$qdj$1@smc.vnet.net>, <gredner@gmail.com> wrote:
> Hi folks,
>
> I'm fighting an issue in which a simple Mathematica program to export a bunch
> of plots is leaking a huge amount of memory - enough to exhaust the 12G
> available on my machine. The problem is, it looks like the program should
> run in bounded memory! A simplified version of my code which demonstrates
> the problem follows:
>
> ====================
> n = 10000;
> (* just some bogus data *)
> makeData[_] := Table[{{x, y} = RandomReal[{-3, 3}, {2}], {y, -x}}, {200}];
> data = Array[makeData, n];
> ParallelMap[
> Export[
> "movie-" <> IntegerString[#, 10, 4] <> ".png",
> ListVectorPlot[data[[#]]]] &,
> Range[1, n]];
> ====================
>
> The problem occurs whether the loop is Map or Do, Parallel- or not. It
> occurs in Mathematica 8 and 9, both under Linux.
>
> The amount of memory leaked per frame is considerable - approximately .6 MB
> in my tests. I don't see where it could be going, unless Export is caching
> every byte which passes through it!
>
> I've also asked this question on stackexchange [1] but so far nobody has a
> clue.
>
> Thanks,
> -Gabe
>
> [1] http://mathematica.stackexchange.com/questions/24461/why-does-this-simple-program-leak-memory
>
Mathematica keeps all In and Out cells, and this can look like a leak.
Set $HistoryLength to zero and see if that helps.
There was a thread on this in September 2012 titled "Kernel exits (for
lack of memory)" or just "Kernel exits".
Joe Gwinn
[toc] | [prev] | [next] | [standalone]
| From | Gabriel Redner <gredner@gmail.com> |
|---|---|
| Date | 2013-05-09 07:56 +0000 |
| Message-ID | <kmfks5$bqt$1@smc.vnet.net> |
| In reply to | #14451 |
Hi Joe, I don't see how this could be the issue. All of the statements have semicolons after them, and in any case I've tried changing $HistoryLength with no effect. Thanks, -Gabe
[toc] | [prev] | [next] | [standalone]
| From | Joe Gwinn <joegwinn@comcast.net> |
|---|---|
| Date | 2013-05-11 06:27 +0000 |
| Message-ID | <kmkobs$pbk$1@smc.vnet.net> |
| In reply to | #14457 |
Gabe, In article <kmfks5$bqt$1@smc.vnet.net>, Gabriel Redner <gredner@gmail.com> wrote: > Hi Joe, > > I don't see how this could be the issue. All of the statements have > semicolons after them, and in any case I've tried changing $HistoryLength > with no effect. The semicolons prevent printing, not generation. If you did $HistoryLength=0 early in the computations, with a clean start (exit Mathematica and launch Mathematica) and the problem persists, then it is probably some other cause. The most general way to figure out what's going on is to divide the notebook into two notebooks (communicating via files if necessary) and see which half leaks. Do this recursively. If neither half leaks, the cause is something in or near the cut point. This approach is modeled on binary search, and is very fast albeit manual. Joe Gwinn
[toc] | [prev] | [next] | [standalone]
| From | Roland Franzius <roland.franzius@uos.de> |
|---|---|
| Date | 2013-05-13 07:47 +0000 |
| Message-ID | <kmq5r8$ahp$1@smc.vnet.net> |
| In reply to | #14436 |
Am 07.05.2013 09:53, schrieb gredner@gmail.com:
> Hi folks,
>
> I'm fighting an issue in which a simple Mathematica program to export a bunch of plots is leaking a huge amount of memory - enough to exhaust the 12G available on my machine. The problem is, it looks like the program should run in bounded memory! A simplified version of my code which demonstrates the problem follows:
>
FileNames["*.png"]
DeleteFiles["*.png"]
<<VectorFieldPlots`
> ====================
> n = 10000;
> (* just some bogus data *)
> makeData[_] := Table[{{x, y} = RandomReal[{-3, 3}, {2}], {y, -x}}, {200}];
> data = Array[makeData, n];
> ParallelMap[
> Export[
> "movie-" <> IntegerString[#, 10, 4] <> ".png",
strg="movie-" <> IntegerString[#, 10, 4] <> ".png";Print[strg];strg,
> ListVectorPlot[data[[#]]]] &,
ListVectorFieldPlot
> Range[1, n]];
> ====================
>
> The problem occurs whether the loop is Map or Do, Parallel- or not. It occurs in Mathematica 8 and 9, both under Linux.
>
> The amount of memory leaked per frame is considerable - approximately .6 MB in my tests. I don't see where it could be going, unless Export is caching every byte which passes through it!
>
> I've also asked this question on stackexchange [1] but so far nobody has a clue.
>
> Thanks,
> -Gabe
>
> [1] http://mathematica.stackexchange.com/questions/24461/why-does-this-simple-program-leak-memory
>
I can't comment on Linux.
In Windows 7 the Windows Explorer interrupts, after generation of about
1000 files, are stalling suddenly. The Task Manager is still working,
but only on free parts of the screen. Overlapping windows will be cut in
two.
You even cannot open the %user%\Document directory in Windows Explorer,
so I suspect the file name writing interface to the NTFS system has
generated an infinite loop on the medium or its map in memory. Possibly
there may exist an 1024-overflow error in the file stack interface
somewhere.
In Windows 7, one hase to delete the whole bunch of .png files using
DeleteFiles[] in Mathematica or a del in a cmd window.
Since the problem seems to occur on two completely different systems
there will be an error in the Link library of MathLink, Jlink or
whatsoever.
--
Roland Franzius
[toc] | [prev] | [standalone]
Back to top | Article view | comp.soft-sys.math.mathematica
csiph-web