Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.soft-sys.math.mathematica > #4476 > unrolled thread

Memory when reding in data

Started byPeter Rodenbach <peter.rodenbach@googlemail.com>
First post2011-08-19 10:39 +0000
Last post2011-08-21 09:36 +0000
Articles 3 — 3 participants

Back to article view | Back to comp.soft-sys.math.mathematica


Contents

  Memory when reding in data Peter Rodenbach <peter.rodenbach@googlemail.com> - 2011-08-19 10:39 +0000
    Re: Memory when reding in data "Alexey Popkov" <lehin.p@gmail.com> - 2011-08-20 10:22 +0000
    Re: Memory when reding in data David Bailey <dave@removedbailey.co.uk> - 2011-08-21 09:36 +0000

#4476 — Memory when reding in data

FromPeter Rodenbach <peter.rodenbach@googlemail.com>
Date2011-08-19 10:39 +0000
SubjectMemory when reding in data
Message-ID<j2lehm$ngv$1@smc.vnet.net>
Hi everyone.

I am analysing image data and I have the problem that after about 50
images or so Mathematica quits the calculation with the hint out of
memory. The interesting part of the code i wrote is :

(*Import the first image into Maxvals, which will be compared to all \
other images and overwritten if any point is found to be larger*)

SetDirectory[
  "E:\\...];

Initialise = OpenRead["s2_260_0000.edf", BinaryFormat -> True];
MaxValues =
  BinaryRead["s2_260_0000.edf",
   Table["UnsignedInteger16", {512}, {512}]];
counter = Dimensions[MaxValues];

(*Load all experimental images and perform maximum projection*)

MaxProjection[filename_] := {OpenRead[filename, BinaryFormat -> True],
   Imagevalues =
   BinaryRead[filename, Table["UnsignedInteger16", {512}, {512}]],
  For[i = 1, i <= counter[[1]], i++,
   	For [j = 1, j <= counter[[2]], j++,
    If[MaxValues[[i, j]] >= Imagevalues[[i, j]],
     MaxValues[[i, j]] = MaxValues[[i, j]],
     MaxValues[[i, j]] = Imagevalues[[i, j]]]
    	]
   ], Close[Imagevalues]}


MaxProjection[#] & /@ FileNames@"*edf"

So I guess Mathm saves all the data of all the images after each
other. But once the comparision is done with the one MaxProjection
Image I want to have, all the data can be dumped from the memory and
then the next image can be compared with this MaxProjection image.

Does anyone have an idea? Thx.

[toc] | [next] | [standalone]


#4491

From"Alexey Popkov" <lehin.p@gmail.com>
Date2011-08-20 10:22 +0000
Message-ID<j2o1to$4v5$1@smc.vnet.net>
In reply to#4476
Try to evaluate

$HistoryLenght=0;

and Clear variables when their values are no longer needed.


"Peter Rodenbach" <peter.rodenbach@googlemail.com> wrote: 
news:j2lehm$ngv$1@smc.vnet.net...
> Hi everyone.
>
> I am analysing image data and I have the problem that after about 50
> images or so Mathematica quits the calculation with the hint out of
> memory. The interesting part of the code i wrote is :
>
> (*Import the first image into Maxvals, which will be compared to all \
> other images and overwritten if any point is found to be larger*)
>
> SetDirectory[
>  "E:\\...];
>
> Initialise = OpenRead["s2_260_0000.edf", BinaryFormat -> True];
> MaxValues =
>  BinaryRead["s2_260_0000.edf",
>   Table["UnsignedInteger16", {512}, {512}]];
> counter = Dimensions[MaxValues];
>
> (*Load all experimental images and perform maximum projection*)
>
> MaxProjection[filename_] := {OpenRead[filename, BinaryFormat -> True],
>   Imagevalues =
>   BinaryRead[filename, Table["UnsignedInteger16", {512}, {512}]],
>  For[i = 1, i <= counter[[1]], i++,
>   For [j = 1, j <= counter[[2]], j++,
>    If[MaxValues[[i, j]] >= Imagevalues[[i, j]],
>     MaxValues[[i, j]] = MaxValues[[i, j]],
>     MaxValues[[i, j]] = Imagevalues[[i, j]]]
>    ]
>   ], Close[Imagevalues]}
>
>
> MaxProjection[#] & /@ FileNames@"*edf"
>
> So I guess Mathm saves all the data of all the images after each
> other. But once the comparision is done with the one MaxProjection
> Image I want to have, all the data can be dumped from the memory and
> then the next image can be compared with this MaxProjection image.
>
> Does anyone have an idea? Thx.
> 


[toc] | [prev] | [next] | [standalone]


#4510

FromDavid Bailey <dave@removedbailey.co.uk>
Date2011-08-21 09:36 +0000
Message-ID<j2qji4$d28$1@smc.vnet.net>
In reply to#4476
On 19/08/2011 11:39, Peter Rodenbach wrote:
> Hi everyone.
>
> I am analysing image data and I have the problem that after about 50
> images or so Mathematica quits the calculation with the hint out of
> memory. The interesting part of the code i wrote is :
>
> (*Import the first image into Maxvals, which will be compared to all \
> other images and overwritten if any point is found to be larger*)
>
> SetDirectory[
>    "E:\\...];
>
> Initialise = OpenRead["s2_260_0000.edf", BinaryFormat ->  True];
> MaxValues =
>    BinaryRead["s2_260_0000.edf",
>     Table["UnsignedInteger16", {512}, {512}]];
> counter = Dimensions[MaxValues];
>
> (*Load all experimental images and perform maximum projection*)
>
> MaxProjection[filename_] := {OpenRead[filename, BinaryFormat ->  True],
>     Imagevalues =
>     BinaryRead[filename, Table["UnsignedInteger16", {512}, {512}]],
>    For[i = 1, i<= counter[[1]], i++,
>     	For [j = 1, j<= counter[[2]], j++,
>      If[MaxValues[[i, j]]>= Imagevalues[[i, j]],
>       MaxValues[[i, j]] = MaxValues[[i, j]],
>       MaxValues[[i, j]] = Imagevalues[[i, j]]]
>      	]
>     ], Close[Imagevalues]}
>
>
> MaxProjection[#]&  /@ FileNames@"*edf"
>
> So I guess Mathm saves all the data of all the images after each
> other. But once the comparision is done with the one MaxProjection
> Image I want to have, all the data can be dumped from the memory and
> then the next image can be compared with this MaxProjection image.
>
> Does anyone have an idea? Thx.
>
Your code looks wrong because you open each file with OpenRead and never 
close (or use) the stream it produces (I wonder if binary streams open a 
large buffer).
Try something like this:

strm=OpenRead[filename, BinaryFormat ->  True];
Imagevalues=BinaryRead[strm, Table["UnsignedInteger16", {512}, {512}]];
..........
Close[strm]


Notice that it is the stream that should be closed - not the data!

Also, I would use Module to define functions, rather than just using a 
list of expressions { .......}.

With a Module, you can specify certain  variables, such as strm, to be 
local to the module, and also, you won't return a list of garbage as the 
result of the call - which could cause trouble if one of those objects 
is big!

David Bailey
http://www.dbaileyconsultancy.co.uk



[toc] | [prev] | [standalone]


Back to top | Article view | comp.soft-sys.math.mathematica


csiph-web