Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.soft-sys.math.mathematica > #5890 > unrolled thread

Error on importing previously saved data file: Get::bigfile

Started byMatthias Seise <mseise@gmail.com>
First post2011-10-26 21:36 +0000
Last post2011-10-28 09:44 +0000
Articles 2 — 2 participants

Back to article view | Back to comp.soft-sys.math.mathematica


Contents

  Error on importing previously saved data file: Get::bigfile Matthias Seise <mseise@gmail.com> - 2011-10-26 21:36 +0000
    Re: Error on importing previously saved data file: Get::bigfile A Retey <awnl@gmx-topmail.de> - 2011-10-28 09:44 +0000

#5890 — Error on importing previously saved data file: Get::bigfile

FromMatthias Seise <mseise@gmail.com>
Date2011-10-26 21:36 +0000
SubjectError on importing previously saved data file: Get::bigfile
Message-ID<j89ugu$t7k$1@smc.vnet.net>
I'm working in Mathematica 8, 64 bit.

I have saved a data set (Save["filename",data]) - which is about 3.5
MB on the harddisk (a table with numbers and images,
ByteCount=610117226).
When I try to read it again into Memory I get "Get::bigfile: "The file
"filename" is too large to be read into memory."

I can work around it by splitting the data in smaller junks, but there
should be a better way since the data set should not be too large

Any ideas?


Best regards,
Matthias Seise

[toc] | [next] | [standalone]


#5968

FromA Retey <awnl@gmx-topmail.de>
Date2011-10-28 09:44 +0000
Message-ID<j8dtim$kgg$1@smc.vnet.net>
In reply to#5890
Hi,

> I have saved a data set (Save["filename",data]) - which is about 3.5
> MB on the harddisk (a table with numbers and images,
> ByteCount=610117226).
> When I try to read it again into Memory I get "Get::bigfile: "The file
> "filename" is too large to be read into memory."
>
> I can work around it by splitting the data in smaller junks, but there
> should be a better way since the data set should not be too large
>
> Any ideas?

is it important for you to save in a text format? If all you want to do 
is to store the data and read it back into Mathematica at a later time I 
would suggest to use a binary format:

In[45]:= data =
   Table[{#, ArrayPlot[#]} &@RandomReal[{0, 1}, {100, 100}], {30}];

In[46]:= ByteCount[data]

Out[46]= 4838920

In[48]:= Timing[
  Export[ToFileName[{$HomeDirectory, "Desktop"}, "tst.mx"], data]]

Out[48]= {0.047, "C:\\Users\\albert.retey\\Desktop\\tst.mx"}

In[51]:= Timing[
  datain = Import[ToFileName[{$HomeDirectory, "Desktop"}, "tst.mx"]];]

Out[51]= {0.047, Null}

If you need it to be transfered to another machine, there is the 
wdx-format, which has no problem with files of that size but is 
relatively slow:

In[54]:= Timing[
  Export[ToFileName[{$HomeDirectory, "Desktop"}, "tst.wdx"], data]]

Out[54]= {6.927, "C:\\Users\\albert.retey\\Desktop\\tst.wdx"}

In[55]:= Timing[
  datain = Import[ToFileName[{$HomeDirectory, "Desktop"}, "tst.wdx"]];]

Out[55]= {3.089, Null}

hth,

albert

[toc] | [prev] | [standalone]


Back to top | Article view | comp.soft-sys.math.mathematica


csiph-web