Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.soft-sys.math.mathematica > #5968

Re: Error on importing previously saved data file: Get::bigfile

From A Retey <awnl@gmx-topmail.de>
Newsgroups comp.soft-sys.math.mathematica
Subject Re: Error on importing previously saved data file: Get::bigfile
Date 2011-10-28 09:44 +0000
Organization Steven M. Christensen and Associates, Inc and MathTensor, Inc.
Message-ID <j8dtim$kgg$1@smc.vnet.net> (permalink)
References <j89ugu$t7k$1@smc.vnet.net>

Show all headers | View raw


Hi,

> I have saved a data set (Save["filename",data]) - which is about 3.5
> MB on the harddisk (a table with numbers and images,
> ByteCount=610117226).
> When I try to read it again into Memory I get "Get::bigfile: "The file
> "filename" is too large to be read into memory."
>
> I can work around it by splitting the data in smaller junks, but there
> should be a better way since the data set should not be too large
>
> Any ideas?

is it important for you to save in a text format? If all you want to do 
is to store the data and read it back into Mathematica at a later time I 
would suggest to use a binary format:

In[45]:= data =
   Table[{#, ArrayPlot[#]} &@RandomReal[{0, 1}, {100, 100}], {30}];

In[46]:= ByteCount[data]

Out[46]= 4838920

In[48]:= Timing[
  Export[ToFileName[{$HomeDirectory, "Desktop"}, "tst.mx"], data]]

Out[48]= {0.047, "C:\\Users\\albert.retey\\Desktop\\tst.mx"}

In[51]:= Timing[
  datain = Import[ToFileName[{$HomeDirectory, "Desktop"}, "tst.mx"]];]

Out[51]= {0.047, Null}

If you need it to be transfered to another machine, there is the 
wdx-format, which has no problem with files of that size but is 
relatively slow:

In[54]:= Timing[
  Export[ToFileName[{$HomeDirectory, "Desktop"}, "tst.wdx"], data]]

Out[54]= {6.927, "C:\\Users\\albert.retey\\Desktop\\tst.wdx"}

In[55]:= Timing[
  datain = Import[ToFileName[{$HomeDirectory, "Desktop"}, "tst.wdx"]];]

Out[55]= {3.089, Null}

hth,

albert

Back to comp.soft-sys.math.mathematica | Previous | Next — Previous in thread | Find similar | Unroll thread


Thread

Error on importing previously saved data file: Get::bigfile Matthias Seise <mseise@gmail.com> - 2011-10-26 21:36 +0000
  Re: Error on importing previously saved data file: Get::bigfile A Retey <awnl@gmx-topmail.de> - 2011-10-28 09:44 +0000

csiph-web