Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.soft-sys.math.mathematica > #5968
| From | A Retey <awnl@gmx-topmail.de> |
|---|---|
| Newsgroups | comp.soft-sys.math.mathematica |
| Subject | Re: Error on importing previously saved data file: Get::bigfile |
| Date | 2011-10-28 09:44 +0000 |
| Organization | Steven M. Christensen and Associates, Inc and MathTensor, Inc. |
| Message-ID | <j8dtim$kgg$1@smc.vnet.net> (permalink) |
| References | <j89ugu$t7k$1@smc.vnet.net> |
Hi,
> I have saved a data set (Save["filename",data]) - which is about 3.5
> MB on the harddisk (a table with numbers and images,
> ByteCount=610117226).
> When I try to read it again into Memory I get "Get::bigfile: "The file
> "filename" is too large to be read into memory."
>
> I can work around it by splitting the data in smaller junks, but there
> should be a better way since the data set should not be too large
>
> Any ideas?
is it important for you to save in a text format? If all you want to do
is to store the data and read it back into Mathematica at a later time I
would suggest to use a binary format:
In[45]:= data =
Table[{#, ArrayPlot[#]} &@RandomReal[{0, 1}, {100, 100}], {30}];
In[46]:= ByteCount[data]
Out[46]= 4838920
In[48]:= Timing[
Export[ToFileName[{$HomeDirectory, "Desktop"}, "tst.mx"], data]]
Out[48]= {0.047, "C:\\Users\\albert.retey\\Desktop\\tst.mx"}
In[51]:= Timing[
datain = Import[ToFileName[{$HomeDirectory, "Desktop"}, "tst.mx"]];]
Out[51]= {0.047, Null}
If you need it to be transfered to another machine, there is the
wdx-format, which has no problem with files of that size but is
relatively slow:
In[54]:= Timing[
Export[ToFileName[{$HomeDirectory, "Desktop"}, "tst.wdx"], data]]
Out[54]= {6.927, "C:\\Users\\albert.retey\\Desktop\\tst.wdx"}
In[55]:= Timing[
datain = Import[ToFileName[{$HomeDirectory, "Desktop"}, "tst.wdx"]];]
Out[55]= {3.089, Null}
hth,
albert
Back to comp.soft-sys.math.mathematica | Previous | Next — Previous in thread | Find similar | Unroll thread
Error on importing previously saved data file: Get::bigfile Matthias Seise <mseise@gmail.com> - 2011-10-26 21:36 +0000 Re: Error on importing previously saved data file: Get::bigfile A Retey <awnl@gmx-topmail.de> - 2011-10-28 09:44 +0000
csiph-web