Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.soft-sys.math.mathematica > #10520

Re: Structure of "identical" data not equal in size

From Szabolcs Horvát <szhorvat@gmail.com>
Newsgroups comp.soft-sys.math.mathematica
Subject Re: Structure of "identical" data not equal in size
Date 2012-06-18 09:47 +0000
Organization Steven M. Christensen and Associates, Inc and MathTensor, Inc.
Message-ID <jrmtfl$h5u$1@smc.vnet.net> (permalink)
References <jrk2s2$8n0$1@smc.vnet.net>

Show all headers | View raw


On 2012.06.17. 10:01, Paul E McHale wrote:
> (* First, the internal data and then writing it to CSV   *)
> m=Table[{i,Sin[i],Cos[i]},{i,1,1000,1.0}];
> SetDirectory[NotebookDirectory[]]
> Export["Test2.csv",m, "CSV"]
>
> (*Reading it back in *)
> mIn = Import["Test2.csv", "CSV"]
>
> (* compare data *)
> mIn==m
>
>>> True
>
>
> (* see memory usage *)
> ByteCount[m]
>>> 24168
>
> ByteCount[mIn]
>>> 144040
>
> (* what is the minimum size using reals *)
> 1000 * 3 * 8
>>> 24000
>
> Actual size: 24168
> Read in file data: 144040
>
> (* Size *)
> ByteCount[mIn]/ByteCount[m] * 1.0
>>> 5.96
>
> --------------
>
> Why is the "same" data taking up 6x the memory after being written to disk and read back in.  This is a serious problem as we have large data at work being shared by files and Mathematica is currently the only language that can't read it (C# being the other language).
>
> How can they pass for equal yet internally generated takes much less memory?
>
> Any input welcome as I can only reduce the data with an external editor so I can try to work with it.
>

The reason is that mIn is not a packed array:

Developer`PackedArrayQ[m]

(* ==> True *)

Developer`PackedArrayQ[mIn]

(* ==> False *)

I guess the reason Import doesn't return a packed array is that CSV 
files can hold inhomogeneous data (e.g. both numbers and strings) while 
packed arrays are always homogeneous.  You can try exporting to other 
formats, perhaps as binary data (which should still be easy to read 
using C#)

Alternatively convert back to a packed array right after you read it in:

Developer`ToPackedArray[mIn]

Packed arrays give you the same storage efficiency as C#.

-- 
Szabolcs Horvát
Visit Mathematica.SE:  http://mathematica.stackexchange.com/

Back to comp.soft-sys.math.mathematica | Previous | Next — Previous in thread | Find similar | Unroll thread


Thread

Structure of "identical" data not equal in size Paul E McHale <paulmchale@me.com> - 2012-06-17 08:01 +0000
  Re: Structure of "identical" data not equal in size Paul E McHale <paulmchale@me.com> - 2012-06-18 09:43 +0000
  Re: Structure of "identical" data not equal in size awnl <awnl@gmx-topmail.de> - 2012-06-18 09:46 +0000
  Re: Structure of "identical" data not equal in size Szabolcs Horvát <szhorvat@gmail.com> - 2012-06-18 09:47 +0000
  Re: Structure of "identical" data not equal in size Szabolcs Horvát <szhorvat@gmail.com> - 2012-06-18 09:47 +0000

csiph-web