Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.soft-sys.math.mathematica > #10508 > unrolled thread

Structure of "identical" data not equal in size

Started byPaul E McHale <paulmchale@me.com>
First post2012-06-17 08:01 +0000
Last post2012-06-18 09:47 +0000
Articles 5 — 3 participants

Back to article view | Back to comp.soft-sys.math.mathematica


Contents

  Structure of "identical" data not equal in size Paul E McHale <paulmchale@me.com> - 2012-06-17 08:01 +0000
    Re: Structure of "identical" data not equal in size Paul E McHale <paulmchale@me.com> - 2012-06-18 09:43 +0000
    Re: Structure of "identical" data not equal in size awnl <awnl@gmx-topmail.de> - 2012-06-18 09:46 +0000
    Re: Structure of "identical" data not equal in size Szabolcs Horvát <szhorvat@gmail.com> - 2012-06-18 09:47 +0000
    Re: Structure of "identical" data not equal in size Szabolcs Horvát <szhorvat@gmail.com> - 2012-06-18 09:47 +0000

#10508 — Structure of "identical" data not equal in size

FromPaul E McHale <paulmchale@me.com>
Date2012-06-17 08:01 +0000
SubjectStructure of "identical" data not equal in size
Message-ID<jrk2s2$8n0$1@smc.vnet.net>
(* First, the internal data and then writing it to CSV   *)
m=Table[{i,Sin[i],Cos[i]},{i,1,1000,1.0}];
SetDirectory[NotebookDirectory[]]
Export["Test2.csv",m, "CSV"]

(*Reading it back in *)
mIn = Import["Test2.csv", "CSV"]

(* compare data *)
mIn==m

>> True


(* see memory usage *)
ByteCount[m]
>> 24168

ByteCount[mIn]
>> 144040

(* what is the minimum size using reals *)
1000 * 3 * 8
>> 24000

Actual size: 24168
Read in file data: 144040

(* Size *)
ByteCount[mIn]/ByteCount[m] * 1.0
>> 5.96

--------------

Why is the "same" data taking up 6x the memory after being written to disk and read back in.  This is a serious problem as we have large data at work being shared by files and Mathematica is currently the only language that can't read it (C# being the other language).

How can they pass for equal yet internally generated takes much less memory?

Any input welcome as I can only reduce the data with an external editor so I can try to work with it.


Thanks,
Paul





[toc] | [next] | [standalone]


#10514

FromPaul E McHale <paulmchale@me.com>
Date2012-06-18 09:43 +0000
Message-ID<jrmt80$h25$1@smc.vnet.net>
In reply to#10508
On Jun 17, 2012, at 8:21 AM, Szabolcs Horv=E1t wrote:

> On 2012.06.17. 10:01, Paul E McHale wrote:
>> (* First, the internal data and then writing it to CSV   *)
>> m=Table[{i,Sin[i],Cos[i]},{i,1,1000,1.0}];
>> SetDirectory[NotebookDirectory[]]
>> Export["Test2.csv",m, "CSV"]
>>
>> (*Reading it back in *)
>> mIn = Import["Test2.csv", "CSV"]
>>
>> (* compare data *)
>> mIn==m
>>
>>>> True
>>
>>
>> (* see memory usage *)
>> ByteCount[m]
>>>> 24168
>>
>> ByteCount[mIn]
>>>> 144040
>>
>> (* what is the minimum size using reals *)
>> 1000 * 3 * 8
>>>> 24000
>>
>> Actual size: 24168
>> Read in file data: 144040
>>
>> (* Size *)
>> ByteCount[mIn]/ByteCount[m] * 1.0
>>>> 5.96
>>
>> --------------
>>
>> Why is the "same" data taking up 6x the memory after being written to disk and read back in.  This is a serious problem as we have large data at work being shared by files and Mathematica is currently the only language that can't read it (C# being the other language).
>>
>> How can they pass for equal yet internally generated takes much less memory?
>>
>> Any input welcome as I can only reduce the data with an external editor so I can try to work with it.
>>
>
> The reason is that mIn is not a packed array:
>
> Developer`PackedArrayQ[m]
>
> (* ==> True *)
>
> Developer`PackedArrayQ[mIn]
>
> (* ==> False *)
>
> I guess the reason Import doesn't return a packed array is that CSV files can hold inhomogeneous data (e.g. both numbers and strings) while packed arrays are always homogeneous.  You can try exporting to other formats, perhaps as binary data (which should still be easy to read using C#)
>
> Alternatively convert back to a packed array right after you read it in:
>
> Developer`ToPackedArray[mIn]
>
> Packed arrays give you the same storage efficiency as C#.


---------------
Thank you for the reply.  This works acceptably if the data can fit in memory as an unpacked array.  It appears there is no reason of the array to be unpacked other than Mathematica does not pack while importing even if the file homogeneous.  I have tried binary with with homogenous specifications, {Real64, Real64, Real64}.  Is there any format that will read homogenous data in a packed array format?

The unpacked arrays are simply to huge taking over 5x the memory.

Thank you,
Paul



>
> --
> Szabolcs Horv=E1t
> Visit Mathematica.SE:  http://mathematica.stackexchange.com/

[toc] | [prev] | [next] | [standalone]


#10517

Fromawnl <awnl@gmx-topmail.de>
Date2012-06-18 09:46 +0000
Message-ID<jrmtcp$h4c$1@smc.vnet.net>
In reply to#10508
Hi,

> (* First, the internal data and then writing it to CSV   *)
> m=Table[{i,Sin[i],Cos[i]},{i,1,1000,1.0}];
> SetDirectory[NotebookDirectory[]] Export["Test2.csv",m, "CSV"]
>
> (*Reading it back in *) mIn = Import["Test2.csv", "CSV"]
>
> (* compare data *) mIn==m
>
>>> True
>
>
> (* see memory usage *) ByteCount[m]
>>> 24168
>
> ByteCount[mIn]
>>> 144040
>
> (* what is the minimum size using reals *) 1000 * 3 * 8
>>> 24000
>
> Actual size: 24168 Read in file data: 144040
>
> (* Size *) ByteCount[mIn]/ByteCount[m] * 1.0
>>> 5.96
>
> --------------
>
> Why is the "same" data taking up 6x the memory after being written to
> disk and read back in.  This is a serious problem as we have large
> data at work being shared by files and Mathematica is currently the
> only language that can't read it (C# being the other language).

The keyworkd is "packed arrays":

Developer`PackedArrayQ[m]
Developer`PackedArrayQ[mIn]

> How can they pass for equal yet internally generated takes much less
> memory?

you can find some information about packed arrays in the documentation 
and a lot more in this newsgroup's archive and also mathematica stack 
exchange.

> Any input welcome as I can only reduce the data with an external
> editor so I can try to work with it.

What you can try is to convert it to a packed array after importing:

mIn = Developer`ToPackedArray[mIn];

unfortunately that doesn't help if the Import fails because there isn't 
enough memory to hold the intermediate expression. You might have to 
read the data line by line (or a chunk of lines at a time) and convert 
it to a packed array as you go along. Can be done but is some effort as 
you have to ensure that the data is never unpacked.

hth,

albert

[toc] | [prev] | [next] | [standalone]


#10519

FromSzabolcs Horvát <szhorvat@gmail.com>
Date2012-06-18 09:47 +0000
Message-ID<jrmtem$h5d$1@smc.vnet.net>
In reply to#10508
Hi Paul,

If you don't need to import this data in a different program than
Mathematica then I recommend using either the MX format, or the trick with
Compress described e.g. here:

http://mathematica.stackexchange.com/a/1960/12

The problem with MX is that it is not portable between different
systems/architectures, but it's extremely fast to save/load.

The Compress-trick offers reasonable performance while being portable
between systems.

Both preserve packed arrays (without unpacking), but MX requires less
memory to Import/Export.

If you need to exchange data with other programs, BinaryRead/BinaryWrite
may be the way to go.

On 18 June 2012 00:15, Paul E McHale <paulmchale@me.com> wrote:

>
> On Jun 17, 2012, at 8:21 AM, Szabolcs Horv=C3=A1t wrote:
>
> > On 2012.06.17. 10:01, Paul E McHale wrote:
> >> (* First, the internal data and then writing it to CSV   *)
> >> m=Table[{i,Sin[i],Cos[i]},{i,1,1000,1.0}];
> >> SetDirectory[NotebookDirectory[]]
> >> Export["Test2.csv",m, "CSV"]
> >>
> >> (*Reading it back in *)
> >> mIn = Import["Test2.csv", "CSV"]
> >>
> >> (* compare data *)
> >> mIn==m
> >>
> >>>> True
> >>
> >>
> >> (* see memory usage *)
> >> ByteCount[m]
> >>>> 24168
> >>
> >> ByteCount[mIn]
> >>>> 144040
> >>
> >> (* what is the minimum size using reals *)
> >> 1000 * 3 * 8
> >>>> 24000
> >>
> >> Actual size: 24168
> >> Read in file data: 144040
> >>
> >> (* Size *)
> >> ByteCount[mIn]/ByteCount[m] * 1.0
> >>>> 5.96
> >>
> >> --------------
> >>
> >> Why is the "same" data taking up 6x the memory after being written to
> disk and read back in.  This is a serious problem as we have large data a=
t
> work being shared by files and Mathematica is currently the only language
> that can't read it (C# being the other language).
> >>
> >> How can they pass for equal yet internally generated takes much less
> memory?
> >>
> >> Any input welcome as I can only reduce the data with an external edito=
r
> so I can try to work with it.
> >>
> >
> > The reason is that mIn is not a packed array:
> >
> > Developer`PackedArrayQ[m]
> >
> > (* ==> True *)
> >
> > Developer`PackedArrayQ[mIn]
> >
> > (* ==> False *)
> >
> > I guess the reason Import doesn't return a packed array is that CSV
> files can hold inhomogeneous data (e.g. both numbers and strings) while
> packed arrays are always homogeneous.  You can try exporting to other
> formats, perhaps as binary data (which should still be easy to read using
> C#)
> >
> > Alternatively convert back to a packed array right after you read it in=
:
> >
> > Developer`ToPackedArray[mIn]
> >
> > Packed arrays give you the same storage efficiency as C#.
>
>
> ---------------
> Thank you for the reply.  This works acceptably if the data can fit in
> memory as an unpacked array.  It appears there is no reason of the array =
to
> be unpacked other than Mathematica does not pack while importing even if
> the file homogeneous.  I have tried binary with with homogenous
> specifications, {Real64, Real64, Real64}.  Is there any format that will
> read homogenous data in a packed array format?
>
> The unpacked arrays are simply to huge taking over 5x the memory.
>
> Thank you,
> Paul
>
>
>
> >
> > --
> > Szabolcs Horv=C3=A1t
> > Visit Mathematica.SE:  http://mathematica.stackexchange.com/
>
>

[toc] | [prev] | [next] | [standalone]


#10520

FromSzabolcs Horvát <szhorvat@gmail.com>
Date2012-06-18 09:47 +0000
Message-ID<jrmtfl$h5u$1@smc.vnet.net>
In reply to#10508
On 2012.06.17. 10:01, Paul E McHale wrote:
> (* First, the internal data and then writing it to CSV   *)
> m=Table[{i,Sin[i],Cos[i]},{i,1,1000,1.0}];
> SetDirectory[NotebookDirectory[]]
> Export["Test2.csv",m, "CSV"]
>
> (*Reading it back in *)
> mIn = Import["Test2.csv", "CSV"]
>
> (* compare data *)
> mIn==m
>
>>> True
>
>
> (* see memory usage *)
> ByteCount[m]
>>> 24168
>
> ByteCount[mIn]
>>> 144040
>
> (* what is the minimum size using reals *)
> 1000 * 3 * 8
>>> 24000
>
> Actual size: 24168
> Read in file data: 144040
>
> (* Size *)
> ByteCount[mIn]/ByteCount[m] * 1.0
>>> 5.96
>
> --------------
>
> Why is the "same" data taking up 6x the memory after being written to disk and read back in.  This is a serious problem as we have large data at work being shared by files and Mathematica is currently the only language that can't read it (C# being the other language).
>
> How can they pass for equal yet internally generated takes much less memory?
>
> Any input welcome as I can only reduce the data with an external editor so I can try to work with it.
>

The reason is that mIn is not a packed array:

Developer`PackedArrayQ[m]

(* ==> True *)

Developer`PackedArrayQ[mIn]

(* ==> False *)

I guess the reason Import doesn't return a packed array is that CSV 
files can hold inhomogeneous data (e.g. both numbers and strings) while 
packed arrays are always homogeneous.  You can try exporting to other 
formats, perhaps as binary data (which should still be easy to read 
using C#)

Alternatively convert back to a packed array right after you read it in:

Developer`ToPackedArray[mIn]

Packed arrays give you the same storage efficiency as C#.

-- 
Szabolcs Horvát
Visit Mathematica.SE:  http://mathematica.stackexchange.com/

[toc] | [prev] | [standalone]


Back to top | Article view | comp.soft-sys.math.mathematica


csiph-web