Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.soft-sys.math.mathematica > #1571 > unrolled thread

Importing text file as machine-precision array

Back to article view | Back to comp.soft-sys.math.mathematica

Contents

  Importing text file as machine-precision array Sidney Cadot <sidney@jigsaw.nl> - 2011-04-09 11:29 +0000
    Re: Importing text file as machine-precision array David Bailey <dave@removedbailey.co.uk> - 2011-04-09 22:00 +0000
      Re: Importing text file as machine-precision array Sidney Cadot <sidney.cadot@gmail.com> - 2011-04-11 11:05 +0000
        Re: Importing text file as machine-precision array David Bailey <dave@removedbailey.co.uk> - 2011-04-17 11:53 +0000

#1571 — Importing text file as machine-precision array

Hi all,

I am trying to import a big text file containing non-sparse 2D
numerical data into Mathematica. My aim is to get an array where the
numbers are stored as machine-precision numbers. In other words,
ByteCount[data] should be approximately equal to 8 times the number of
table elements (assuming machine precision corresponds to 8-byte
IEEE-754 numbers).

If I do a straight data=Import["file","Table"], the file is parsed
with all numbers represented as arbitrary precision floating point
numbers; the data structure is ten times as big as it should be. If I
do data=N[data] afterwards, the data is still four times as big as I
would have expected.

What is the appropriate way to accomplish what I want? Unfortunately,
I cannot seem to find an option to Import[] to do this immediately
when reading the data.

[toc] | [next] | [standalone]

#1575

On 09/04/2011 12:29, Sidney Cadot wrote:
> Hi all,
>
> I am trying to import a big text file containing non-sparse 2D
> numerical data into Mathematica. My aim is to get an array where the
> numbers are stored as machine-precision numbers. In other words,
> ByteCount[data] should be approximately equal to 8 times the number of
> table elements (assuming machine precision corresponds to 8-byte
> IEEE-754 numbers).
>
> If I do a straight data=Import["file","Table"], the file is parsed
> with all numbers represented as arbitrary precision floating point
> numbers; the data structure is ten times as big as it should be. If I
> do data=N[data] afterwards, the data is still four times as big as I
> would have expected.
>
> What is the appropriate way to accomplish what I want? Unfortunately,
> I cannot seem to find an option to Import[] to do this immediately
> when reading the data.
>
The only way that you will obtain an array as compact as you calculate 
(plus a small overhead) is to use the function Developer`ToPackedArray 
on the result after applying N.

You need to realise that arrays of reals (and some other basic types) 
come in two formats. That packed array format assumes that all elements 
are Real, or whatever. The other format can accommodate any type of 
object in any position, so it is less compact. Mathematica algorithms 
produce the same results on either format, but sometimes (as with 
Import) an array is not packed when it could be.

Packed arrays are also more efficient in terms of speed of processing.

David Bailey
http://www.dbaileyconsultancy.co.uk

[toc] | [prev] | [next] | [standalone]

#1603

> The only way that you will obtain an array as compact as you calculate
> (plus a small overhead) is to use the function Developer`ToPackedArray
> on the result after applying N.

Thanks for the hint -- I didn't know about thgat package. As I
discovered by trial and error, this also works:

data = data + ConstantArray[0., Dimensions[data]];

Regards Sidney

[toc] | [prev] | [next] | [standalone]

#1762

On 11/04/2011 12:05, Sidney Cadot wrote:
>> The only way that you will obtain an array as compact as you calculate
>> (plus a small overhead) is to use the function Developer`ToPackedArray
>> on the result after applying N.
>
> Thanks for the hint -- I didn't know about thgat package. As I
> discovered by trial and error, this also works:
>
> data = data + ConstantArray[0., Dimensions[data]];
>
> Regards Sidney
>
Mathematica uses some heuristics to decide when to convert to packed 
arrays, and one of these obviously kicks in in this case, but I would 
guess that it would be marginally faster to use the ToPackedArray 
function. Although the Developer context could in theory change at 
future versions, the packed array functions have been there for many 
versions, and are widely used, so I can't imagine these would get moved 
without a lot of warning.

David Bailey
http://www.dbaileyconsultancy.co.uk

[toc] | [prev] | [standalone]

Back to top | Article view | comp.soft-sys.math.mathematica

csiph-web