Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.soft-sys.math.mathematica > #2099 > unrolled thread

Import speed from different file types

Back to article view | Back to comp.soft-sys.math.mathematica

Contents

  Import speed from different file types CPE Bach <hisicon@gmail.com> - 2011-05-04 23:47 +0000
    Re: Import speed from different file types Albert Retey <awnl@gmx-topmail.de> - 2011-05-08 11:37 +0000
    Re: Import speed from different file types Armand Tamzarian <mike.honeychurch@gmail.com> - 2011-05-09 10:20 +0000

#2099 — Import speed from different file types

Hi users,
So I have hundreds of data files of time series of stock returns. To make th=
e import process as fast as possible in Mathematica 8, should I prepare those data files in what format?

xls? csv? dat? txt?

Thanks for your input in advance.

MP

Sent from my iPhone=

[toc] | [next] | [standalone]

#2208

Hi,

> So I have hundreds of data files of time series of stock returns. To make the import process as fast as possible in Mathematica 8, should I prepare those data files in what format?
>
> xls? csv? dat? txt?
>
> Thanks for your input in advance.

it really depends on the content of the files and on the number of 
records. I would also not be too sure that xls is much slower than the 
others, again depending on the file content. You really should go for a 
try, there are plenty details that might make a suggestion a complete 
desaster. For example we found that xls files containing diagrams was 
terribly slow in version 8, but not in version 7. If the diagrams were 
deleted, all was back to normal. Another thing that you should be aware 
is that with csv, dat and txt you are basically parsing a text file and 
Mathematica tries to be very smart to e.g. get reasonable dates from all 
kind of formatted strings which might be relatively slow (especially if 
that is done by calls to java as at least some versions do/did, I 
believe). I was able to get lot of speed improvement when avoiding the 
standard import for csv files and and rather import them as strings and 
then do some very simple StringReplace + ToExpression to convert dates 
with a fixed well known format. I would be very sceptical that it is a 
save bet that those text based file formats will always be faster than xls.

If you will have to process your files only once I would try to save 
your work time: choose something that works out of the box and be happy 
to keep a machine or two busy for a day or two. I found xls to be a good 
choice in many cases, I had never to solve problems about line breaks, 
differing record separators, irregular formatted date strings, character 
encodings and all such, which might or might not occur with csv and friends.

Only if you will process those hundreds of files over and over again, I 
would consider additional work on your side to be justified. Then, if 
possible, I would try to save the time series as binary files of e.g. 
just reals and read them with BinaryRead, my expectation is that this 
could be a lot faster than any file format that can be Import-ed, 
including text-based formats like csv.

hth,

albert

[toc] | [prev] | [next] | [standalone]

#2229

On May 5, 9:47 am, CPE Bach <hisi...@gmail.com> wrote:
> Hi users,
> So I have hundreds of data files of time series of stock returns. To make the import process as fast as possible in Mathematica 8, should I prepare those data files in what format?
>
> xls? csv? dat? txt?
>
> Thanks for your input in advance.
>
> MP
>


If you want to regularly analyze the data then your best bet is
definitely to get all of this data into a database and use the
DatabaseLink. Seriously you'll wonder why you ever bothered importing
Excel etc. files. Also in the future add your data directly into the
database.

Mike

[toc] | [prev] | [standalone]

Back to top | Article view | comp.soft-sys.math.mathematica

csiph-web