Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.soft-sys.math.mathematica > #9172 > unrolled thread

Problems reading text files with embedded delimiters

Started byMark Coleman <markspcoleman@gmail.com>
First post2012-03-20 07:26 +0000
Last post2012-03-22 10:48 +0000
Articles 3 — 3 participants

Back to article view | Back to comp.soft-sys.math.mathematica


Contents

  Problems reading text files with embedded delimiters Mark Coleman <markspcoleman@gmail.com> - 2012-03-20 07:26 +0000
    Re: Problems reading text files with embedded delimiters Rodrigo Murta <murta@rodrigomurta.com> - 2012-03-21 10:50 +0000
    Re: Problems reading text files with embedded delimiters David Reiss <dbreiss@gmail.com> - 2012-03-22 10:48 +0000

#9172 — Problems reading text files with embedded delimiters

FromMark Coleman <markspcoleman@gmail.com>
Date2012-03-20 07:26 +0000
SubjectProblems reading text files with embedded delimiters
Message-ID<jk9bfb$bo1$1@smc.vnet.net>
Hi,

I'm struggling with importing a large flat file into Mathematica. The file,
which was pulled off one of my company's legacy mainframe systems,
contains about 100 fields and several million rows, including a mix of
dates, reals, and text. The fields are delimited by commas.

One of the text fields, however, is quite large and contains commas,
line feeds, carriage returns, etc -- lots of characters commonly used
as delimiters. It is also a variable length field. There does not seem
to be anything characters within the field that would allow the
specification of a simple delimiter. I've tried the default Import[]
command as well as various permutations of ReadList[] using Record,
Word, and String specifications.

I'm wondering if anyone could suggest an approach that would allow me
to read this file. More precisely, is there a way to avoid reading
characters embedded in the large text field that Mathematica thinks are field
delimiters?

Thanks,

Mark

[toc] | [next] | [standalone]


#9187

FromRodrigo Murta <murta@rodrigomurta.com>
Date2012-03-21 10:50 +0000
Message-ID<jkcbpa$qht$1@smc.vnet.net>
In reply to#9172
On Mar 20, 4:26 am, Mark Coleman <markspcole...@gmail.com> wrote:
> Hi,
>
> I'm struggling with importing a large flat file into Mathematica. The file,
> which was pulled off one of my company's legacy mainframe systems,
> contains about 100 fields and several million rows, including a mix of
> dates, reals, and text. The fields are delimited by commas.
>
> One of the text fields, however, is quite large and contains commas,
> line feeds, carriage returns, etc -- lots of characters commonly used
> as delimiters. It is also a variable length field. There does not seem
> to be anything characters within the field that would allow the
> specification of a simple delimiter. I've tried the default Import[]
> command as well as various permutations of ReadList[] using Record,
> Word, and String specifications.
>
> I'm wondering if anyone could suggest an approach that would allow me
> to read this file. More precisely, is there a way to avoid reading
> characters embedded in the large text field that Mathematica thinks are field
> delimiters?
>
> Thanks,
>
> Mark

Hi Mark
I didn't completely get your point. What is your delimiter? Your large
text is quoted? Can you sendo one example line? Or a toy code?
For large text string you could use some low level programming.
Maybe this tutorial could help you:
http://reference.wolfram.com/mathematica/guide/LowLevelFileOperations.html
Go to the stream topic.
best regards
Murta

[toc] | [prev] | [next] | [standalone]


#9189

FromDavid Reiss <dbreiss@gmail.com>
Date2012-03-22 10:48 +0000
Message-ID<jkf02m$8hi$1@smc.vnet.net>
In reply to#9172
One possibility for this is to read the file as Lines and, given that
you know that there is only one column with the unstructured text
field, to parse the other columns from the left and the right up to
that column's data since each of those columns are delimited uniquely
by commas.

Best,
David

On Mar 20, 3:26 am, Mark Coleman <markspcole...@gmail.com> wrote:
> Hi,
>
> I'm struggling with importing a large flat file into Mathematica. The file,
> which was pulled off one of my company's legacy mainframe systems,
> contains about 100 fields and several million rows, including a mix of
> dates, reals, and text. The fields are delimited by commas.
>
> One of the text fields, however, is quite large and contains commas,
> line feeds, carriage returns, etc -- lots of characters commonly used
> as delimiters. It is also a variable length field. There does not seem
> to be anything characters within the field that would allow the
> specification of a simple delimiter. I've tried the default Import[]
> command as well as various permutations of ReadList[] using Record,
> Word, and String specifications.
>
> I'm wondering if anyone could suggest an approach that would allow me
> to read this file. More precisely, is there a way to avoid reading
> characters embedded in the large text field that Mathematica thinks are field
> delimiters?
>
> Thanks,
>
> Mark

[toc] | [prev] | [standalone]


Back to top | Article view | comp.soft-sys.math.mathematica


csiph-web