Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.soft-sys.math.mathematica > #14859 > unrolled thread

Importing a file and extracting data

Started byhowardfink@gmail.com
First post2013-06-14 08:54 +0000
Last post2013-06-15 08:13 +0000
Articles 3 — 3 participants

Back to article view | Back to comp.soft-sys.math.mathematica


Contents

  Importing a file and extracting data howardfink@gmail.com - 2013-06-14 08:54 +0000
    Re: Importing a file and extracting data David Bailey <dave@removedbailey.co.uk> - 2013-06-15 08:13 +0000
    Re: Importing a file and extracting data Helen Read <readhpr@gmail.com> - 2013-06-15 08:13 +0000

#14859 — Importing a file and extracting data

Fromhowardfink@gmail.com
Date2013-06-14 08:54 +0000
SubjectImporting a file and extracting data
Message-ID<kpeln9$htb$1@smc.vnet.net>
I have a series of files of this form:

June 7, 2013      
Tc+Naphthalene  vs Temperature (oC)

Run 1
Tc_Naph_84_2C			3.740 ns
Tc_Naph_87_1C			3.731 ns
Tc_Naph_89_9C			3.720 ns
Tc_Naph_92_9C			3.704 ns
Tc_Naph_94_7C			3.687 ns
Tc_Naph_97_6C			3.694 ns


Run 2
Tc_Naph_83_2C			3.758 ns
Tc_Naph_83_4C			3.750 ns
Tc_Naph_86_4C			3.728 ns
Tc_Naph_88_1C			3.725 ns
Tc_Naph_90_2C			3.716 ns
Tc_Naph_93_1C			3.704 ns
Tc_Naph_94_7C			3.673 ns
Tc_Naph_97_7C			3.684 ns
Tc_Naph_97_9C			3.665 ns




I used an Import command to read in the file, but now I am just sitting and=
 staring, without a clue how to get the 84_2 converted to the number 84.2,e=
tc. and ending up with two lists: Run 1 and Run 2, consisting of pairs of t=
emperature and time.  The temperature will eventually be converted to 1/abs=
olute temperature.

I've read lots and lots of help, thumbed through dozens of pages of a Mathe=
matica 5 manual, and don't know where to start.  I'm trying to help a 90-ye=
ar-old chemistry professor, who is currently using a calculator, but  there=
 will be dozens of runs of this experiment.

[toc] | [next] | [standalone]


#14869

FromDavid Bailey <dave@removedbailey.co.uk>
Date2013-06-15 08:13 +0000
Message-ID<kph7na$2di$1@smc.vnet.net>
In reply to#14859
On 14/06/2013 09:54, howardfink@gmail.com wrote:
> I have a series of files of this form:
>
> June 7, 2013
> Tc+Naphthalene  vs Temperature (oC)
>
> Run 1
> Tc_Naph_84_2C			3.740 ns
> Tc_Naph_87_1C			3.731 ns
> Tc_Naph_89_9C			3.720 ns
> Tc_Naph_92_9C			3.704 ns
> Tc_Naph_94_7C			3.687 ns
> Tc_Naph_97_6C			3.694 ns
>
>
> Run 2
> Tc_Naph_83_2C			3.758 ns
> Tc_Naph_83_4C			3.750 ns
> Tc_Naph_86_4C			3.728 ns
> Tc_Naph_88_1C			3.725 ns
> Tc_Naph_90_2C			3.716 ns
> Tc_Naph_93_1C			3.704 ns
> Tc_Naph_94_7C			3.673 ns
> Tc_Naph_97_7C			3.684 ns
> Tc_Naph_97_9C			3.665 ns
>
>
>
>
> I used an Import command to read in the file, but now I am just sitting and=
>   staring, without a clue how to get the 84_2 converted to the number 84.2,e=
> tc. and ending up with two lists: Run 1 and Run 2, consisting of pairs of t=
> emperature and time.  The temperature will eventually be converted to 1/abs=
> olute temperature.
>
> I've read lots and lots of help, thumbed through dozens of pages of a Mathe=
> matica 5 manual, and don't know where to start.  I'm trying to help a 90-ye=
> ar-old chemistry professor, who is currently using a calculator, but  there=
>   will be dozens of runs of this experiment.
>
Import is designed to read text or binary data formatted in a standard 
form - e.g. CSV. Clearly your files have an ad-hoc format, so you can't 
expect Mathematica (or anything else) to read them without some effort!

The job is complicated by the fact that you (or the prof) used "_" 
rather than a decimal point, and that the number is joined on to other 
textual data. I am going to assume that the units (C and ns) are the 
same throughout, and can be discarded, and that all the temperatures 
have a decimal part, even if it is 0.

First define a couple of functions:

dataConvert[{{run_}, samples_}] := {run, Map[process, samples]};

process[line_] := Module[{tmp},
   tmp = StringReplace[
     "\"" <> line, {"_" ~~ a : (DigitCharacter ..) ~~ "_" ~~
        b : (DigitCharacter ..) ~~ "C" :>
       "\"," <> a <> "." <> b <> ",", " ns" :> ""}];
   ToExpression["{" <> tmp <> "}"]
   ];

It is best to avoid Import if the data is not in a recognised format, 
and to read it in as strings, discarding the empty lines, and extracting 
the data in the first two lines:

In[4]:= data = ReadList["c:\\maths\\data.dat", String];

In[7]:= data = DeleteCases[data, {}];

In[5]:= fileDate = data[[1]]

Out[5]= "June 7, 2013      "

In[9]:= fileTitle = data[[2]]

Out[9]= "Tc+Naphthalene  vs Temperature (oC)"

Break up the rest by detecting the 'Run' lines:
In[29]:= tmp =
  Partition[SplitBy[data[[3 ;;]], StringMatchQ[#, "Run" ~~ ___] &], 2]

Out[29]= {{{"Run 1"}, {"Tc_Naph_84_2C			3.740 ns",
    "Tc_Naph_87_1C			3.731 ns", "Tc_Naph_89_9C			3.720 ns",
    "Tc_Naph_92_9C			3.704 ns", "Tc_Naph_94_7C			3.687 ns",
    "Tc_Naph_97_6C			3.694 ns"}}, {{"Run 2"}, {"Tc_Naph_83_2C			3.758 \
ns", "Tc_Naph_83_4C			3.750 ns", "Tc_Naph_86_4C			3.728 ns",
    "Tc_Naph_88_1C			3.725 ns", "Tc_Naph_90_2C			3.716 ns",
    "Tc_Naph_93_1C			3.704 ns", "Tc_Naph_94_7C			3.673 ns",
    "Tc_Naph_97_7C			3.684 ns", "Tc_Naph_97_9C			3.665 ns"}}}

Now apply the previous functions to produce a nested list structure of 
strings and real numbers:

In[38]:= Map[dataConvert, tmp]

Out[38]= {{"Run 1", {{"Tc_Naph", 84.2, 3.74}, {"Tc_Naph", 87.1,
     3.731}, {"Tc_Naph", 89.9, 3.72}, {"Tc_Naph", 92.9,
     3.704}, {"Tc_Naph", 94.7, 3.687}, {"Tc_Naph", 97.6,
     3.694}}}, {"Run 2", {{"Tc_Naph", 83.2, 3.758}, {"Tc_Naph", 83.4,
     3.75}, {"Tc_Naph", 86.4, 3.728}, {"Tc_Naph", 88.1,
     3.725}, {"Tc_Naph", 90.2, 3.716}, {"Tc_Naph", 93.1,
     3.704}, {"Tc_Naph", 94.7, 3.673}, {"Tc_Naph", 97.7,
     3.684}, {"Tc_Naph", 97.9, 3.665}}}}

Clearly it is best in future to record data in an easier format - 
whatever language you use to process it!

You may want to lookup StringReplace and StringExpression to understand 
the above, and help you with other problems of this  type.

David Bailey
http://www.dbaileyconsultancy.co.uk

[toc] | [prev] | [next] | [standalone]


#14870

FromHelen Read <readhpr@gmail.com>
Date2013-06-15 08:13 +0000
Message-ID<kph7ml$2dd$1@smc.vnet.net>
In reply to#14859
For starters, make sure you import your file as a Table.

data = Import["filename", "Table"]

Now find where Run 1 and Run 2 begin.


p1 = Position[data, {"Run", 1}][[1, 1]]

p2 = Position[data, {"Run", 2}][[1, 1]]


For Run 1, we want the part of the table from just after p1 to right 
before p2. Delete the blank rows {} while you are at it.

run1 = DeleteCases[data[[p1 + 1 ;; p2 - 1]], {}]


For Run 2, we want everything from right after p2 to the end.

run2 = DeleteCases[data[[p2 + 1 ;; Length[data]]], {}]

So now we have two tables, run1 and run2. We need to extract the 
temperature and time from each row of the tables. The rows look for 
example like the following.

{"Tc_Naph_83_2C", 3.758, "ns"}

The basic idea is to construct a function to operate on one row of the 
table, then Map it across the table. You can do this quite compactly 
using pure functions, but if you are new to this sort of thing you might 
find it easier to follow if you write an explicit function that operates 
on one row of the table at a time. I will use x to represent one row.

To begin, we get the first two parts of row x.

x[[1]] contains the temperature information (but needs some processing).
x[[2]] is the time.

In the example row, x[[1]] is  "Tc_Naph_83_2C"

Use StringSplit on the "_"  to get at the pieces.

StringSplit["Tc_Naph_83_2C", "_"]
results in {"Tc", "Naph", "83", "2C"}

Now, we get the 2nd-to-the-last part of this (the "83" in the example), 
and the last part (the "2C"). We drop the last character from the "2C". 
Then join the "83" with a "." and the "2". Finally, use ToExpression to 
convert "83.2" from a string to a number.

Putting this all together, here is a function to extract the temperature 
and time from one row of the table. I've used some local variables 
{split,temperature,time} to keep track of things as we go.

f[x_] := Module[{split, temperature, time},
   time = x[[2]];
   split = StringSplit[x[[1]], "_"];
   temperature =
    ToExpression[
     StringJoin[split[[-2]], ".", StringDrop[split[[-1]], -1]]];
   {temperature, time}
   ]

The input of the function, x, represents one row of the table. The 
output is the ordered pair {temperature,time}

Test it out on a single row.

f[{"Tc_Naph_84_2C", 3.74`, "ns"}]

Now simply Map the function over each run.

Map[f, run1]

Map[f,run2]


Helen Read
University of Vermont


On 6/14/2013 4:54 AM, howardfink@gmail.com wrote:
> I have a series of files of this form:
>
> June 7, 2013
> Tc+Naphthalene  vs Temperature (oC)
>
> Run 1
> Tc_Naph_84_2C			3.740 ns
> Tc_Naph_87_1C			3.731 ns
> Tc_Naph_89_9C			3.720 ns
> Tc_Naph_92_9C			3.704 ns
> Tc_Naph_94_7C			3.687 ns
> Tc_Naph_97_6C			3.694 ns
>
>
> Run 2
> Tc_Naph_83_2C			3.758 ns
> Tc_Naph_83_4C			3.750 ns
> Tc_Naph_86_4C			3.728 ns
> Tc_Naph_88_1C			3.725 ns
> Tc_Naph_90_2C			3.716 ns
> Tc_Naph_93_1C			3.704 ns
> Tc_Naph_94_7C			3.673 ns
> Tc_Naph_97_7C			3.684 ns
> Tc_Naph_97_9C			3.665 ns
>
>
>
>
> I used an Import command to read in the file, but now I am just sitting and=
>   staring, without a clue how to get the 84_2 converted to the number 84.2,e=
> tc. and ending up with two lists: Run 1 and Run 2, consisting of pairs of t=
> emperature and time.  The temperature will eventually be converted to 1/abs=
> olute temperature.
>
> I've read lots and lots of help, thumbed through dozens of pages of a Mathe=
> matica 5 manual, and don't know where to start.  I'm trying to help a 90-ye=
> ar-old chemistry professor, who is currently using a calculator, but  there=
>   will be dozens of runs of this experiment.
>


.

[toc] | [prev] | [standalone]


Back to top | Article view | comp.soft-sys.math.mathematica


csiph-web