Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder2.enfer-du-nord.net!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <BANLkTikF95KTC2-noGoPx=NL0NtrE7wO7Q@mail.gmail.com>
References: <BANLkTikF95KTC2-noGoPx=NL0NtrE7wO7Q@mail.gmail.com>
Date: Sun, 12 Jun 2011 23:08:05 -0700
Subject: Re: Subsetting a dataset
From: Benjamin Kaplan <benjamin.kaplan@case.edu>
To: python-list@python.org
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.161.1307945290.11593.python-list@python.org>
Lines: 58
NNTP-Posting-Host: 82.94.164.166
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:7503

On Sun, Jun 12, 2011 at 9:53 PM, Kumar Mainali <kpmainali@gmail.com> wrote:
> I have a huge dataset containing millions of rows and several dozen colum=
ns
> in a tab delimited text file. =A0I need to extract a small subset of rows=
 and
> only three columns. One of the three columns has two word string with hea=
der
> =93Scientific Name=94. The other two columns carry numbers for Longitude =
and
> Latitude, as below.
> Sci Name Longitude Latitude Column4
> Gen sp1 82.5 28.4 =85
> Gen sp2 45.9 29.7 =85
> Gen sp1 57.9 32.9 =85
> =85 =85 =85 =85
> Of the many species listed under the column =93Sci Name=94, I am interest=
ed in
> only one species which will have multiple records interspersed in the
> millions of rows, and I will probably have to use filename.readline() to
> read the rows one at a time. How would I search for a particular species =
in
> the dataset and create a new dataset for the species with only the three
> columns?
> Next, I have to create such datasets for hundreds of species. All these
> species are listed in another text file. There must be a way to define an
> iterative function that looks at one species at a time in the list of
> species and creates separate dataset for each species. The huge dataset
> contains more species than those listed in the list of my interest.
> I very much appreciate any help. I am a beginner in Python. So, complete
> code would be more helpful.
> - Kumar

Read in the file with the lists of species. For each line in that
list, open up a file and then put it into a dictionary where the key
is the species name and the value is the file. Then, once you have all
your files created, open up the second file. For each line in the
second file, split it on the tabs and check to see if the first item
is in the dict. If it is, grab your necessary values and write it to
that corresponding file. In rough, untested code

animals =3D dict()
for line in open('species_list) :
    #make a file for that animal and associate it with the name
    animals[line.strip()] =3D open('%s_data.csv' % line.strip(),'w')


#now open the second file
for line in open('animal_data') :
    data =3D line.split('\t')
    if data[name_col] in animals :
        animals[data[name_col]].write('%s\t%s\t%s' & (data[name_col],
data[lat_col], data[lon_col])

replacing the respective file names and column numbers as appropriate,
of course.

>
>