Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #12986 > unrolled thread

Python: Deleting specific words from a file.

Back to article view | Back to comp.lang.python

Contents

  Python: Deleting specific words from a file. papu <prachar@gmail.com> - 2011-09-08 18:09 -0700
    Re: Python: Deleting specific words from a file. MRAB <python@mrabarnett.plus.com> - 2011-09-09 02:31 +0100
    Re: Python: Deleting specific words from a file. John Gordon <gordon@panix.com> - 2011-09-09 03:16 +0000
    Re: Python: Deleting specific words from a file. Terry Reedy <tjreedy@udel.edu> - 2011-09-09 02:04 -0400
      Re: Python: Deleting specific words from a file. gry <georgeryoung@gmail.com> - 2011-09-12 12:49 -0700
        Re: Python: Deleting specific words from a file. MRAB <python@mrabarnett.plus.com> - 2011-09-12 21:42 +0100

#12986 — Python: Deleting specific words from a file.

Hello, I have a data file (un-structed messy file) from which I have
to scrub specific list of words (delete words).

Here is what I am doing but with no result:

infile = "messy_data_file.txt"
outfile = "cleaned_file.txt"

delete_list = ["word_1","word_2"....,"word_n"]
new_file = []
fin=open(infile,"")
fout = open(outfile,"w+")
for line in fin:
    for word in delete_list:
        line.replace(word, "")
    fout.write(line)
fin.close()
fout.close()

I have put the code above in a file. When I execute it, I dont see the
result file. I am not sure what the error is. Please let me know what
I am doing wrong.

[toc] | [next] | [standalone]

#12987

On 09/09/2011 02:09, papu wrote:
> Hello, I have a data file (un-structed messy file) from which I have
> to scrub specific list of words (delete words).
>
> Here is what I am doing but with no result:
>
> infile = "messy_data_file.txt"
> outfile = "cleaned_file.txt"
>
> delete_list = ["word_1","word_2"....,"word_n"]
> new_file = []
> fin=open(infile,"")
> fout = open(outfile,"w+")
> for line in fin:
>      for word in delete_list:
>          line.replace(word, "")
>      fout.write(line)
> fin.close()
> fout.close()
>
> I have put the code above in a file. When I execute it, I dont see the
> result file. I am not sure what the error is. Please let me know what
> I am doing wrong.

The .replace method _returns_ its result.

Strings are immutable, they can't be changed in-place.

[toc] | [prev] | [next] | [standalone]

#12996

In <30f9b718-bb3c-4c92-8a03-0f760c993939@a12g2000yqi.googlegroups.com> papu <prachar@gmail.com> writes:

> Hello, I have a data file (un-structed messy file) from which I have
> to scrub specific list of words (delete words).

> Here is what I am doing but with no result:

> infile = "messy_data_file.txt"
> outfile = "cleaned_file.txt"

> delete_list = ["word_1","word_2"....,"word_n"]
> new_file = []

What does new_file do?  I don't see it used anywhere.

> fin=open(infile,"")

There should be an "r" inside those quotes.  In fact this is an error
and it will stop your program from running.

> fout = open(outfile,"w+")

What is the "+" supposed to do?

> for line in fin:
>     for word in delete_list:
>         line.replace(word, "")

replace() returns the modified string; it does not alter the existing
string.

Do this instead:

  line = line.replace(word, "")

>     fout.write(line)
> fin.close()
> fout.close()

> I have put the code above in a file. When I execute it, I dont see the
> result file. I am not sure what the error is. Please let me know what
> I am doing wrong.

When you say you don't see the result file, do you mean it doesn't get
created at all?

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon@panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"

[toc] | [prev] | [next] | [standalone]

#13003

On 9/8/2011 9:09 PM, papu wrote:
> Hello, I have a data file (un-structed messy file) from which I have
> to scrub specific list of words (delete words).
>
> Here is what I am doing but with no result:
>
> infile = "messy_data_file.txt"
> outfile = "cleaned_file.txt"
>
> delete_list = ["word_1","word_2"....,"word_n"]
> new_file = []
> fin=open(infile,"")
> fout = open(outfile,"w+")
> for line in fin:
>      for word in delete_list:
>          line.replace(word, "")
>      fout.write(line)
> fin.close()
> fout.close()

If you have very many words (and you will need all possible forms of 
each word if you do exact matches), The following (untested and 
incomplete) should run faster.

delete_set = {"word_1","word_2"....,"word_n"}
...
for line in fin:
     for word in line.split()
         if word not in delete_set:
             fout.write(word) # also write space and nl.


Depending on what your file is like, you might be better with 
re.split('(\W+)', line). An example from the manual:
 >>> re.split('(\W+)', '...words, words...')
['', '...', 'words', ', ', 'words', '...', '']

so all non-word separator sequences are preserved and written back out 
(as they will not match delete set).

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#13196

On Sep 9, 2:04 am, Terry Reedy <tjre...@udel.edu> wrote:
> On 9/8/2011 9:09 PM, papu wrote:
>
>
>
> > Hello, I have a data file (un-structed messy file) from which I have
> > to scrub specific list of words (delete words).
>
> > Here is what I am doing but with no result:
>
> > infile = "messy_data_file.txt"
> > outfile = "cleaned_file.txt"
>
> > delete_list = ["word_1","word_2"....,"word_n"]
> > new_file = []
> > fin=open(infile,"")
> > fout = open(outfile,"w+")
> > for line in fin:
> >      for word in delete_list:
> >          line.replace(word, "")
> >      fout.write(line)
> > fin.close()
> > fout.close()
>
> If you have very many words (and you will need all possible forms of
> each word if you do exact matches), The following (untested and
> incomplete) should run faster.
>
> delete_set = {"word_1","word_2"....,"word_n"}
> ...
> for line in fin:
>      for word in line.split()
>          if word not in delete_set:
>              fout.write(word) # also write space and nl.
>
> Depending on what your file is like, you might be better with
> re.split('(\W+)', line). An example from the manual:
>  >>> re.split('(\W+)', '...words, words...')
> ['', '...', 'words', ', ', 'words', '...', '']
>
> so all non-word separator sequences are preserved and written back out
> (as they will not match delete set).
>
> --
> Terry Jan Reedy

re.sub is handy too:
import re
delete_list=('the','rain','in','spain')
regex =  re.compile('\W' + '|'.join(delete_list) + '\W')
infile='messy'
with open(infile, 'r') as f:
    for l in f:
        print regex.sub('', l)

[toc] | [prev] | [next] | [standalone]

#13197

On 12/09/2011 20:49, gry wrote:
> On Sep 9, 2:04 am, Terry Reedy<tjre...@udel.edu>  wrote:
>> On 9/8/2011 9:09 PM, papu wrote:
>>
>>
>>
>>> Hello, I have a data file (un-structed messy file) from which I have
>>> to scrub specific list of words (delete words).
>>
>>> Here is what I am doing but with no result:
>>
>>> infile = "messy_data_file.txt"
>>> outfile = "cleaned_file.txt"
>>
>>> delete_list = ["word_1","word_2"....,"word_n"]
>>> new_file = []
>>> fin=open(infile,"")
>>> fout = open(outfile,"w+")
>>> for line in fin:
>>>       for word in delete_list:
>>>           line.replace(word, "")
>>>       fout.write(line)
>>> fin.close()
>>> fout.close()
>>
>> If you have very many words (and you will need all possible forms of
>> each word if you do exact matches), The following (untested and
>> incomplete) should run faster.
>>
>> delete_set = {"word_1","word_2"....,"word_n"}
>> ...
>> for line in fin:
>>       for word in line.split()
>>           if word not in delete_set:
>>               fout.write(word) # also write space and nl.
>>
>> Depending on what your file is like, you might be better with
>> re.split('(\W+)', line). An example from the manual:
>>   >>>  re.split('(\W+)', '...words, words...')
>> ['', '...', 'words', ', ', 'words', '...', '']
>>
>> so all non-word separator sequences are preserved and written back out
>> (as they will not match delete set).
>>
>> --
>> Terry Jan Reedy
>
> re.sub is handy too:
> import re
> delete_list=('the','rain','in','spain')
> regex =  re.compile('\W' + '|'.join(delete_list) + '\W')

You need parentheses around the words (I'm using non-capturing
parentheses):

regex =  re.compile(r'\W(?:' + '|'.join(delete_list) + r')\W')

otherwise you'd get: '\Wthe|rain|in|spain\W'.

Even better is the word-boundary, in case there's no previous or next
character:

regex =  re.compile(r'\b(?:' + '|'.join(delete_list) + r')\b')

Raw string literals are recommended for regexes.

> infile='messy'
> with open(infile, 'r') as f:
>      for l in f:
>          print regex.sub('', l)

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web