Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #13003

Re: Python: Deleting specific words from a file.

Show all headers | View raw

On 9/8/2011 9:09 PM, papu wrote:
> Hello, I have a data file (un-structed messy file) from which I have
> to scrub specific list of words (delete words).
>
> Here is what I am doing but with no result:
>
> infile = "messy_data_file.txt"
> outfile = "cleaned_file.txt"
>
> delete_list = ["word_1","word_2"....,"word_n"]
> new_file = []
> fin=open(infile,"")
> fout = open(outfile,"w+")
> for line in fin:
>      for word in delete_list:
>          line.replace(word, "")
>      fout.write(line)
> fin.close()
> fout.close()

If you have very many words (and you will need all possible forms of 
each word if you do exact matches), The following (untested and 
incomplete) should run faster.

delete_set = {"word_1","word_2"....,"word_n"}
...
for line in fin:
     for word in line.split()
         if word not in delete_set:
             fout.write(word) # also write space and nl.


Depending on what your file is like, you might be better with 
re.split('(\W+)', line). An example from the manual:
 >>> re.split('(\W+)', '...words, words...')
['', '...', 'words', ', ', 'words', '...', '']

so all non-word separator sequences are preserved and written back out 
(as they will not match delete set).

-- 
Terry Jan Reedy

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Python: Deleting specific words from a file. papu <prachar@gmail.com> - 2011-09-08 18:09 -0700
  Re: Python: Deleting specific words from a file. MRAB <python@mrabarnett.plus.com> - 2011-09-09 02:31 +0100
  Re: Python: Deleting specific words from a file. John Gordon <gordon@panix.com> - 2011-09-09 03:16 +0000
  Re: Python: Deleting specific words from a file. Terry Reedy <tjreedy@udel.edu> - 2011-09-09 02:04 -0400
    Re: Python: Deleting specific words from a file. gry <georgeryoung@gmail.com> - 2011-09-12 12:49 -0700
      Re: Python: Deleting specific words from a file. MRAB <python@mrabarnett.plus.com> - 2011-09-12 21:42 +0100

csiph-web