Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!feeder.news-service.com!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Mon, 12 Sep 2011 21:42:21 +0100
From: MRAB <python@mrabarnett.plus.com>
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:6.0.2) Gecko/20110902 Thunderbird/6.0.2
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Python: Deleting specific words from a file.
References: <30f9b718-bb3c-4c92-8a03-0f760c993939@a12g2000yqi.googlegroups.com> <mailman.898.1315548327.27778.python-list@python.org> <22e15921-d543-4604-8996-3bcb2770ef18@et6g2000vbb.googlegroups.com>
In-Reply-To: <22e15921-d543-4604-8996-3bcb2770ef18@et6g2000vbb.googlegroups.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Reply-To: python-list@python.org
Newsgroups: comp.lang.python
Message-ID: <mailman.1046.1315860149.27778.python-list@python.org>
Lines: 70
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:13197

On 12/09/2011 20:49, gry wrote:
> On Sep 9, 2:04 am, Terry Reedy<tjre...@udel.edu>  wrote:
>> On 9/8/2011 9:09 PM, papu wrote:
>>
>>
>>
>>> Hello, I have a data file (un-structed messy file) from which I have
>>> to scrub specific list of words (delete words).
>>
>>> Here is what I am doing but with no result:
>>
>>> infile = "messy_data_file.txt"
>>> outfile = "cleaned_file.txt"
>>
>>> delete_list = ["word_1","word_2"....,"word_n"]
>>> new_file = []
>>> fin=open(infile,"")
>>> fout = open(outfile,"w+")
>>> for line in fin:
>>>       for word in delete_list:
>>>           line.replace(word, "")
>>>       fout.write(line)
>>> fin.close()
>>> fout.close()
>>
>> If you have very many words (and you will need all possible forms of
>> each word if you do exact matches), The following (untested and
>> incomplete) should run faster.
>>
>> delete_set = {"word_1","word_2"....,"word_n"}
>> ...
>> for line in fin:
>>       for word in line.split()
>>           if word not in delete_set:
>>               fout.write(word) # also write space and nl.
>>
>> Depending on what your file is like, you might be better with
>> re.split('(\W+)', line). An example from the manual:
>>   >>>  re.split('(\W+)', '...words, words...')
>> ['', '...', 'words', ', ', 'words', '...', '']
>>
>> so all non-word separator sequences are preserved and written back out
>> (as they will not match delete set).
>>
>> --
>> Terry Jan Reedy
>
> re.sub is handy too:
> import re
> delete_list=('the','rain','in','spain')
> regex =  re.compile('\W' + '|'.join(delete_list) + '\W')

You need parentheses around the words (I'm using non-capturing
parentheses):

regex =  re.compile(r'\W(?:' + '|'.join(delete_list) + r')\W')

otherwise you'd get: '\Wthe|rain|in|spain\W'.

Even better is the word-boundary, in case there's no previous or next
character:

regex =  re.compile(r'\b(?:' + '|'.join(delete_list) + r')\b')

Raw string literals are recommended for regexes.

> infile='messy'
> with open(infile, 'r') as f:
>      for l in f:
>          print regex.sub('', l)