Re: How can I export data from a website and write the contents to a text file?

From	Denis McMahon <denismfmcmahon@gmail.com>
Newsgroups	comp.lang.python
Subject	Re: How can I export data from a website and write the contents to a text file?
Date	2015-11-18 17:19 +0000
Organization	A noiseless patient Spider
Message-ID	<n2ibuh$bm9$2@dont-email.me> (permalink)
References	<9365cf2f-e9c7-4338-83b4-ce3d1d7ce1d6@googlegroups.com>

Show all headers | View raw

On Wed, 18 Nov 2015 08:37:47 -0800, ryguy7272 wrote:

> I'm trying the script below...

The problem isn't that you're over-writing the lines (although it may 
seem that way to you), the problem is that you're overwriting the whole 
file every time you write a link to it. This is because you open and 
close the file for every link you write, and you do so in file mode "wb" 
which restarts writing at the first byte of the file every time.

You only need to open and close the text file once, instead of for every 
link you output. Try moving the lines to open and close the file outside 
the outer for loop to change the loop from:

for item in soup.find_all(class_='lister-list'):
    for link in item.find_all('a'):
        # open file
        # write link to file
        # close file

to:

# open file
for item in soup.find_all(class_='lister-list'):
    for link in item.find_all('a'):
        # write link to file
# close file

Alternatively, use the with form:

with open("blah","wb") as text_file:
    for item in soup.find_all(class_='lister-list'):
        for link in item.find_all('a'):
            # write link to file

-- 
Denis McMahon, denismfmcmahon@gmail.com

Thread

csiph-web