Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed4a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
Date: Sat, 25 Jan 2014 13:22:14 -0800
Subject: Pls help me...I want to save scraped data automatically to my database(cleaner version)
From: Max Cuban <edzeame@gmail.com>
To: python-list@python.org
Content-Type: multipart/alternative; boundary=001a113653707e0eeb04f0d2119c
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.5982.1390684944.18130.python-list@python.org>
Lines: 126
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:64753

--001a113653707e0eeb04f0d2119c
Content-Type: text/plain; charset=ISO-8859-1

I have asked this question earlier but this should make more sense than the
earlier version and I don't want anyone who could potentially helped to be
put off by the initial mess even if I updated it with my cleaner version as
a reply

I want to save the links scraped to be save in my database so that on
subsequent run, it only scrapes and append only new links to the list.

This is my code below but  at the end of the day my database is empty. What
changes can I make to overcome this? Thanks in advance


    from django.template.loader import get_template
    from django.shortcuts import render_to_response
    from bs4 import BeautifulSoup
    import urllib2, sys
    import urlparse
    import re
    from listing.models import jobLinks
 #this function extract the links
    def businessghana():
        site = "http://www.businessghana.com/portal/jobs"
        hdr = {'User-Agent' : 'Mozilla/5.0'}
        req = urllib2.Request(site, headers=hdr)
        jobpass = urllib2.urlopen(req)
        soup = BeautifulSoup(jobpass)
        for tag in soup.find_all('a', href = True):
            tag['href'] = urlparse.urljoin('
http://www.businessghana.com/portal/', tag['href'])
        return map(str, soup.find_all('a', href =
re.compile('.getJobInfo')))
 # result from businssghana() saved to a variable to make them iterable as
a list
    all_links  = businessghana()

#this function should be saving the links to the database unless the link
already exist
    def save_new_links(all_links):
        current_links = jobLinks.objects.all()
        for i in all_links:
            if i not in current_links:
                jobLinks.objects.create(url=i)

# I called the above function here hoping that it will save to database
    save_new_links(all_links)

# return my httpResponse with this function
    def display_links(request):
        name = all_links()
        return render_to_response('jobs.html', {'name' : name})
 My django models.py looks like this:
 from django.db import models class jobLinks(models.Model): links =
models.URLField() pub_date = models.DateTimeField('date retrieved') def
__unicode__(self): return self.links

--001a113653707e0eeb04f0d2119c
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><br></div><div>I have asked this question earlier but=
 this should make more sense than the earlier version and I don&#39;t want =
anyone who could potentially helped to be put off by the initial mess even =
if I updated it with my cleaner version as a reply=A0</div>
<div><br></div><div>I want to save the links scraped to be save in my datab=
ase so that on subsequent run, it only scrapes and append only new links to=
 the list.</div><div><br></div><div>This is my code below but =A0at the end=
 of the day my database is empty. What changes can I make to overcome this?=
 Thanks in advance</div>
<div><br></div><div>=A0 =A0</div><div>=A0 =A0 from django.template.loader i=
mport get_template</div><div>=A0 =A0 from django.shortcuts import render_to=
_response=A0</div><div>=A0 =A0 from bs4 import BeautifulSoup</div><div>=A0 =
=A0 import urllib2, sys</div>
<div>=A0 =A0 import urlparse</div><div>=A0 =A0 import re</div><div>=A0 =A0 =
from listing.models import jobLinks</div><div><span class=3D"" style=3D"whi=
te-space:pre">	</span></div><div><span class=3D"" style=3D"white-space:pre"=
>	</span>#this function extract the links</div>
<div>=A0 =A0 def businessghana():</div><div>=A0 =A0 =A0 =A0 site =3D &quot;=
<a href=3D"http://www.businessghana.com/portal/jobs">http://www.businessgha=
na.com/portal/jobs</a>&quot;</div><div>=A0 =A0 =A0 =A0 hdr =3D {&#39;User-A=
gent&#39; : &#39;Mozilla/5.0&#39;}</div>
<div>=A0 =A0 =A0 =A0 req =3D urllib2.Request(site, headers=3Dhdr)</div><div=
>=A0 =A0 =A0 =A0 jobpass =3D urllib2.urlopen(req)</div><div>=A0 =A0 =A0 =A0=
 soup =3D BeautifulSoup(jobpass)</div><div>=A0 =A0 =A0 =A0 for tag in soup.=
find_all(&#39;a&#39;, href =3D True):</div>
<div>=A0 =A0 =A0 =A0 =A0 =A0 tag[&#39;href&#39;] =3D urlparse.urljoin(&#39;=
<a href=3D"http://www.businessghana.com/portal/">http://www.businessghana.c=
om/portal/</a>&#39;, tag[&#39;href&#39;])</div><div>=A0 =A0 =A0 =A0 return =
map(str, soup.find_all(&#39;a&#39;, href =3D re.compile(&#39;.getJobInfo=
9;)))</div>
<div><span class=3D"" style=3D"white-space:pre">	</span></div><div><span cl=
ass=3D"" style=3D"white-space:pre">	</span># result from businssghana() sav=
ed to a variable to make them iterable as a list</div><div>=A0 =A0 all_link=
s =A0=3D businessghana()</div>
<div><br></div><div><span class=3D"" style=3D"white-space:pre">	</span>#thi=
s function should be saving the links to the database unless the link alrea=
dy exist</div><div>=A0 =A0 def save_new_links(all_links):</div><div>=A0 =A0=
 =A0 =A0 current_links =3D jobLinks.objects.all()</div>
<div>=A0 =A0 =A0 =A0 for i in all_links:</div><div>=A0 =A0 =A0 =A0 =A0 =A0 =
if i not in current_links:</div><div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 jobLin=
ks.objects.create(url=3Di)</div><div>=A0 =A0=A0</div><div><span class=3D"" =
style=3D"white-space:pre">	</span># I called the above function here hoping=
 that it will save to database</div>
<div>=A0 =A0 save_new_links(all_links)</div><div><br></div><div><span class=
=3D"" style=3D"white-space:pre">	</span># return my httpResponse with this =
function</div><div>=A0 =A0 def display_links(request):</div><div>=A0 =A0 =
=A0 =A0 name =3D all_links() =A0 =A0=A0</div>
<div>=A0 =A0 =A0 =A0 return render_to_response(&#39;jobs.html&#39;, {&#39;n=
ame&#39; : name})</div><div><span class=3D"" style=3D"white-space:pre">		</=
span></div><div><span class=3D"" style=3D"white-space:pre">	</span></div><d=
iv><span class=3D"" style=3D"white-space:pre">	</span>My django models.py l=
ooks like this:</div>
<div><span class=3D"" style=3D"white-space:pre">	</span></div><div><span st=
yle=3D"white-space:pre">    from django.db import models

    class jobLinks(models.Model):
        links =3D models.URLField()
        pub_date =3D models.DateTimeField(&#39;date retrieved&#39;)
 =20
        def __unicode__(self):
            return self.links</span></div><span style=3D"white-space:pre">=
=A0<br></span><div><br></div></div>

--001a113653707e0eeb04f0d2119c--