Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.mixmin.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.003 X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'subject:Django': 0.03; 'say,': 0.05; 'subject:Python': 0.06; 'append': 0.09; 'extracted': 0.09; 'links.': 0.09; 'logic': 0.09; 'pages.': 0.09; 'req': 0.09; 'try:': 0.09; 'python': 0.11; 'django': 0.11; 'def': 0.12; '2.7': 0.14; 'template': 0.14; 'windows': 0.15; 'endif': 0.16; 'foo()': 0.16; 'foo():': 0.16; 'skip:} 10': 0.16; 'soup': 0.16; 'url:example': 0.16; 'url:foo': 0.16; 'code.': 0.18; '8bit%:5': 0.22; 'programming': 0.22; 'previously': 0.22; 'putting': 0.22; ' To: python-list@python.org Content-Type: multipart/alternative; boundary=e89a8fb2088874105804eed61618 X-Mailman-Approved-At: Tue, 31 Dec 2013 18:04:31 +0100 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 281 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1388509472 news.xs4all.nl 2868 [2001:888:2000:d::a6]:43512 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:62934 --e89a8fb2088874105804eed61618 Content-Type: text/plain; charset=ISO-8859-1 I am putting together a project using Python 2.7 Django 1.5 on Windows 7. I believe this should be on the django group but I haven't had help from there so I figured I would try the python list I have the following view: views.py: def foo(): site = "http://www.foo.com/portal/jobs" hdr = {'User-Agent' : 'Mozilla/5.0'} req = urllib2.Request(site, headers=hdr) jobpass = urllib2.urlopen(req) soup = BeautifulSoup(jobpass) for tag in soup.find_all('a', href = True): tag['href'] = urlparse.urljoin('http://www.businessghana.com/portal/', tag['href']) return map(str, soup.find_all('a', href = re.compile('.getJobInfo'))) def example(): site = "http://example.com" hdr = {'User-Agent' : 'Mozilla/5.0'} req = urllib2.Request(site, headers=hdr) jobpass = urllib2.urlopen(req) soup = BeautifulSoup(jobpass) return map(str, soup.find_all('a', href = re.compile('.display-job'))) foo_links = foo() example_links = example() def all_links(): return (foo_links + example_links) def display_links(request): name = all_links() paginator = Paginator(name, 25) page = request.GET.get('page') try: name = paginator.page(page) except PageNotAnInteger: name = paginator.page(1) except EmptyPage: name = paginator.page(paginator.num_pages) return render_to_response('jobs.html', {'name' : name}) My template looks like this:
    {% for link in name %}
  1. {{ link|safe }}
    2. {% endfor %}
Right now as my code stands, anytime I run it, it scraps all the links on the frontpage of the sites selected and presents them paginated *all afresh*. However, I don't think its a good idea for the script to read/write all the links that had previously extracted links all over again and therefore would like to check for and append only new links. I would like to save the previously scraped links so that over the course of say, a week, all the links that have appeared on the frontpage of these sites will be available on my site as older pages. It's my first programming project and don't know how to incorporate this logic into my code. Any help/pointers/references will be greatly appreciated. regards, Max --e89a8fb2088874105804eed61618 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I am putting together a project using Python 2.7 Django 1.5 on Windows 7. <= /div>
I believe this should be on the django group but I haven't had help fro= m there so I figured I would try the python list
I have the following view:
views.py:

def foo():
=A0=A0=A0 site =3D "http://www.foo.com/portal/jobs"
=A0=A0= =A0 hdr =3D {'User-Agent' : 'Mozilla/5.0'}
=A0=A0=A0 req= =3D urllib2.Request(site, headers=3Dhdr)
=A0=A0=A0 jobpass =3D urllib2.urlopen(req)
=A0=A0=A0 soup =3D BeautifulS= oup(jobpass)
=A0=A0=A0 for tag in soup.find_all('a', href =3D Tr= ue):
=A0=A0=A0=A0=A0=A0=A0 tag['href'] =3D urlparse.urljoin('= ;http://= www.businessghana.com/portal/',=A0 tag['href'])
=A0=A0=A0 return map(str, soup.find_all('a', href =3D re.compile(&#= 39;.getJobInfo')))
def example():
=A0=A0=A0 site =3D "http://example.com"
=A0=A0=A0 hdr =3D {'Us= er-Agent' : 'Mozilla/5.0'}
=A0=A0=A0 req =3D urllib2.Request= (site, headers=3Dhdr)
=A0=A0=A0 jobpass =3D urllib2.urlopen(req)
=A0=A0=A0 soup =3D BeautifulSoup(jobpass)
=A0=A0=A0 return map(str, soup= .find_all('a', href =3D re.compile('.display-job')))
<= div style=3D"font:14px/18px Arial,"Liberation Sans","DejaVu = Sans",sans-serif;margin:0px 0px 1em;padding:0px;border:0px currentColo= r;text-align:left;text-transform:none;text-indent:0px;letter-spacing:normal= ;clear:both;word-spacing:0px;vertical-align:baseline;white-space:normal;fon= t-size-adjust:none;font-stretch:normal"> =A0foo_links =3D foo()
=A0example_links =3D example()
def all_links():
=A0=A0=A0 return (foo_links + example_links)
def display_links(request):
=A0=A0=A0 name =3D all_links()
=A0=A0=A0 = paginator =3D Paginator(name, 25)
=A0=A0=A0 page =3D request.GET.get(= 9;page')
=A0=A0=A0 try:
=A0=A0=A0=A0=A0=A0=A0 name =3D paginator.= page(page)
=A0=A0=A0 except PageNotAnInteger:
=A0=A0=A0=A0=A0=A0=A0 name =3D paginator.page(1)
=A0=A0=A0 except EmptyP= age:
=A0=A0=A0=A0=A0=A0=A0 name =3D paginator.page(paginator.num_pages)<= /div>
=A0=A0=A0 return render_to_response('jobs.html', {'name' : = name})=A0=A0

My template looks like this:

<ol>
{% for link in name %}
=A0 <li> {{ link|safe }}</= li>
{% endfor %}
=A0</ol>
=A0<div class=3D"pagina= tion">
<span class=3D "step-links">
=A0=A0=A0= {% if name.has_previous %}
=A0=A0=A0=A0=A0=A0=A0 <a href=3D"?page=3D{{ names.previous_page_num= ber }}">Previous</a>
=A0=A0=A0 {% endif %}
=A0=A0=A0 <span class =3D "current">
=A0=A0=A0=A0=A0=A0= =A0 Page {{ name.number }} of {{ name.paginator.num_pages}}.
=A0=A0=A0 &= lt;/span>
=A0=A0=A0 {% if name.has_next %}
=A0=A0=A0=A0=A0=A0=A0 <a href=3D&quo= t;?page=3D{{ name.next_page_number}}">next</a>
=A0=A0=A0 {= % endif %}
</span>
=A0</div>

Right now as my code stands, anytime I run it, it scraps all the links on t= he frontpage of the sites selected and presents them paginated=A0all afresh. However, I don't think its a good idea for the = script to read/write all the links that had previously extracted links all = over again and therefore would like to check for and append only new links.= I would like to save the previously scraped links so that over the course = of say, a week, all the links that have appeared on the frontpage of these = sites will be available on my site as older pages.

It's my first programming project and don't know how to incorporate= this logic into my code.

Any help/pointers/references will be greatly appreciated.

regards, Max

--e89a8fb2088874105804eed61618--