Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.lang.python > #79110

Re:downloading from links within a webpage

Path csiph.com!usenet.pasdenom.info!news.franciliens.net!fdn.fr!usenet-fr.net!nerim.net!novso.com!newsfeed.xs4all.nl!newsfeed3a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path <python-python-list@m.gmane.org>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.003
X-Spam-Evidence '*H*': 0.99; '*S*': 0.00; 'matches': 0.07; 'arguments,': 0.09; 'definition,': 0.09; 'function,': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'to)': 0.09; 'python': 0.11; 'wrote': 0.14; 'fetch': 0.16; 'finds': 0.16; 'limiting': 0.16; 'received:80.91.229.3': 0.16; 'received:97': 0.16; 'received:plane.gmane.org': 0.16; 'regex,': 0.16; 'skip:u 50': 0.16; 'url=""': 0.16; '(you': 0.16; 'thanks,': 0.17; 'import': 0.22; 'header:X-Complaints-To:1': 0.27; 'function': 0.29; 'rest': 0.29; 'external': 0.29; 'moved': 0.30; 'specified': 0.30; "i'm": 0.30; 'code': 0.31; 'depth': 0.31; 'safely': 0.31; 'skip:m 30': 0.32; 'subject:from': 0.34; 'skip:u 20': 0.35; 'something': 0.35; 'add': 0.35; 'doing': 0.36; 'hi,': 0.36; 'should': 0.36; 'searching': 0.37; 'two': 0.37; 'level': 0.37; 'starting': 0.37; 'to:addr:python-list': 0.38; 'to:addr:python.org': 0.39; 'enough': 0.39; 'received:org': 0.40; 'called': 0.40; 'how': 0.40; 'skip:u 10': 0.60; 'most': 0.60; 'first': 0.61; 'here': 0.66; 'webpage': 0.68; 'limit': 0.70; 'ending': 0.78; '3.4': 0.84; 'subject::': 0.85
X-Injected-Via-Gmane http://gmane.org/
To python-list@python.org
From Dave Angel <davea@davea.name>
Subject Re:downloading from links within a webpage
Date Tue, 14 Oct 2014 13:47:20 -0400 (EDT)
Organization news.gmane.org
References <loom.20141014T163517-784@post.gmane.org>
Mime-Version 1.0
Content-Type text/plain; charset=UTF-8
Content-Transfer-Encoding 7bit
X-Gmane-NNTP-Posting-Host 97.73.240.9
X-Newsreader PiaoHong.Usenet.Client.VIP:1.62
X-Archive encrypt
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.15
Precedence list
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list/>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups comp.lang.python
Message-ID <mailman.14857.1413308664.18130.python-list@python.org> (permalink)
Lines 55
NNTP-Posting-Host 2001:888:2000:d::a6
X-Trace 1413308664 news.xs4all.nl 2973 [2001:888:2000:d::a6]:44499
X-Complaints-To abuse@xs4all.nl
Xref csiph.com comp.lang.python:79110

Show key headers only | View raw


Shiva <shivaji_tn@yahoo.com.dmarc.invalid> Wrote in message:
> Hi,
> 
> Here is a small code that I wrote that downloads images from a webpage url
> specified (you can limit to how many downloads you want). However, I am
> looking at adding functionality and searching external links from this page
> and downloading the same number of images from that page as well.(And
> limiting the depth it can go to)
> 
> Any ideas?  (I am using Python 3.4 & I am a beginner)
> 
> import urllib.request
> import re
> url="http://www.abc.com"
> 
> pagehtml = urllib.request.urlopen(url)
> myfile = pagehtml.read()
> matches=re.findall(r'http://\S+jpg|jpeg',str(myfile))
> 
> 
> for urltodownload in matches[0:50]:
>   imagename=urltodownload[-12:]
>   urllib.request.urlretrieve(urltodownload,imagename)
> 
> print('Done!')
>  
> Thanks,
> Shiva
> 
> 

I'm going to make the wild assumption that you can safely do both
 parses using regex, and that finding the jpegs works well enough
 with your present one.

First thing is to make most of your present code a function
 fetch(), starting with pagehtml and ending before the print. The
 function should take two arguments,  url and depthlimit.

Now, at the end of the function,  add something like

    If depthlimit > 0:
        matches = ... some regex that finds links
        for link in matches [:40]:
             fetch (link, depthlimit - 1)

Naturally, the rest of the top level code needs to be moved after
 the function definition,  and is called by doing something
 like:

fetch (url, 10)   to have a depth limit of 10.
            
-- 
DaveA

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread


Thread

Re:downloading from links within a webpage Dave Angel <davea@davea.name> - 2014-10-14 13:47 -0400

csiph-web