Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.python > #79110
| Path | csiph.com!usenet.pasdenom.info!news.franciliens.net!fdn.fr!usenet-fr.net!nerim.net!novso.com!newsfeed.xs4all.nl!newsfeed3a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail |
|---|---|
| Return-Path | <python-python-list@m.gmane.org> |
| X-Original-To | python-list@python.org |
| Delivered-To | python-list@mail.python.org |
| X-Spam-Status | OK 0.003 |
| X-Spam-Evidence | '*H*': 0.99; '*S*': 0.00; 'matches': 0.07; 'arguments,': 0.09; 'definition,': 0.09; 'function,': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'to)': 0.09; 'python': 0.11; 'wrote': 0.14; 'fetch': 0.16; 'finds': 0.16; 'limiting': 0.16; 'received:80.91.229.3': 0.16; 'received:97': 0.16; 'received:plane.gmane.org': 0.16; 'regex,': 0.16; 'skip:u 50': 0.16; 'url=""': 0.16; '(you': 0.16; 'thanks,': 0.17; 'import': 0.22; 'header:X-Complaints-To:1': 0.27; 'function': 0.29; 'rest': 0.29; 'external': 0.29; 'moved': 0.30; 'specified': 0.30; "i'm": 0.30; 'code': 0.31; 'depth': 0.31; 'safely': 0.31; 'skip:m 30': 0.32; 'subject:from': 0.34; 'skip:u 20': 0.35; 'something': 0.35; 'add': 0.35; 'doing': 0.36; 'hi,': 0.36; 'should': 0.36; 'searching': 0.37; 'two': 0.37; 'level': 0.37; 'starting': 0.37; 'to:addr:python-list': 0.38; 'to:addr:python.org': 0.39; 'enough': 0.39; 'received:org': 0.40; 'called': 0.40; 'how': 0.40; 'skip:u 10': 0.60; 'most': 0.60; 'first': 0.61; 'here': 0.66; 'webpage': 0.68; 'limit': 0.70; 'ending': 0.78; '3.4': 0.84; 'subject::': 0.85 |
| X-Injected-Via-Gmane | http://gmane.org/ |
| To | python-list@python.org |
| From | Dave Angel <davea@davea.name> |
| Subject | Re:downloading from links within a webpage |
| Date | Tue, 14 Oct 2014 13:47:20 -0400 (EDT) |
| Organization | news.gmane.org |
| References | <loom.20141014T163517-784@post.gmane.org> |
| Mime-Version | 1.0 |
| Content-Type | text/plain; charset=UTF-8 |
| Content-Transfer-Encoding | 7bit |
| X-Gmane-NNTP-Posting-Host | 97.73.240.9 |
| X-Newsreader | PiaoHong.Usenet.Client.VIP:1.62 |
| X-Archive | encrypt |
| X-BeenThere | python-list@python.org |
| X-Mailman-Version | 2.1.15 |
| Precedence | list |
| List-Id | General discussion list for the Python programming language <python-list.python.org> |
| List-Unsubscribe | <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe> |
| List-Archive | <http://mail.python.org/pipermail/python-list/> |
| List-Post | <mailto:python-list@python.org> |
| List-Help | <mailto:python-list-request@python.org?subject=help> |
| List-Subscribe | <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe> |
| Newsgroups | comp.lang.python |
| Message-ID | <mailman.14857.1413308664.18130.python-list@python.org> (permalink) |
| Lines | 55 |
| NNTP-Posting-Host | 2001:888:2000:d::a6 |
| X-Trace | 1413308664 news.xs4all.nl 2973 [2001:888:2000:d::a6]:44499 |
| X-Complaints-To | abuse@xs4all.nl |
| Xref | csiph.com comp.lang.python:79110 |
Show key headers only | View raw
Shiva <shivaji_tn@yahoo.com.dmarc.invalid> Wrote in message:
> Hi,
>
> Here is a small code that I wrote that downloads images from a webpage url
> specified (you can limit to how many downloads you want). However, I am
> looking at adding functionality and searching external links from this page
> and downloading the same number of images from that page as well.(And
> limiting the depth it can go to)
>
> Any ideas? (I am using Python 3.4 & I am a beginner)
>
> import urllib.request
> import re
> url="http://www.abc.com"
>
> pagehtml = urllib.request.urlopen(url)
> myfile = pagehtml.read()
> matches=re.findall(r'http://\S+jpg|jpeg',str(myfile))
>
>
> for urltodownload in matches[0:50]:
> imagename=urltodownload[-12:]
> urllib.request.urlretrieve(urltodownload,imagename)
>
> print('Done!')
>
> Thanks,
> Shiva
>
>
I'm going to make the wild assumption that you can safely do both
parses using regex, and that finding the jpegs works well enough
with your present one.
First thing is to make most of your present code a function
fetch(), starting with pagehtml and ending before the print. The
function should take two arguments, url and depthlimit.
Now, at the end of the function, add something like
If depthlimit > 0:
matches = ... some regex that finds links
for link in matches [:40]:
fetch (link, depthlimit - 1)
Naturally, the rest of the top level code needs to be moved after
the function definition, and is called by doing something
like:
fetch (url, 10) to have a depth limit of 10.
--
DaveA
Back to comp.lang.python | Previous | Next | Find similar | Unroll thread
Re:downloading from links within a webpage Dave Angel <davea@davea.name> - 2014-10-14 13:47 -0400
csiph-web