Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #39421

Re: Urllib's urlopen and urlretrieve

Path csiph.com!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed1.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path <hermanmu@gmail.com>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.017
X-Spam-Evidence '*H*': 0.97; '*S*': 0.00; '21,': 0.07; 'python': 0.09; 'fetch': 0.09; 'information?': 0.09; 'whole.': 0.09; 'cc:addr:python-list': 0.10; 'archive': 0.11; 'skip:f 30': 0.15; 'formatting,': 0.16; 'urllib': 0.16; 'webpage,': 0.16; 'wrote:': 0.17; 'library,': 0.17; 'thu,': 0.17; 'requests': 0.18; 'feb': 0.19; 'email addr:gmail.com&gt;': 0.20; 'trying': 0.21; 'bit': 0.21; 'import': 0.21; 'cc:2**0': 0.23; 'programming': 0.23; 'cc:no real name:2**0': 0.24; 'cc:addr:python.org': 0.25; 'header:In- Reply-To:1': 0.25; 'am,': 0.27; 'language.': 0.27; 'message- id:@mail.gmail.com': 0.27; 'this?': 0.28; 'really,': 0.29; 'url:mailman': 0.29; 'skip:& 10': 0.29; 'url:python': 0.32; 'quickly': 0.32; 'could': 0.32; 'url:listinfo': 0.32; 'certain': 0.33; 'extract': 0.33; 'hopefully': 0.33; 'programming,': 0.33; 'problem': 0.33; 'anyone': 0.33; 'code:': 0.33; 'know.': 0.33; 'languages': 0.33; 'another': 0.33; 'received:google.com': 0.34; 'programming.': 0.35; 'skip:* 20': 0.35; 'so,': 0.35; 'received:209.85.220': 0.35; 'received:209.85': 0.35; 'something': 0.35; 'url:org': 0.36; 'skip:t 40': 0.37; 'does': 0.37; 'why': 0.37; 'communicate': 0.37; 'received:209': 0.37; 'data': 0.37; 'subject:: ': 0.38; 'easier': 0.38; 'skip:o 20': 0.38; 'some': 0.38; 'things': 0.38; 'nothing': 0.38; 'advice': 0.39; 'where': 0.40; 'url:mail': 0.40; 'skip:u 10': 0.60; 'first': 0.61; 'kind': 0.61; 'more': 0.63; 'making': 0.64; 'webpage': 0.65; 'internet': 0.71; 'lack': 0.71; 'topic,': 0.78; '2013': 0.84; 'lastly,': 0.84; 'url:st': 0.84
DKIM-Signature v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=/tF67nMFkmvYGhmylCjGYsYTRHc4gAXzDEeUVCUJZpQ=; b=FFpc3vW7Ix6sNLvQHoGlpc8B/UNLcyV/13fNJ4egYBkj2OdDJ2OGlM0lDAlP1szi2D qLm5mwzWBzIbSD14yGVmcfLVBg8+Na1jfqPG5wTFX2Ikeud87Ple74qdOQtlVKVEcGlC PimS7eeCoz+Bz7r8sWLUbWRGxfexYW0BJcTa1q1TRGxspMoGqqDRQ/3Kpyd4cYqh4RcT EZcy0JiqxFJs3sxE5NAHPD1Coc7ega/aiX9347fYT0bg8JvGsEpwzqMFOGT6TltMqf0j WIV/b5yCn4P20hXnGtZASFgJtUtyknqbs/pJBUDV7KuOFzQPfUn8aq6eyiwnOgiltT7+ Yb4A==
X-Received by 10.220.116.5 with SMTP id k5mr30956808vcq.55.1361451587079; Thu, 21 Feb 2013 04:59:47 -0800 (PST)
MIME-Version 1.0
In-Reply-To <34998ea2-6b19-4a98-8ea0-389aca0192ca@googlegroups.com>
References <34998ea2-6b19-4a98-8ea0-389aca0192ca@googlegroups.com>
From Michael Herman <hermanmu@gmail.com>
Date Thu, 21 Feb 2013 04:59:26 -0800
Subject Re: Urllib's urlopen and urlretrieve
To qoresucks@gmail.com
Content-Type multipart/alternative; boundary=f46d043894633a0e5e04d63ba6e7
Cc python-list@python.org
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.15
Precedence list
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list/>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups comp.lang.python
Message-ID <mailman.2162.1361451589.2939.python-list@python.org> (permalink)
Lines 123
NNTP-Posting-Host 2001:888:2000:d::a6
X-Trace 1361451589 news.xs4all.nl 6911 [2001:888:2000:d::a6]:54672
X-Complaints-To abuse@xs4all.nl
Xref csiph.com comp.lang.python:39421

Show key headers only | View raw

[Multipart message — attachments visible in raw view] - view raw

Are you just trying to get the html? If so, you can use this code-

*import urllib*
*
*
*# fetch the and download a webpage, nameing it test.html*
*urllib.urlretrieve("http://www.web2py.com/", filename="test.html")*


I recommend using the requests library, as it's easier to use and more
powerful:

*import requests*

*# retrive the webpage
r = requests.get("http://www.web2py.com/")*

*# write the content to test_request.html
with open("test_requests.html", "wb") as code:
*

*code.write(r.content)*


If you want to get up to speed quickly on internet programming, I have a
course I am developing. It's on kickstarter - http://kck.st/VQj8hq. The
first section of the book dives into web fundamentals and internet
programming.


On Thu, Feb 21, 2013 at 4:12 AM, <qoresucks@gmail.com> wrote:

> I only just started Python and given that I know nothing about network
> programming or internet programming of any kind really, I thought it would
> be interesting to try write something that could create an archive of a
> website for myself. With this I started trying to use the urllib library,
> however I am having a problem understanding why certain things wont work
> with the urllib.urlretrieve and urllib.urlopen then reading.
>
> Why is it that when using urllib.urlopen then reading or
> urllib.urlretrieve, does it only give me parts of the sites, loosing the
> formatting, images, etc...? How can I get around this?
>
> Lastly, while its a bit off topic, I lack a good understanding of network
> programming as a whole. From making programs communicate or to simply
> extract data from URL's, I don't know where to even begin, which has lead
> me to learning python to better understand it hopefully then carry it over
> to other languages I know. Can anyone give me some advice on where to begin
> learning this information? Even if its in another language.
> --
> http://mail.python.org/mailman/listinfo/python-list
>

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Urllib's urlopen and urlretrieve qoresucks@gmail.com - 2013-02-21 04:12 -0800
  Re: Urllib's urlopen and urlretrieve Michael Herman <hermanmu@gmail.com> - 2013-02-21 04:59 -0800
    Re: Urllib's urlopen and urlretrieve qoresucks@gmail.com - 2013-02-21 21:09 -0800
      Re: Urllib's urlopen and urlretrieve Dave Angel <davea@davea.name> - 2013-02-22 12:05 -0500
      Re: Urllib's urlopen and urlretrieve MRAB <python@mrabarnett.plus.com> - 2013-02-22 17:18 +0000
    Re: Urllib's urlopen and urlretrieve qoresucks@gmail.com - 2013-02-21 21:09 -0800
  Re: Urllib's urlopen and urlretrieve Dave Angel <davea@davea.name> - 2013-02-21 10:56 -0500
  Re: Urllib's urlopen and urlretrieve rh <richard_hubbe11@lavabit.com> - 2013-02-21 09:47 -0800
  Re: Urllib's urlopen and urlretrieve rh <richard_hubbe11@lavabit.com> - 2013-02-21 09:55 -0800
  Re: Urllib's urlopen and urlretrieve Dave Angel <davea@davea.name> - 2013-02-21 13:04 -0500
  Re: Urllib's urlopen and urlretrieve Dave Angel <davea@davea.name> - 2013-02-21 13:53 -0500

csiph-web