Path: csiph.com!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed1.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.017 X-Spam-Evidence: '*H*': 0.97; '*S*': 0.00; '21,': 0.07; 'python': 0.09; 'fetch': 0.09; 'information?': 0.09; 'whole.': 0.09; 'cc:addr:python-list': 0.10; 'archive': 0.11; 'skip:f 30': 0.15; 'formatting,': 0.16; 'urllib': 0.16; 'webpage,': 0.16; 'wrote:': 0.17; 'library,': 0.17; 'thu,': 0.17; 'requests': 0.18; 'feb': 0.19; 'email addr:gmail.com>': 0.20; 'trying': 0.21; 'bit': 0.21; 'import': 0.21; 'cc:2**0': 0.23; 'programming': 0.23; 'cc:no real name:2**0': 0.24; 'cc:addr:python.org': 0.25; 'header:In- Reply-To:1': 0.25; 'am,': 0.27; 'language.': 0.27; 'message- id:@mail.gmail.com': 0.27; 'this?': 0.28; 'really,': 0.29; 'url:mailman': 0.29; 'skip:& 10': 0.29; 'url:python': 0.32; 'quickly': 0.32; 'could': 0.32; 'url:listinfo': 0.32; 'certain': 0.33; 'extract': 0.33; 'hopefully': 0.33; 'programming,': 0.33; 'problem': 0.33; 'anyone': 0.33; 'code:': 0.33; 'know.': 0.33; 'languages': 0.33; 'another': 0.33; 'received:google.com': 0.34; 'programming.': 0.35; 'skip:* 20': 0.35; 'so,': 0.35; 'received:209.85.220': 0.35; 'received:209.85': 0.35; 'something': 0.35; 'url:org': 0.36; 'skip:t 40': 0.37; 'does': 0.37; 'why': 0.37; 'communicate': 0.37; 'received:209': 0.37; 'data': 0.37; 'subject:: ': 0.38; 'easier': 0.38; 'skip:o 20': 0.38; 'some': 0.38; 'things': 0.38; 'nothing': 0.38; 'advice': 0.39; 'where': 0.40; 'url:mail': 0.40; 'skip:u 10': 0.60; 'first': 0.61; 'kind': 0.61; 'more': 0.63; 'making': 0.64; 'webpage': 0.65; 'internet': 0.71; 'lack': 0.71; 'topic,': 0.78; '2013': 0.84; 'lastly,': 0.84; 'url:st': 0.84 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=/tF67nMFkmvYGhmylCjGYsYTRHc4gAXzDEeUVCUJZpQ=; b=FFpc3vW7Ix6sNLvQHoGlpc8B/UNLcyV/13fNJ4egYBkj2OdDJ2OGlM0lDAlP1szi2D qLm5mwzWBzIbSD14yGVmcfLVBg8+Na1jfqPG5wTFX2Ikeud87Ple74qdOQtlVKVEcGlC PimS7eeCoz+Bz7r8sWLUbWRGxfexYW0BJcTa1q1TRGxspMoGqqDRQ/3Kpyd4cYqh4RcT EZcy0JiqxFJs3sxE5NAHPD1Coc7ega/aiX9347fYT0bg8JvGsEpwzqMFOGT6TltMqf0j WIV/b5yCn4P20hXnGtZASFgJtUtyknqbs/pJBUDV7KuOFzQPfUn8aq6eyiwnOgiltT7+ Yb4A== X-Received: by 10.220.116.5 with SMTP id k5mr30956808vcq.55.1361451587079; Thu, 21 Feb 2013 04:59:47 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <34998ea2-6b19-4a98-8ea0-389aca0192ca@googlegroups.com> References: <34998ea2-6b19-4a98-8ea0-389aca0192ca@googlegroups.com> From: Michael Herman Date: Thu, 21 Feb 2013 04:59:26 -0800 Subject: Re: Urllib's urlopen and urlretrieve To: qoresucks@gmail.com Content-Type: multipart/alternative; boundary=f46d043894633a0e5e04d63ba6e7 Cc: python-list@python.org X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 123 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1361451589 news.xs4all.nl 6911 [2001:888:2000:d::a6]:54672 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:39421 --f46d043894633a0e5e04d63ba6e7 Content-Type: text/plain; charset=ISO-8859-1 Are you just trying to get the html? If so, you can use this code- *import urllib* * * *# fetch the and download a webpage, nameing it test.html* *urllib.urlretrieve("http://www.web2py.com/", filename="test.html")* I recommend using the requests library, as it's easier to use and more powerful: *import requests* *# retrive the webpage r = requests.get("http://www.web2py.com/")* *# write the content to test_request.html with open("test_requests.html", "wb") as code: * *code.write(r.content)* If you want to get up to speed quickly on internet programming, I have a course I am developing. It's on kickstarter - http://kck.st/VQj8hq. The first section of the book dives into web fundamentals and internet programming. On Thu, Feb 21, 2013 at 4:12 AM, wrote: > I only just started Python and given that I know nothing about network > programming or internet programming of any kind really, I thought it would > be interesting to try write something that could create an archive of a > website for myself. With this I started trying to use the urllib library, > however I am having a problem understanding why certain things wont work > with the urllib.urlretrieve and urllib.urlopen then reading. > > Why is it that when using urllib.urlopen then reading or > urllib.urlretrieve, does it only give me parts of the sites, loosing the > formatting, images, etc...? How can I get around this? > > Lastly, while its a bit off topic, I lack a good understanding of network > programming as a whole. From making programs communicate or to simply > extract data from URL's, I don't know where to even begin, which has lead > me to learning python to better understand it hopefully then carry it over > to other languages I know. Can anyone give me some advice on where to begin > learning this information? Even if its in another language. > -- > http://mail.python.org/mailman/listinfo/python-list > --f46d043894633a0e5e04d63ba6e7 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Are you just trying to get the html? If so, you can use this code-

=
import urllib

# f= etch the and download a webpage, nameing it test.html
urll= ib.urlretrieve("http://www.web2py.c= om/", filename=3D"test.html")


I recommend using the requests library, = as it's easier to use and more powerful:

import requests

# retrive the webpage
r =3D requests.get("http://www.web2py.com/")

# write the content to test_request.html
with open(&q= uot;test_requests.html", "wb") as code: =A0

code.write(r.content)


If you want to get up to speed quickly on intern= et programming, I have a course I am developing. It's on kickstarter -= =A0http://kck.st/VQj8hq. The first sec= tion of the book dives into web fundamentals and internet programming.=A0


On Thu, Feb 21, 2013 at = 4:12 AM, <qoresucks@gmail.com> wrote:
I only just started Python and given that I know nothing about network prog= ramming or internet programming of any kind really, I thought it would be i= nteresting to try write something that could create an archive of a website= for myself. With this I started trying to use the urllib library, however = I am having a problem understanding why certain things wont work with the u= rllib.urlretrieve and urllib.urlopen then reading.

Why is it that when using urllib.urlopen then reading or urllib.urlretrieve= , does it only give me parts of the sites, loosing the formatting, images, = etc...? How can I get around this?

Lastly, while its a bit off topic, I lack a good understanding of network p= rogramming as a whole. From making programs communicate or to simply extrac= t data from URL's, I don't know where to even begin, which has lead= me to learning python to better understand it hopefully then carry it over= to other languages I know. Can anyone give me some advice on where to begi= n learning this information? Even if its in another language.
--
http://mail.python.org/mailman/listinfo/python-list

--f46d043894633a0e5e04d63ba6e7--