Path: csiph.com!usenet.pasdenom.info!goblin3!goblin2!goblin.stu.neva.ru!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Sun, 24 Feb 2013 19:30:00 -0500
From: Dave Angel <davea@davea.name>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130106 Thunderbird/17.0.2
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: using urllib on a more complex site
References: <e3e061a0-493b-4b8b-992b-a175dbecd4ac@googlegroups.com>
In-Reply-To: <e3e061a0-493b-4b8b-992b-a175dbecd4ac@googlegroups.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.2465.1361752225.2939.python-list@python.org>
Lines: 42
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:39830

On 02/24/2013 07:02 PM, Adam W. wrote:
> I'm trying to write a simple script to scrape http://www.vudu.com/movies/#tag/99centOfTheDay/99c%20Rental%20of%20the%20day
>
> in order to send myself an email every day of the 99c movie of the day.
>
> However, using a simple command like (in Python 3.0):
> urllib.request.urlopen('http://www.vudu.com/movies/#tag/99centOfTheDay/99c%20Rental%20of%20the%20day').read()
>
> I don't get the all the source I need, its just the navigation buttons.  Now I assume they are using some CSS/javascript witchcraft to load all the useful data later, so my question is how do I make urllib "wait" and grab that data as well?
>

The CSS and the jpegs, and many other aspects of a web "page" are loaded 
explicitly, by the browser, when parsing the tags of the page you 
downloaded.  There is no sooner or later.  The website won't send the 
other files until you request them.

For example, that site at the moment has one image (prob. jpeg) 
highlighted,

<img class="gwt-Image" src="http://images2.vudu.com/poster2/179186-m" 
alt="Sex and the City: The Movie (Theatrical)">

if you want to look at that jpeg, you need to download the file url 
specified by the src attribute of that img element.

Or perhaps you can just look at the 'alt' attribute, which is mainly 
there for browsers who don't happen to do graphics, for example, the 
ones for the blind.

Naturally, there may be dozens of images on the page, and there's no 
guarantee that the website author is trying to make it easy for you. 
Why not check if there's a defined api for extracting the information 
you want?  Check the site, or send a message to the webmaster.

No guarantee that tomorrow, the information won't be buried in some 
javascript fragment.  Again, if you want to see that, you might need to 
write a javascript interpreter.  it could use any algorithm at all to 
build webpage information, and the encoding could change day by day, or 
hour by hour.

-- 
DaveA