Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.143 X-Spam-Level: * X-Spam-Evidence: '*H*': 0.71; '*S*': 0.00; 'python?': 0.07; 'url:code': 0.11; 'am,': 0.14; 'result.': 0.14; 'wrote:': 0.14; 'selenium': 0.16; 'subject:download': 0.16; 'tue,': 0.20; 'cc:2**0': 0.20; 'cheers,': 0.20; 'header:In-Reply-To:1': 0.22; 'cc:addr:python-list': 0.22; 'version': 0.25; 'chris': 0.27; 'message-id:@mail.gmail.com': 0.28; 'hi,': 0.29; 'cc:addr:python.org': 0.31; 'page.': 0.32; 'source': 0.32; "i've": 0.33; 'page': 0.33; 'updated': 0.33; 'using': 0.34; 'requires': 0.35; 'file': 0.35; 'open': 0.35; 'run': 0.37; 'apr': 0.38; 'subject:web': 0.38; 'received:google.com': 0.38; 'url:google': 0.38; 'how': 0.39; 'i.e.': 0.40; 'header:Received:5': 0.40; 'subject:that': 0.60; '2011': 0.62; 'full': 0.62; 'url:p': 0.63; 'browser': 0.64; 'human': 0.64; 'grab': 0.65; 'visit': 0.67; 'url:gov': 0.68; 'webpage': 0.68; 'received:74.125.83': 0.69; 'calls,': 0.84; 'subject:updated': 0.84; 'subject:are': 0.86 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rebertia.com; s=google; h=domainkey-signature:mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=o4lxgfsZ2rLynP3ttzgqspA2uLTG/XHoxPbkoXM0WRY=; b=LS2zKMFsa6eJSPOwNNE6iqqi2LUPZGZdNwx3pEbg67VaqtO7Im5rF+6olGp7Uy3lWB yw5fNKTSwgbfXliVZAdFe1IzFOhZcd2KlxbRMw0jSKA5prVsGtapRpHKxxdnnpu/QkX9 sr/qdF42NHRwUWLIkYlrnpbJxgANlzrzeu2BI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=rebertia.com; s=google; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; b=I8kTBySKh8iBH4a9m2fRIgZ1K2VcykFL/NfU/Z+ICG0OOu6PgnXe4D1CGdkDTxIquA xMhm+Nb4Sj35pLXdC1qEuwaNZ0EX1QVwLnoLLmKLh2uC7sfe0O2bO3O/iBkWgqULjXUU SydFPg/R4FhvXa25vtENdK9YajNZ/uIgDYyjM= MIME-Version: 1.0 Sender: chris@rebertia.com In-Reply-To: References: Date: Tue, 12 Apr 2011 11:30:39 -0700 X-Google-Sender-Auth: -uOPxB8crk9jnOnV4q6ehAK7SCo Subject: Re: download web pages that are updated by ajax From: Chris Rebert To: Jabba Laci Content-Type: text/plain; charset=UTF-8 Cc: Python mailing list X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 21 NNTP-Posting-Host: 82.94.164.166 X-Trace: 1302633041 news.xs4all.nl 81473 [::ffff:82.94.164.166]:55887 X-Complaints-To: abuse@xs4all.nl Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:3075 On Tue, Apr 12, 2011 at 7:47 AM, Jabba Laci wrote: > Hi, > > I want to download a web page that is updated by AJAX. The page > requires no human interaction, it is updated automatically: > http://www.ncbi.nlm.nih.gov/nuccore/CP002059.1 > > If I download it with wget, I get a file of size 97 KB. The source is > full of AJAX calls, i.e. the content of the page is not expanded. > If I open it in a browser and save it manually, the result is a file > of almost 5 MB whose content is expanded. > > (1) How to download such a page with Python? I need the post-AJAX > version of the page. I've heard you can drive a web browser using Selenium (http://code.google.com/p/selenium/ ), have it visit the webpage and run the JavaScript on it, and then grab the final result. Cheers, Chris