Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.001 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'python,': 0.02; 'subject:Python': 0.06; '21,': 0.07; 'subject:would': 0.07; 'welcome.': 0.07; 'apis': 0.09; 'pages.': 0.09; 'parsing': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'subject:using': 0.09; 'api': 0.11; 'jan': 0.12; 'journal,': 0.16; 'layout,': 0.16; 'received:80.91.229.3': 0.16; 'received:plane.gmane.org': 0.16; 'reedy': 0.16; 'retrieving': 0.16; 'subject: \n ': 0.16; 'do,': 0.16; 'wrote:': 0.18; 'wed,': 0.18; 'bit': 0.19; 'subject:page': 0.19; 'seems': 0.21; 'aug': 0.22; 'header:User-Agent:1': 0.23; 'mind.': 0.24; 'of.': 0.24; 'simpler': 0.24; 'task': 0.26; 'header:X-Complaints-To:1': 0.27; 'header:In-Reply-To:1': 0.27; 'wondering': 0.29; 'code': 0.31; 'accomplished': 0.31; 'probably': 0.32; 'guess': 0.33; 'comment': 0.34; 'subject:from': 0.34; 'really': 0.36; 'subject:data': 0.36; 'thanks': 0.36; 'searching': 0.37; 'stable': 0.38; 'to:addr :python-list': 0.38; 'pm,': 0.38; 'to:addr:python.org': 0.39; 'received:org': 0.40; 'how': 0.40; 'break': 0.61; 'received:173': 0.61; "you're": 0.61; 'you.': 0.62; 'more': 0.64; 'wall': 0.65; 'beat': 0.68; 'websites': 0.72; ':).': 0.84; 'answered,': 0.84; 'popped': 0.84; 'received:fios.verizon.net': 0.84; 'safer': 0.84; 'joel': 0.91; '2013': 0.98 X-Injected-Via-Gmane: http://gmane.org/ To: python-list@python.org From: Terry Reedy Subject: Re: I wonder if I would be able to collect data from such page using Python Date: Wed, 21 Aug 2013 15:18:19 -0400 References: <02caf0a8-1506-4746-9136-3452cbdea14b@googlegroups.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Gmane-NNTP-Posting-Host: pool-173-75-251-66.phlapa.fios.verizon.net User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 In-Reply-To: X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 21 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1377112715 news.xs4all.nl 15916 [2001:888:2000:d::a6]:55378 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:52786 On 8/21/2013 1:52 PM, Joel Goldstick wrote: > On Wed, Aug 21, 2013 at 1:41 PM, Comment Holder wrote: >> Many thanks for your help - I think I shall start with this way and see how it goes. My concerns were if the task can be accomplished with Python, and from your posts, I guess it can - so I shall give it a try :). CM: You still seem a bit doubtful. If you are wondering why no one else has answered, it is because Joel has given you a really good answer that cannot be beat without writing your code for you. > You're welcome. One thought popped into my mind. Since the site > seems to be from the Wall Street Journal, you may want to look into > whether they have an api for searching and retrieving articles. If > they do, this would be simpler and probably safer than parsing web > pages. From time to time, websites change their layout, which would > probably break your program. However APIs are more stable Including this suggestion, which I did not think of. -- Terry Jan Reedy