Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder4.news.weretis.net!news.mixmin.net!hq-usenetpeers.eweka.nl!81.171.88.15.MISMATCH!eweka.nl!lightspeed.eweka.nl!194.134.4.91.MISMATCH!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.008 X-Spam-Evidence: '*H*': 0.98; '*S*': 0.00; 'wiki': 0.03; 'source.': 0.07; 'urllib2': 0.07; 'parsing': 0.09; 'received:209.85.219': 0.09; 'rows': 0.09; 'cc:addr:python-list': 0.11; 'changes': 0.15; '"|"': 0.16; 'separated': 0.16; 'subject:format': 0.16; 'thanks,': 0.17; 'wrote:': 0.18; 'wed,': 0.18; 'command': 0.22; '>>>': 0.22; 'import': 0.22; 'email addr:gmail.com>': 0.22; 'cc:addr:python.org': 0.22; '>>>': 0.24; 'parse': 0.24; 'cc:2**0': 0.24; 'source': 0.25; 'script': 0.25; '>': 0.26; 'task': 0.26; 'skip:" 20': 0.27; 'header:In-Reply-To:1': 0.27; 'idea': 0.28; 'wonder': 0.29; 'message-id:@mail.gmail.com': 0.30; 'url:wiki': 0.31; 'extract': 0.31; "skip:' 40": 0.31; 'url:wikipedia': 0.31; 'way?': 0.31; 'figure': 0.32; 'skip:- 30': 0.32; 'skip:& 30': 0.33; 'table': 0.34; 'could': 0.34; 'info': 0.35; 'received:209.85': 0.35; 'skip:s 30': 0.35; 'skip:u 20': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'there': 0.35; 'format.': 0.36; 'html,': 0.36; 'ubuntu': 0.36; 'shows': 0.36; 'hi,': 0.36; 'url:org': 0.36; 'should': 0.36; 'too': 0.37; 'project': 0.37; 'received:209': 0.37; 'starting': 0.37; 'skip:& 10': 0.38; 'skip:[ 10': 0.38; 'pm,': 0.38; 'called': 0.40; 'release': 0.40; 'url:index': 0.63; 'july': 0.63; 'skip:6 10': 0.63; 'great': 0.65; 'to:addr:gmail.com': 0.65; 'bottom': 0.67; 'webpage': 0.68; 'url:php': 0.85; '2013': 0.98 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=P2140CAbKFSiUIbAdeUmj5MmVEpdxxAlQxsG6hkF0xU=; b=RPQPPCgcNq/Th7N95Eejc22S2qPQJ7MqrWAQZMOaB1EQyIxELbjSOL/CWiekK1EFOs TajbiwhOkNWiS78l4uqtNki0cUFi05zNskCPgrJa0MQ65ZQIwzSYggVFmVKmxi+oAS2S KXk5HuWINahXq7J8bOJiGBMCQh0/XCHHUkcT69j4es7nQWJJ3iL7a4Aa+lTGb99v6toJ fBDmU4MyA2EuCV0oXij9iLPztJqgvRmK8VSTL6KgfjEyZJLocHC9sA6btPa+drsGxCX9 mp172zZlPpUi53ToXR4Ebj0HNyjeeqNcmMFKKjbA64IDGJZ78SLSRdGJw770hgge51nL jwjw== X-Received: by 10.60.65.233 with SMTP id a9mr1166330oet.67.1365617738441; Wed, 10 Apr 2013 11:15:38 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: References: From: Jabba Laci Date: Wed, 10 Apr 2013 20:15:17 +0200 Subject: Re: extract HTML table in a structured format To: Arnaud Delobelle Content-Type: multipart/alternative; boundary=001a11c20b823339fe04da05a849 Cc: Python mailing list X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 206 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1365617746 news.xs4all.nl 2581 [2001:888:2000:d::a6]:50580 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:43288 --001a11c20b823339fe04da05a849 Content-Type: text/plain; charset=ISO-8859-1 Hi, Thanks, it's a great idea to parse the wiki source. Since then I have found a command for this particular task called "ubuntu-support-status" that shows what I need. But the idea to extract an HTML table from a webpage in a structured format could be an interesting (and useful) project idea. Best, Laszlo On Wed, Apr 10, 2013 at 8:11 PM, Arnaud Delobelle wrote: > On 10 April 2013 09:44, Jabba Laci wrote: > > Hi, > > > > I wonder if there is a nice way to extract a whole HTML table and have > the > > result in a nice structured format. What I want is to have the lifetime > > table at the bottom of this page: > > http://en.wikipedia.org/wiki/List_of_Ubuntu_releases (then figure out > with a > > script until when my Ubuntu release is supported). > > > > I could do it with BeautifulSoup or lxml but is there a better way? There > > should be :) > > Instead of parsing HTML, you could just parse the source of the page > (available via action=raw): > > ------------------------------ > import urllib2 > > url = ( > 'http://en.wikipedia.org/w/index.php' > '?title=List_of_Ubuntu_releases&action=raw' > ) > > source = urllib2.urlopen(url).read() > > # Table rows are separated with the line "|-" > # Then there is a line starting with "|" > potential_rows = source.split("\n|-\n|") > > rows = [] > > for row in potential_rows: > # Rows in the table start with a link (' [[ ... ]]') > if row.startswith(" [["): > row = [item.strip() for item in row.split("\n|")] > rows.append(row) > ------------------------------ > > >>> import pprint > >>> pprint.pprint(rows) > [['[[Warty Warthog|4.10]]', > 'Warty Warthog', > '20 October 2004', > 'colspan="2" {{Version |o |30 April 2006}}', > '2.6.8'], > ['[[Hoary Hedgehog|5.04]]', > 'Hoary Hedgehog', > '8 April 2005', > 'colspan="2" {{Version |o |31 October 2006}}', > '2.6.10'], > ['[[Breezy Badger|5.10]]', > 'Breezy Badger', > '13 October 2005', > 'colspan="2" {{Version |o |13 April 2007}}', > '2.6.12'], > ['[[Ubuntu 6.06|6.06 LTS]]', > 'Dapper Drake', > '1 June 2006', > '{{Version |o | 14 July 2009}}', > '{{Version |o | 1 June 2011}}', > '2.6.15'], > ['[[Ubuntu 6.10|6.10]]', > 'Edgy Eft', > '26 October 2006', > 'colspan="2" {{Version |o | 25 April 2008}}', > '2.6.17'], > [...] > ] > >>> > > That should give you the info you need (until the wiki page changes too > much!) > > -- > Arnaud > --001a11c20b823339fe04da05a849 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi,

Thanks, it's a great idea to parse the wiki= source. Since then I have found a command for this particular task called = "ubuntu-support-status" that shows what I need.

But the id= ea to extract an HTML table from a webpage in a structured format could be = an interesting (and useful) project idea.

Best,

Laszlo


On Wed, Apr 10, 2013 at 8:11 PM, Arnaud Delobelle <arn= odel@gmail.com> wrote:
On 1= 0 April 2013 09:44, Jabba Laci <= jabba.laci@gmail.com> wrote:
> Hi,
>
> I wonder if there is a nice way to extract a whole HTML table and have= the
> result in a nice structured format. What I want is to have the lifetim= e
> table at the bottom of this page:
> http://en.wikipedia.org/wiki/List_of_Ubuntu_releases (then= figure out with a
> script until when my Ubuntu release is supported).
>
> I could do it with BeautifulSoup or lxml but is there a better way? Th= ere
> should be :)

Instead of parsing HTML, you could just parse the source of the= page
(available via action=3Draw):

------------------------------
import urllib2

url =3D (
=A0 =A0 =A0 =A0 'http://en.wikipedia.org/w/index.php'
=A0 =A0 =A0 =A0 '?title=3DList_of_Ubuntu_releases&action=3Draw'=
)

source =3D urllib2.urlopen(url).read()

# Table rows are separated with the line "|-"
# Then there is a line starting with "|"
potential_rows =3D source.split("\n|-\n|")

rows =3D []

for row in potential_rows:
=A0 =A0 =A0 =A0 # Rows in the table start with a link (' [[ ... ]]'= )
=A0 =A0 =A0 =A0 if row.startswith(" [["):
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 row =3D [item.strip() for item in row.split= ("\n|")]
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 rows.append(row)
------------------------------

>>> import pprint
>>> pprint.pprint(rows)
[['[[Warty Warthog|4.10]]',
=A0 'Warty Warthog',
=A0 '20 October 2004',
=A0 'colspan=3D"2" {{Version |o |30 April 2006}}',
=A0 '2.6.8'],
=A0['[[Hoary Hedgehog|5.04]]',
=A0 'Hoary Hedgehog',
=A0 '8 April 2005',
=A0 'colspan=3D"2" {{Version |o |31 October 2006}}',
=A0 '2.6.10'],
=A0['[[Breezy Badger|5.10]]',
=A0 'Breezy Badger',
=A0 '13 October 2005',
=A0 'colspan=3D"2" {{Version |o |13 April 2007}}',
=A0 '2.6.12'],
=A0['[[Ubuntu 6.06|6.06 LTS]]',
=A0 'Dapper Drake',
=A0 '1 June 2006',
=A0 '{{Version |o | 14 July 2009}}',
=A0 '{{Version |o | 1 June 2011}}',
=A0 '2.6.15'],
=A0['[[Ubuntu 6.10|6.10]]',
=A0 'Edgy Eft',
=A0 '26 October 2006',
=A0 'colspan=3D"2" {{Version |o | 25 April 2008}}',
=A0 '2.6.17'],
=A0 [...]
]
>>>

That should give you the info you need (until the wiki page changes too muc= h!)

--
Arnaud

--001a11c20b823339fe04da05a849--