XML/XHTML/HTML differences, bugs... and howto

Path	csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.mixmin.net!newsreader4.netcologne.de!news.netcologne.de!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!194.109.133.86.MISMATCH!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path	<andrew3@r3dsolutions.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.003
X-Spam-Evidence	'H': 0.99; 'S': 0.00; 'method.': 0.05; 'modify': 0.05; 'problem:': 0.07; 'python': 0.09; 'happens.': 0.09; 'parsed': 0.09; 'parsers': 0.09; 'portions': 0.09; 'subject:howto': 0.09; 'subset': 0.09; 'underlying': 0.09; 'xml.': 0.09; '"this': 0.13; 'library': 0.15; 'file,': 0.15; 'disk.': 0.16; 'dropping': 0.16; 'iteratively': 0.16; 'optionally': 0.16; 'python;': 0.16; 'subject:XML': 0.16; 'text?': 0.16; 'alternate': 0.17; 'issue,': 0.17; 'memory': 0.18; 'file.': 0.20; 'trying': 0.21; 'supposed': 0.21; 'converted': 0.22; 'dropped': 0.22; 'file:': 0.22; 'parse': 0.22; 'example': 0.23; "i've": 0.23; 'header:User-Agent:1': 0.26; 'thanks!': 0.26; 'scanned': 0.27; 'library.': 0.27; 'start,': 0.27; 'tree': 0.27; 'subject:/': 0.28; 'bold': 0.29; 'restricted': 0.29; 'convert': 0.29; 'handled': 0.29; 'source': 0.29; "i'm": 0.29; 'error': 0.30; 'figure': 0.30; 'gets': 0.32; 'file': 0.32; 'impression': 0.33; 'like:': 0.33; 'to:addr:python-list': 0.33; 'another': 0.33; 'text': 0.34; '(1)': 0.34; 'or,': 0.34; 'needed': 0.35; 'text.': 0.35; 'so,': 0.35; 'open': 0.35; 'continue': 0.35; 'there': 0.35; 'but': 0.36; 'problems': 0.36; 'xml': 0.37; 'files': 0.38; 'some': 0.38; 'to:addr:python.org': 0.39; 'step': 0.39; 'where': 0.40; 'end': 0.40; 'think': 0.40; 'subject:, ': 0.61; 'telling': 0.61; 'first': 0.61; 'free': 0.61; 'received:phx3.secureserver.net': 0.62; 'received:prod.phx3.secureserver.net': 0.62; 'close': 0.63; 'header:Reply-To:1': 0.68; 'reply-to:no real name:2**0': 0.72; 'day': 0.73; 'grow': 0.74; 'tags,': 0.81; '(2),': 0.84; 'me;': 0.84; 'subject:XHTML': 0.84; 'tree,': 0.84; 'do:': 0.91; 'drops': 0.91; 'received:173.201': 0.91
Date	Wed, 23 Jan 2013 15:22:23 +0000
From	Andrew Robinson <andrew3@r3dsolutions.com>
User-Agent	Mozilla/5.0 (X11; Linux i686; rv:15.0) Gecko/20120909 Thunderbird/15.0.1
MIME-Version	1.0
To	python-list@python.org
Subject	XML/XHTML/HTML differences, bugs... and howto
Content-Type	text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding	7bit
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.15
Precedence	list
Reply-To	andrew3@r3dsolutions.com
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.930.1358983473.2939.python-list@python.org> (permalink)
Lines	69
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1358983473 news.xs4all.nl 6985 [2001:888:2000:d::a6]:42983
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:37517

Show key headers only | View raw

Good day :),

I've been exploring XML parsers in python; particularly: 
xml.etree.cElementTree; and I'm trying to figure out how to do it 
incrementally, for very large XML files -- although I don't think the 
problems are restricted to incremental parsing.

First problem:
I've come across an issue where etree silently drops text without 
telling me; and separate.

I am under the impression that XHTML is a subset of XML (eg:defined 
tags), and that once an HTML file is converted to XHTML, the body of the 
document can be handled entirely as XML.

If I convert a (partial/contrived) html file like:

<html>
     <div>
         <p> This is example <b>bold</b> text.
     </div>
</html>

to XHTML, I might do --right or wrong-- (1):

<html>
     <div>
         <p /> This is example <b>bold</b> text.
     </div>
</html>

or, alternate difference: (2): "<p> This is example <b>bold</b> text. </p>"

But, when I parse with etree,  in example (1) both "This is an example" 
and "text." are dropped;
The missing text is part of the start, or end event tags, in the 
incrementally parsed method.

Likewise: In example (2), only "text" gets dropped.

So, etree is silently dropping all text following a close tag, but 
before another open tag happens.

Q:
Isn't XML supposed to error out when invalid xml is parsed?
Is there a way in etree to recover/access the dropped text?
If not -- is the a python library issue, or the underlying expat.so, 
etc. library.

Secondly;
I have an XML file which will grow larger than memory on a target 
machine, so here's what I want to do:

Given a source XML file, and a destination file:
1) iteratively scan part of the source tree.
2) Optionally Modify some of scanned tree.
3) Write partial scan/tree out to the destination file.
4) Free memory of no-longer needed (partial) source XML.
5) continue scanning a new section of the source file... eg: goto step 1 
until source file is exhausted.

But, I don't see a way to write portions of an XML tree, or iteratively 
write a tree to disk.
How can this be done?

:)  Thanks!

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

XML/XHTML/HTML differences, bugs... and howto Andrew Robinson <andrew3@r3dsolutions.com> - 2013-01-23 15:22 +0000

csiph-web