Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!news.albasani.net!.POSTED!not-for-mail
From: Sebastian <news@seyweiler.dyndns.org>
Newsgroups: comp.lang.java.programmer
Subject: Out of memory error in SAX parsing with validation
Date: Fri, 26 Dec 2014 00:32:48 +0100
Organization: albasani.net
Lines: 60
Message-ID: <m7i6qs$sch$1@news.albasani.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 7bit
X-Trace: news.albasani.net oGfQhQY8wkmVbXEPlrKDVHB0uW+g23sODSovgJ49nhgR/kzN2WFqaithravUy+P6RXrcpR+ZqaFoxR0d4nj+1g==
NNTP-Posting-Date: Thu, 25 Dec 2014 23:32:44 +0000 (UTC)
Injection-Info: news.albasani.net; logging-data="NPqB3vsVbcuOpsrLUzdPCU0QoVLuPCoDsWmatONlvmvzI8GP3l5GB3DXFAcKGkscVwNZccQ9eLaVqbRW9Gz5jPFTxR4YYc5pIbUCTmnYvzhd8u1yuMYCntBOapzR4O0g"; mail-complaints-to="abuse@albasani.net"
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0
Cancel-Lock: sha1:L/d8O5YmttSeG5uxsNLruD3+cUM=
Xref: csiph.com comp.lang.java.programmer:30613

Hi there,

does anyone here know something about the memory requirements for 
validating XML with SAX? I've encountered what I think is a memory leak 
with the Xerces version included in JDK 7 and 8.

I'm using a SAX parser (XMLReader) to parse a large XML file. The XML 
basically looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE eventlist SYSTEM "elem.dtd">
<elemlist>
    <elem>
       <subelem type="abc">xyz</subelem>
       <subelem type="def">rst</subelem>
       ...
    </elem>
    <elem>
       <subelem type="ghj">uvw</subelem>
       ...
    </elem>
    ...
</elemlist>

Using a non-validating parser, I can process a 7 GB file containing 25 
million small elements (each having ca. 3 - 5 subelements) with just 64 
MB of heap space. With XML validation against a DTD turned on, 1024 MB 
do not suffice. I have taken a cursory glance at the heap with 
JVisualVM, and see millions of QName instances being created and never 
being GC'ed. I suspect this to be at least a part of the problem.

Can anyone enlighten me as to why SAX would require so much memory for 
validation? Isn't it enough to know that each element is well-formed? 
Why in particular should there be  so many instances of
com.sun.org.apache.xerces.internal.xni.QName ? The Javadoc says:

<quote>
A structure that holds the components of an XML Namespaces qualified 
name. To be used correctly, the strings must be identical references for 
equal strings. Within the parser, these values are considered symbols 
and should always be retrieved from the SymbolTable.
</quote>

I don't understand that at all. I'm not using namespaces. Can it be that 
the parser provides unique qualified names, regardless of the value of 
"http://xml.org/sax/features/namespace-prefixes" (which is false by 
default, as far as I know)? How would I be able to find out? And even if 
it did, why couldn't it garbage collect the instances?

Here's how the parser is created. The second line makes all the difference:

XMLReader parser = XMLReaderFactory.createXMLReader();
parser.setFeature("http://xml.org/sax/features/validation", validating);
parser.setContentHandler(myDocHandler);
parser.setErrorHandler(myErrorHandler);
parser.setEntityResolver(myEntityResolver);


-- Sebastian