Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #30613

Out of memory error in SAX parsing with validation

Show all headers | View raw

Hi there,

does anyone here know something about the memory requirements for 
validating XML with SAX? I've encountered what I think is a memory leak 
with the Xerces version included in JDK 7 and 8.

I'm using a SAX parser (XMLReader) to parse a large XML file. The XML 
basically looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE eventlist SYSTEM "elem.dtd">
<elemlist>
    <elem>
       <subelem type="abc">xyz</subelem>
       <subelem type="def">rst</subelem>
       ...
    </elem>
    <elem>
       <subelem type="ghj">uvw</subelem>
       ...
    </elem>
    ...
</elemlist>

Using a non-validating parser, I can process a 7 GB file containing 25 
million small elements (each having ca. 3 - 5 subelements) with just 64 
MB of heap space. With XML validation against a DTD turned on, 1024 MB 
do not suffice. I have taken a cursory glance at the heap with 
JVisualVM, and see millions of QName instances being created and never 
being GC'ed. I suspect this to be at least a part of the problem.

Can anyone enlighten me as to why SAX would require so much memory for 
validation? Isn't it enough to know that each element is well-formed? 
Why in particular should there be  so many instances of
com.sun.org.apache.xerces.internal.xni.QName ? The Javadoc says:

<quote>
A structure that holds the components of an XML Namespaces qualified 
name. To be used correctly, the strings must be identical references for 
equal strings. Within the parser, these values are considered symbols 
and should always be retrieved from the SymbolTable.
</quote>

I don't understand that at all. I'm not using namespaces. Can it be that 
the parser provides unique qualified names, regardless of the value of 
"http://xml.org/sax/features/namespace-prefixes" (which is false by 
default, as far as I know)? How would I be able to find out? And even if 
it did, why couldn't it garbage collect the instances?

Here's how the parser is created. The second line makes all the difference:

XMLReader parser = XMLReaderFactory.createXMLReader();
parser.setFeature("http://xml.org/sax/features/validation", validating);
parser.setContentHandler(myDocHandler);
parser.setErrorHandler(myErrorHandler);
parser.setEntityResolver(myEntityResolver);


-- Sebastian

Back to comp.lang.java.programmer | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Out of memory error in SAX parsing with validation Sebastian <news@seyweiler.dyndns.org> - 2014-12-26 00:32 +0100
  Re: Out of memory error in SAX parsing with validation Arne Vajhøj <arne@vajhoej.dk> - 2014-12-25 18:45 -0500
    Re: Out of memory error in SAX parsing with validation Sebastian <news@seyweiler.dyndns.org> - 2014-12-26 11:45 +0100
      Re: Out of memory error in SAX parsing with validation Arne Vajhøj <arne@vajhoej.dk> - 2014-12-26 08:53 -0500
        Re: Out of memory error in SAX parsing with validation Sebastian <news@seyweiler.dyndns.org> - 2014-12-27 15:48 +0100

csiph-web