Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Thu, 12 Jan 2012 12:02:41 -0500
From: Dave Angel <d@davea.name>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Thunderbird/3.1.15
MIME-Version: 1.0
To: dreadpiratejeff@gmail.com
Subject: Re: Is there a way to merge two XML files via Python?
References: <CAFB6qZvnBSLzZ7gRSxSnSm6Dj8cTr+bpMQOuNTO3CnSdMDLXfQ@mail.gmail.com> <jen2ba$2kq$1@dough.gmane.org>
In-Reply-To: <jen2ba$2kq$1@dough.gmane.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: python-list@python.org, Stefan Behnel <stefan_ml@behnel.de>
Precedence: list
Reply-To: d@davea.name
Newsgroups: comp.lang.python
Message-ID: <mailman.4687.1326387773.27778.python-list@python.org>
Lines: 64
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:18879

On 01/12/2012 11:39 AM, Stefan Behnel wrote:
> J, 12.01.2012 17:04:
>> This is more a theory exercise and something I'm trying to figure out,
>> and this is NOT a homework assignment...
>>
>> I'm trying to make a tool I use at work more efficient :)
>>
>> So this is at test tool that generates an XML file as it's output that
>> is eventually used by a web service to display test results and system
>> information.
>>
>> The problem is that the testing is broken down into to different runs:
>> Functional and Automated where the Functional tests are all manual,
>> then the automated tests are run separately, usually overnight.
>>
>> Each of those test runs generates essentially an identical XML file.
>> What I want to learn is a way to merge them.
> Ok - how large are these files? (i.e., do they easily fit into memory?)
>
>
>> In abstract terms, the idea is essentially to diff the two files
>> creating a patch and then use that patch to merge the two files into a
>> single XML file.
> I wouldn't go through patch. If they fit into memory, just load both, merge
> one into the other eliminating duplicates, and save that.
>
> Or rather, load just one and process the other one incrementally using
> ElementTree's iterparse().
>
>
>> SO what I was hoping I could get pointers on from those of you who are
>> experienced in using Python with XML is what python libraries or means
>> are there for working with XML files specifically, and how easy or
>> difficult would this be?
> Depends on how easy it is to recognise duplicates in your specific data
> format. Once you've managed to do that, the rest is trivial.
>
>
>> I'm also doing research on my own in my spare time on this, but I also
>> wanted to ask here to get the opinion of developers who are more
>> experienced in working with XML than I am.
> I recommend looking at the stdlib xml.etree.ElementTree module or the
> external lxml package (which contains the ElementTree compatible lxml.etree
> module). The latter will (likely) make things easier due to full XPath
> support and some other goodies, but ElementTree is also quite quick and
> easy to use by itself.
>
> Stefan
>
Question for jeff:

Have you tried doing it by hand?  Do you know when a duplicate should be 
ignored, when it should be replicated, when it should be represented by 
incrementing a count?

xml is very flexible, but the final reader of your file may not be so 
flexible.  (e.g. if it has to match a wsdl)  If two runs differ only by 
some timing field, then you might need to sum those times, and produce 
an average in the final run.  Or a max value, or both.

-- 

DaveA