Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #93158 > unrolled thread

enumerate XML tags (keys that will become headers) along with text (values) and write to CSV in one row (as opposed to "stacked" values with one header)

Back to article view | Back to comp.lang.python

Contents

  enumerate XML tags (keys that will become headers) along with text (values) and write to CSV in one row (as opposed to "stacked" values with one header) kbtyo <ahlusar.ahluwalia@gmail.com> - 2015-06-25 11:39 -0700
    Re: enumerate XML tags (keys that will become headers) along with text (values) and write to CSV in one row (as opposed to "stacked" values with one header) Denis McMahon <denismfmcmahon@gmail.com> - 2015-06-26 07:44 +0000
      Re: enumerate XML tags (keys that will become headers) along with text (values) and write to CSV in one row (as opposed to "stacked" values with one header) Stefan Behnel <stefan_ml@behnel.de> - 2015-06-28 09:46 +0200
        Re: enumerate XML tags (keys that will become headers) along with text (values) and write to CSV in one row (as opposed to "stacked" values with one header) Denis McMahon <denismfmcmahon@gmail.com> - 2015-06-28 21:00 +0000
          Re: enumerate XML tags (keys that will become headers) along with text (values) and write to CSV in one row (as opposed to "stacked" values with one header) Ned Batchelder <ned@nedbatchelder.com> - 2015-06-28 17:07 -0700
            Re: enumerate XML tags (keys that will become headers) along with text (values) and write to CSV in one row (as opposed to "stacked" values with one header) Denis McMahon <denismfmcmahon@gmail.com> - 2015-06-30 00:54 +0000
              Re: enumerate XML tags (keys that will become headers) along with text (values) and write to CSV in one row (as opposed to "stacked" values with one header) Robert Kern <robert.kern@gmail.com> - 2015-06-30 10:16 +0100
                Re: enumerate XML tags (keys that will become headers) along with text (values) and write to CSV in one row (as opposed to "stacked" values with one header) Marko Rauhamaa <marko@pacujo.net> - 2015-06-30 19:32 +0300
          Re: enumerate XML tags (keys that will become headers) along with text (values) and write to CSV in one row (as opposed to "stacked" values with one header) Robert Kern <robert.kern@gmail.com> - 2015-06-29 14:04 +0100
        Re: enumerate XML tags (keys that will become headers) along with text (values) and write to CSV in one row (as opposed to "stacked" values with one header) Sahlusar <sahluwalia@wynyardgroup.com> - 2015-06-29 07:52 -0700
          Re: enumerate XML tags (keys that will become headers) along with text (values) and write to CSV in one row (as opposed to "stacked" values with one header) Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2015-06-29 21:26 -0400
          Re: enumerate XML tags (keys that will become headers) along with text (values) and write to CSV in one row (as opposed to "stacked" values with one header) Chris Angelico <rosuav@gmail.com> - 2015-06-30 22:40 +1000

#93158 — enumerate XML tags (keys that will become headers) along with text (values) and write to CSV in one row (as opposed to "stacked" values with one header)

My question can be found here:


http://stackoverflow.com/questions/31058100/enumerate-column-headers-in-csv-that-belong-to-the-same-tag-key-in-python


Here is an additional sample sample of the XML that I am working with: 

<Response ID="123546 - 7831" RequestType="Moverview">        
        <MonthDayCount>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
            <Int32>0</Int32>
        </MonthDayCount>
            <FeeCount>
                    <Int32>0</Int32>
                    <Int32>0</Int32>
                    <Int32>0</Int32>
                    <Int32>0</Int32>
                    <Int32>0</Int32>
                    <Int32>0</Int32>
            </FeeCount>
            <PaymentBucketAmount>
                        <Double>0</Double>
                        <Double>0</Double>
                        <Double>0</Double>
                        <Double>0</Double>
                        <Double>0</Double>
                        <Double>0</Double>
                        <Double>0</Double>
                        <Double>0</Double>
                    </PaymentBucketAmount>
                    <PaymentBucketDueDate>
                        <DateTime>1/1/0001 12:00:00 AM</DateTime>
                        <DateTime>1/1/0001 12:00:00 AM</DateTime>
                        <DateTime>1/1/0001 12:00:00 AM</DateTime>
                        <DateTime>1/1/0001 12:00:00 AM</DateTime>
                        <DateTime>1/1/0001 12:00:00 AM</DateTime>
                        <DateTime>1/1/0001 12:00:00 AM</DateTime>
                        <DateTime>1/1/0001 12:00:00 AM</DateTime>
                        <DateTime>1/1/0001 12:00:00 AM</DateTime>
                    </PaymentBucketDueDate>
        <Warnings />
        <SList />
        <LList />
        <PA>False</PA>
        <PL>False</PL>
        <PC>False</PC>
        <PCs>False</PCs>
        <PJ>False</PJ>
        <OITC>0</OITC>
        <MG />
        <R />
        <CCGoods />
</Response>

[toc] | [next] | [standalone]

#93192

On Thu, 25 Jun 2015 11:39:53 -0700, kbtyo wrote:

> My question can be found here:
> 
> http://stackoverflow.com/questions/31058100/enumerate-column-headers-in-
csv-that-belong-to-the-same-tag-key-in-python

I suggest you look on stack overflow for the answer.

You appear to have failed to comprehend (again) the fact that the xml 
data is an unordered list, and are trying to assign an order to it.

If the xml data was ordered, either each tag would be different, or each 
tag would have an attribute specifying a sequence number.

If the original data is ordered, then you need to go back to the code 
that creates the xml and make it add the ordering information to the xml 
(ideally as an attribute of the tags concerned) to specify the order.

You can not reliably and with any certainty of being accurate try and 
assign some sequence to the list of similarly named elements simply from 
their position in the list.

You don't know if the code that creates the xml is walking the array like 
this:

i = sizeof(arr);
while (i--) addXmlElement(arr[i]);

or like this:

for (i = 0; i < arr.len; i++) addXmlElement(arr[i]);

or even like this:

while (sizeof(arr)) addXmlElement(popArrElement(arr[sizeof(arr)/2]));

Granted the latter is unlikely and perverse, but still not wholly 
impossible. More perverse and less likely mechanisms are also available.

But my point remains - that if the ordering data is not passed into the 
xml, then you can not magically add it when you later use the xml.

-- 
Denis McMahon, denismfmcmahon@gmail.com

[toc] | [prev] | [next] | [standalone]

#93269

Denis McMahon schrieb am 26.06.2015 um 09:44:
> xml data is an unordered list, and are trying to assign an order to it.
> 
> If the xml data was ordered, either each tag would be different, or each 
> tag would have an attribute specifying a sequence number.

XML is not unordered. The document order is well defined and entirely
obvious from the data. Whether this order is relevant and has a meaning or
not is, however, not part of XML itself but is left to the semantics of the
specific document format at hand. Meaning, XML document formats can choose
to ignore that order and define it as irrelevant. That doesn't mean it's
not there for a given document, but it may mean that a re-transmission of
the same document would be allowed to use a different order without
changing the information.

This property applies to pretty much all structured data formats and not
just XML, by the way, also to CSV and other tabular formats.

Stefan

[toc] | [prev] | [next] | [standalone]

#93275

On Sun, 28 Jun 2015 09:46:36 +0200, Stefan Behnel wrote:

> Denis McMahon schrieb am 26.06.2015 um 09:44:
>> xml data is an unordered list, and are trying to assign an order to it.
>> 
>> If the xml data was ordered, either each tag would be different, or
>> each tag would have an attribute specifying a sequence number.
> 
> XML is not unordered. The document order is well defined and entirely
> obvious from the data. Whether this order is relevant and has a meaning
> or not is, however, not part of XML itself but is left to the semantics
> of the specific document format at hand. Meaning, XML document formats
> can choose to ignore that order and define it as irrelevant. That
> doesn't mean it's not there for a given document, but it may mean that a
> re-transmission of the same document would be allowed to use a different
> order without changing the information.
> 
> This property applies to pretty much all structured data formats and not
> just XML, by the way, also to CSV and other tabular formats.

The point I am trying to make to OP is that the following two XML 
fragments define the same data:

<things>
  <thing>string 1</thing>
  <thing>string 2</thing>
  <thing>string 3</thing>
</things>

and:

<things>
  <thing>string 3</thing>
  <thing>string 2</thing>
  <thing>string 1</thing>
</things>

Each <thing> is just a member of the collection things, the xml does not 
contain sufficient information to state that <things> is an ordered 
collection containing a specific sequence of <thing>.

Mechanisms such as node.firstChild and node.getChild(x) are all very well 
for manipulating the xml, but any specific ordering of the original data 
should be carried out by using an appropriate attribute of the ordered 
data elements at the point where the xml representation is created.

-- 
Denis McMahon, denismfmcmahon@gmail.com

[toc] | [prev] | [next] | [standalone]

#93277

On Sunday, June 28, 2015 at 5:02:19 PM UTC-4, Denis McMahon wrote:
> On Sun, 28 Jun 2015 09:46:36 +0200, Stefan Behnel wrote:
> 
> > Denis McMahon schrieb am 26.06.2015 um 09:44:
> >> xml data is an unordered list, and are trying to assign an order to it.
> >> 
> >> If the xml data was ordered, either each tag would be different, or
> >> each tag would have an attribute specifying a sequence number.
> > 
> > XML is not unordered. The document order is well defined and entirely
> > obvious from the data. Whether this order is relevant and has a meaning
> > or not is, however, not part of XML itself but is left to the semantics
> > of the specific document format at hand. Meaning, XML document formats
> > can choose to ignore that order and define it as irrelevant. That
> > doesn't mean it's not there for a given document, but it may mean that a
> > re-transmission of the same document would be allowed to use a different
> > order without changing the information.
> > 
> > This property applies to pretty much all structured data formats and not
> > just XML, by the way, also to CSV and other tabular formats.
> 
> The point I am trying to make to OP is that the following two XML 
> fragments define the same data:
> 
> <things>
>   <thing>string 1</thing>
>   <thing>string 2</thing>
>   <thing>string 3</thing>
> </things>
> 
> and:
> 
> <things>
>   <thing>string 3</thing>
>   <thing>string 2</thing>
>   <thing>string 1</thing>
> </things>
> 
> Each <thing> is just a member of the collection things, the xml does not 
> contain sufficient information to state that <things> is an ordered 
> collection containing a specific sequence of <thing>.

You are right that XML does not specify that <things> is an ordered collection.
But XML does preserve the order of the children.  There are many XML schema
that rely on XML's order-preserving nature.

> 
> Mechanisms such as node.firstChild and node.getChild(x) are all very well 
> for manipulating the xml, but any specific ordering of the original data 
> should be carried out by using an appropriate attribute of the ordered 
> data elements at the point where the xml representation is created.
> 
> -- 
> Denis McMahon, denismfmcmahon@gmail.com

[toc] | [prev] | [next] | [standalone]

#93304

On Sun, 28 Jun 2015 17:07:00 -0700, Ned Batchelder wrote:

> On Sunday, June 28, 2015 at 5:02:19 PM UTC-4, Denis McMahon wrote:

>> <things>
>>   <thing>string 3</thing>
>>   <thing>string 2</thing>
>>   <thing>string 1</thing>
>> </things>

>> Each <thing> is just a member of the collection things, the xml does
>> not contain sufficient information to state that <things> is an ordered
>> collection containing a specific sequence of <thing>.
> 
> You are right that XML does not specify that <things> is an ordered
> collection.
> But XML does preserve the order of the children.  There are many XML
> schema that rely on XML's order-preserving nature.

But what we *don't* know is whether the order of the umpteen identical 
tags in the XML has any significance in terms of the original data, 
although the OP seems intent on assigning some significance to that order 
without any basis for doing so.

Consider the following tuple:

t = (tuplemember_1, tuplemember_2, .... tuplemember_n)

Can we safely assume that if the tuple is ever converted to xml, either 
now or at some future date using whatever the code implementation is 
then, that the order of the items will be preserved:

<tuple>
  <item>tuplemember_1</item>
  <item>tuplemember_2</item>
....
  <item>tuplemember_n/item>
</tuple>

And if we're reading that xml structure at some point in the future, is 
it safe to assume that the tuple members are in the same order in the xml 
as they were in the original tuple?

For sanity <item> should have an attribute specifying the sequence of the 
item in it's tuple.
-- 
Denis McMahon, denismfmcmahon@gmail.com

[toc] | [prev] | [next] | [standalone]

#93319

On 2015-06-30 01:54, Denis McMahon wrote:
> On Sun, 28 Jun 2015 17:07:00 -0700, Ned Batchelder wrote:
>
>> On Sunday, June 28, 2015 at 5:02:19 PM UTC-4, Denis McMahon wrote:
>
>>> <things>
>>>    <thing>string 3</thing>
>>>    <thing>string 2</thing>
>>>    <thing>string 1</thing>
>>> </things>
>
>>> Each <thing> is just a member of the collection things, the xml does
>>> not contain sufficient information to state that <things> is an ordered
>>> collection containing a specific sequence of <thing>.
>>
>> You are right that XML does not specify that <things> is an ordered
>> collection.
>> But XML does preserve the order of the children.  There are many XML
>> schema that rely on XML's order-preserving nature.
>
> But what we *don't* know is whether the order of the umpteen identical
> tags in the XML has any significance in terms of the original data,
> although the OP seems intent on assigning some significance to that order
> without any basis for doing so.
>
> Consider the following tuple:
>
> t = (tuplemember_1, tuplemember_2, .... tuplemember_n)
>
> Can we safely assume that if the tuple is ever converted to xml, either
> now or at some future date using whatever the code implementation is
> then, that the order of the items will be preserved:
>
> <tuple>
>    <item>tuplemember_1</item>
>    <item>tuplemember_2</item>
> ....
>    <item>tuplemember_n/item>
> </tuple>

Barring bugs, yes!

> And if we're reading that xml structure at some point in the future, is
> it safe to assume that the tuple members are in the same order in the xml
> as they were in the original tuple?

Yes! Any conforming XML implementation will preserve the order.

> For sanity <item> should have an attribute specifying the sequence of the
> item in it's tuple.

While it may make you more comfortable, it's hardly a requirement for sanity.

I think you had a point in your first paragraph here, but you are obscuring it 
with FUD. The problem is not whether unadorned XML elements can be used to 
represent an ordered collection. They can and are, frequently, without any 
problem because XML elements are intrinsically ordered.

The real problem that you almost get around to articulating is that XML elements 
can *also* be used to represent unordered collections simply by ignoring the 
(preserved) order of the elements. And if you are completely blind as to the 
schema as the OP apparently is, and you are simply given a load of XML and told 
to do "something" with it, you don't know if any given collection is meant to be 
ordered or unordered. Of course, the only sensible thing to do is just preserve 
the order given to you as that is what the semantics of XML requires of you in 
the absence of a schema that says otherwise. You can always disregard the order 
later.

That said, if the data is regular enough to actually be restructured into a 
table (i.e. if <MonthDayCount> always has the same number of children, etc.), 
then it probably does represent an ordered collection. If it's variable, then 
putting it into a table structure probably doesn't make any sense regardless of 
ordering issues.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

[toc] | [prev] | [next] | [standalone]

#93339

Robert Kern <robert.kern@gmail.com>:

>> <tuple>
>>    <item>tuplemember_1</item>
>>    <item>tuplemember_2</item>
>> ....
>>    <item>tuplemember_n/item>
>> </tuple>
>
> [...]
>
> Yes! Any conforming XML implementation will preserve the order.

And not only the order: the newlines and other whitespace around the
<item>s are also preserved as children of <tuple>.

> And if you are completely blind as to the schema as the OP apparently
> is, and you are simply given a load of XML and told to do "something"
> with it, you don't know if any given collection is meant to be ordered
> or unordered. Of course, the only sensible thing to do is just
> preserve the order given to you as that is what the semantics of XML
> requires of you in the absence of a schema that says otherwise.

XML is an unfortunate creation. You cannot fully parse it without
knowing the schema (or document type). For example, these constructs
might or might not be equivalent:

    <stuff>&aring;</stuff>

    <stuff>å</stuff>

That is because &...; entities are defined in the document type.

Similarly, these two constructs might or might not be equivalent:

   <stuff greet="hello"/>

   <stuff greet=" hello "/>

By default, they would be, but the document type can declare whitespce
significant around an attribute value.

Finally, these two constructs might or might not be equivalent:

   <count> 7 </count>
   <count>7</count>

By default, they wouldn't be, but the document type can declare
whitespace *insignificant* around element contents.

Sigh, S-expressions would have been a much better universal data format.


Marko

[toc] | [prev] | [next] | [standalone]

#93288

On 2015-06-28 22:00, Denis McMahon wrote:
> On Sun, 28 Jun 2015 09:46:36 +0200, Stefan Behnel wrote:
>
>> Denis McMahon schrieb am 26.06.2015 um 09:44:
>>> xml data is an unordered list, and are trying to assign an order to it.
>>>
>>> If the xml data was ordered, either each tag would be different, or
>>> each tag would have an attribute specifying a sequence number.
>>
>> XML is not unordered. The document order is well defined and entirely
>> obvious from the data. Whether this order is relevant and has a meaning
>> or not is, however, not part of XML itself but is left to the semantics
>> of the specific document format at hand. Meaning, XML document formats
>> can choose to ignore that order and define it as irrelevant. That
>> doesn't mean it's not there for a given document, but it may mean that a
>> re-transmission of the same document would be allowed to use a different
>> order without changing the information.
>>
>> This property applies to pretty much all structured data formats and not
>> just XML, by the way, also to CSV and other tabular formats.
>
> The point I am trying to make to OP is that the following two XML
> fragments define the same data:
>
> <things>
>    <thing>string 1</thing>
>    <thing>string 2</thing>
>    <thing>string 3</thing>
> </things>
>
> and:
>
> <things>
>    <thing>string 3</thing>
>    <thing>string 2</thing>
>    <thing>string 1</thing>
> </things>
>
> Each <thing> is just a member of the collection things, the xml does not
> contain sufficient information to state that <things> is an ordered
> collection containing a specific sequence of <thing>.

Without reference to a schema that explicitly defines the children of <things> 
to be unordered, this is not true. The XML Information Set defines the children 
of all elements to be an ordered set (in contrast to attributes, which are 
unordered). A particular XML schema may choose to consider the order of children 
of a particular element to be irrelevant, but XML itself keeps them ordered.

http://www.w3.org/TR/xml-infoset/#infoitem.element

If it didn't, then XHTML would have a hell of a time with ordered constructs 
like this:

   <ol>
     <li>First item</li>
     <li>Second item</li>
     <li>Third item</li>
   </ol>

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

[toc] | [prev] | [next] | [standalone]

#93290

On Sunday, 28 June 2015 03:46:56 UTC-4, Stefan Behnel  wrote:
> Denis McMahon schrieb am 26.06.2015 um 09:44:
> > xml data is an unordered list, and are trying to assign an order to it.
> > 
> > If the xml data was ordered, either each tag would be different, or each 
> > tag would have an attribute specifying a sequence number.
> 
> XML is not unordered. The document order is well defined and entirely
> obvious from the data. Whether this order is relevant and has a meaning or
> not is, however, not part of XML itself but is left to the semantics of the
> specific document format at hand. Meaning, XML document formats can choose
> to ignore that order and define it as irrelevant. That doesn't mean it's
> not there for a given document, but it may mean that a re-transmission of
> the same document would be allowed to use a different order without
> changing the information.
> 
> This property applies to pretty much all structured data formats and not
> just XML, by the way, also to CSV and other tabular formats.
> 
> Stefan

@Stefan, Ned, and Robert: You have all hit the nail on the head. I do not have an authentic and veritable XSD (or XML data structures for that matter). So far it is all deprecated and/or anonymized data from the client. Therefore, I can only hypothesize what the end output will be for the database architecture that I am working with. 

From what I understand, therefore, based on your constructive insight, is that the 14 occurrences of the same tag (regardless of placement relative to neighbouring children and the root are all being defined as the same key. However, their individual values are also being treated as the same (from the algorithm that I wrote in my Stack Overflow post (please see above)). The constraint is that I am anticipating terabytes of data every day from the client in the coming months. The algorithm should be able to parse, and write out to CSV in the most efficient manner. That is my design constraint. I welcome your feedback on this. 

Here is the post, again, for your convenience: 

http://stackoverflow.com/questions/31058100/enumerate-column-headers-in-csv-that-belong-to-the-same-tag-key-in-python 

[toc] | [prev] | [next] | [standalone]

#93306

On Mon, 29 Jun 2015 07:52:07 -0700 (PDT), Sahlusar
<sahluwalia@wynyardgroup.com> declaimed the following:


>
>From what I understand, therefore, based on your constructive insight, is that the 14 occurrences of the same tag (regardless of placement relative to neighbouring children and the root are all being defined as the same key. However, their individual values are also being treated as the same (from the algorithm that I wrote in my Stack Overflow post (please see above)). The constraint is that I am anticipating terabytes of data every day from the client in the coming months. The algorithm should be able to parse, and write out to CSV in the most efficient manner. That is my design constraint. I welcome your feedback on this. 
>

	I sure hope that "terabytes of data" is hyperbole...

	My system takes something like three hours just to generate a 500GB
backup (one partition each week -- I have a 4TB backup drive with only
740GB free; the other drives are only half full or I'd need an 8TB backup).
And that's using a compiled backup program -- I'd hate to consider what
Python would require to backup the partition.

-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#93327

On Tue, Jun 30, 2015 at 11:26 AM, Dennis Lee Bieber
<wlfraed@ix.netcom.com> wrote:
>         My system takes something like three hours just to generate a 500GB
> backup (one partition each week -- I have a 4TB backup drive with only
> 740GB free; the other drives are only half full or I'd need an 8TB backup).
> And that's using a compiled backup program -- I'd hate to consider what
> Python would require to backup the partition.

Probably about the same, actually. In my experience, there's often
very little speed difference between a straight 'dd' from one
partition to another (say, making a disk image prior to data recovery)
and doing more complicated work (say, archiving and compressing).
Until you actually manage to saturate your CPU with the workload
(video editing or something), the time is most likely to be dominated
by the disk platters.

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web