Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #83230 > unrolled thread

If you were starting a project with XML datasource using python

Back to article view | Back to comp.lang.python

Contents

  If you were starting a project with  XML datasource using python flebber <flebber.crue@gmail.com> - 2015-01-05 03:54 -0800
    Re: If you were starting a project with  XML datasource using python Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2015-01-05 08:56 -0500
      Re: If you were starting a project with  XML datasource using python flebber <flebber.crue@gmail.com> - 2015-01-05 10:24 -0800
        Re: If you were starting a project with  XML datasource using python Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2015-01-05 19:49 -0500
        Re: If you were starting a project with XML datasource using python Chris Angelico <rosuav@gmail.com> - 2015-01-06 12:55 +1100
        Re: If you were starting a project with XML datasource using python George Silva <georger.silva@gmail.com> - 2015-01-06 00:24 -0200
    Re: If you were starting a project with XML datasource using python Dan Stromberg <drsalists@gmail.com> - 2015-01-05 14:45 -0800
    Re: If you were starting a project with  XML datasource using python Steve Hayes <hayesstw@telkomsa.net> - 2015-01-06 05:59 +0200

#83230 — If you were starting a project with XML datasource using python

Hi

I need some advice on managing data in the form of xml. I will have to repeatedly import a small xml file but with many complex attributes.

If I want to retain data integrity and keep the import process simple and querying from the stored source simple what are my best options?

There are several options for converting XML into objects such as:
http://lxml.de/objectify.html
https://pypi.python.org/pypi/pyxml2obj/
http://eulxml.readthedocs.org/en/latest/xmlmap.html

I could push this as an embedded object into mongo and search from there.

Could ignore XML by just converting to json with something like xml2json and pushing to many databases from there.

MySQL, Mongodb, couchdb and existdb are all viable options. Existdb is  directly an XML db so I could directly store from there and use the REST API to access http://exist-db.org/exist/apps/doc/devguide_rest.xml

somewhat bamboozled by the range of options and no clear obvious solution.

If you were starting a project, it relied on XML as its datasource what would you use and why? And have you used it or just speculating?

Thank you for your time

Sayth

[toc] | [next] | [standalone]

#83241

On Mon, 5 Jan 2015 03:54:06 -0800 (PST), flebber <flebber.crue@gmail.com>
declaimed the following:

>If you were starting a project, it relied on XML as its datasource what would you use and why? And have you used it or just speculating?
>
	If I were starting a project, I'd argue furiously that XML is NOT
something that should be used for dynamic data storage (the original GRAMPS
not withstanding -- I see it now uses BSDdb). It may be a useful
representation for transferring/transforming data, but not for
persistance/updates.

	If used for data transfer, the schema should be well thought out to
facilitate updates to the long-term storage format.
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#83252

> 
> >If you were starting a project, it relied on XML as its datasource what would you use and why? And have you used it or just speculating?
> >
> 	If I were starting a project, I'd argue furiously that XML is NOT
> something that should be used for dynamic data storage (the original GRAMPS
> not withstanding -- I see it now uses BSDdb). It may be a useful
> representation for transferring/transforming data, but not for
> persistance/updates.
> 
> 	If used for data transfer, the schema should be well thought out to
> facilitate updates to the long-term storage format.

So you would convert it to json so it can then be stored? 

Note I don't control the XML at all that is a separate organisation I just have to deal with it?

Sayth

[toc] | [prev] | [next] | [standalone]

#83259

On Mon, 5 Jan 2015 10:24:25 -0800 (PST), flebber <flebber.crue@gmail.com>
declaimed the following:


>So you would convert it to json so it can then be stored? 
>
	NO!

	JSON and XML, in my mind, are equivalents -- (human-readable) notations
for transferring data between applications. JSON may just have fewer
"options" for how to notate basic data.

-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#83262 — Re: If you were starting a project with XML datasource using python

On Tue, Jan 6, 2015 at 11:49 AM, Dennis Lee Bieber
<wlfraed@ix.netcom.com> wrote:
>>So you would convert it to json so it can then be stored?
>>
>         NO!
>
>         JSON and XML, in my mind, are equivalents -- (human-readable) notations
> for transferring data between applications. JSON may just have fewer
> "options" for how to notate basic data.

JSON corresponds much more closely to Python data types, but
converting from one to the other can't improve matters any.

ChrisA

[toc] | [prev] | [next] | [standalone]

#83263 — Re: If you were starting a project with XML datasource using python

[Multipart message — attachments visible in raw view] — view raw

As people already said, don't use XML for persistence.

Read it, parse it, and persist it in another format. databases are quite
good for that. even sqlite3 will outperfom any few MB xml files.

On Mon, Jan 5, 2015 at 11:55 PM, Chris Angelico <rosuav@gmail.com> wrote:

> On Tue, Jan 6, 2015 at 11:49 AM, Dennis Lee Bieber
> <wlfraed@ix.netcom.com> wrote:
> >>So you would convert it to json so it can then be stored?
> >>
> >         NO!
> >
> >         JSON and XML, in my mind, are equivalents -- (human-readable)
> notations
> > for transferring data between applications. JSON may just have fewer
> > "options" for how to notate basic data.
>
> JSON corresponds much more closely to Python data types, but
> converting from one to the other can't improve matters any.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>



-- 
George R. C. Silva
SIGMA Consultoria
----------------------------
http://www.consultoriasigma.com.br/

[toc] | [prev] | [next] | [standalone]

#83257 — Re: If you were starting a project with XML datasource using python

On Mon, Jan 5, 2015 at 3:54 AM, flebber <flebber.crue@gmail.com> wrote:
> Hi
>
> I need some advice on managing data in the form of xml. I will have to repeatedly import a small xml file but with many complex attributes.

If I have a choice, I choose JSON over XML.

If stuck with XML, I like xmltodict: https://pypi.python.org/pypi/xmltodict

I've used xmltodict in two projects - both worked well, except I
needed to define:

def dict_constructor(*args, **kwargs):
    '''
    With this higher order function passed to xmltodict.parse, we're
forcing singleton tags to be returned as lists.
    See also https://github.com/martinblech/xmltodict/issues/14 .
    '''
    return collections.defaultdict(list, *args, **kwargs)

...and use it like:
    xml_dict = xmltodict.parse(xml_string, dict_constructor=dict_constructor)

This was necessary to keep singletons from being returned as
singletons, instead returning them as single-element lists.  Sometimes
singletons are fine, but other times you need to be able to accept 1
or more of something  without a traceback.

HTH

[toc] | [prev] | [next] | [standalone]

#83264

On Mon, 5 Jan 2015 03:54:06 -0800 (PST), flebber <flebber.crue@gmail.com>
wrote:

Hi

I need some advice on managing data in the form of xml. I will have to
repeatedly import a small xml file but with many complex attributes.

If I want to retain data integrity and keep the import process simple and
querying from the stored source simple what are my best options?

There are several options for converting XML into objects such as:
http://lxml.de/objectify.html
https://pypi.python.org/pypi/pyxml2obj/
http://eulxml.readthedocs.org/en/latest/xmlmap.html

I could push this as an embedded object into mongo and search from there.

Could ignore XML by just converting to json with something like xml2json and
pushing to many databases from there.

(reposted to fix the long lines produced by the lame GoogleGroups editor)

-- 
Steve Hayes from Tshwane, South Africa
Web:  http://www.khanya.org.za/stevesig.htm
Blog: http://khanya.wordpress.com
E-mail - see web page, or parse: shayes at dunelm full stop org full stop uk

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web