Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #97970

Parsing adhoc structured data

Show all headers | View raw

I do a lot of data analysis with Python, and most of the time I have to work with structured data, but where the structure is typically undocumented, and/or unclear. So a lot of my time is spent trying to discern a structure from the data, so that I can work with it.

For example, a recent file I needed to parse looked something like this:

    # Schema version - will update if the format changes
    schema: 1

    group: IT
        type: Developer
            installation:
                package: Python
                version: 3.4
            installation:
                package: Vim
            installation:
                package: Java
                version: 8
        type: Manager
            ...

So it's key: value, with nested blocks represented by indentation (but I can't be sure each indent is 4 spaces, at least not without scanning all the data and hoping new files don't break that rule). Some keys can have no value, and/or can be repeated multiple times.

It's got comments and blank lines (easy to handle) and a schema version, which would be nice if I could get a spec for the schema (which, of course, I can't, but at least I can see if my assumptions may no longer be valid...)

All I want to do in the first instance is to (relatively quickly) load the data into some sort of Python data structure, and then start analyzing it. For example, check the "installation" items to see what keys they contain, which keys are always present and which are optional, whether any are numeric, etc. Once I have that information, I can start putting the data into a more structured or manageable form (possibly a database, possibly simply a Python data structure that I'll serialise to disk for later analysis).

Every time I encounter data like this I find myself writing large amounts of either adhoc parser code (which usually takes me a couple of goes to get right :-() or way too much error handling checking for cases that end up never happening in practice.

This feels to me like a natural type of activity to use Python for (it's essentially a "data cleansing" type of activity) but I've had no luck finding libraries or tools that can make it easy.

Does anyone know of any good libraries for this type of "adhoc parsing" type of activity? Or any websites, articles or books discussing this sort of thing? I'm doing this as a sort of supporting activity to my "real" job, which is usually the data analysis (that I can't do till I get the data loaded!) so I'm not a particular expert on the best techniques for something like this.

Thanks,
Paul

Back to comp.lang.python | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Parsing adhoc structured data Paul  Moore <p.f.moore@gmail.com> - 2015-10-27 07:32 -0700
  Re: Parsing adhoc structured data "TommyVee" <xxxxxxxx@xxxxxx.xxx> - 2015-10-29 06:58 -0400

csiph-web