Path: csiph.com!usenet.pasdenom.info!news.redatomik.org!newsfeed.xs4all.nl!newsfeed7.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Fri, 17 Jul 2015 23:30:19 +0200
From: Anders Wegge Keller <wegge@wegge.dk>
To: python-list@python.org
Subject: Parsing and displaying C structs in a semi-intelligent way.
Organization: Disorganized
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.662.1437169020.3674.python-list@python.org>
Lines: 55
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:94034

 In my day job, we have a large code base of mostly identical projects, each
with their own customizations. The customizations can be a real pain
sometimes. Especially when debugging binary data. The interesting part of
the binary dumps are most often the stuff that are tweaked from project to
project. The prime suspect is a structure like this:

typedef struct
{
    byte        next ;                  /* Index to next wedge          */
    byte        flags ;                 /* ricd-control flags           */
    word        alt [MAX_alt] ;         /* Alternative wedges           */
...
} ConvResult ;

typedef struct
{
    byte                state ;         /* UCART_ (see cart.h)          */
    word                headcart ;      /* Cart number for header cart  */
    ConvResult          conv ;
...
} cartUTable_t ;

 The actual structure with substructures are much larger. This structure
contains all the information used when sorting stuff[1]. For historical
reasons, the data is dumped to a binary file at the end of the items
life-cycle. We do have a debugging tool, that reads an in-house file format
specifying the members of the structure, and how to display them. However,
updating this description is error-prone, and just about as funny as waiting
for the Norwegian Blue to fly out over the fiords. So most often it's either
not done, or is unusable. 

 To clean up this mess, I've devised a cunning plan, involving a C-parser,
that can write a structure definition as XML. Reading this is simple enough,
and going from that to a format string for struct.unpack() is mostly
trivial. Except for the arrays of some C-type, that happens to be something
else than an array of char. Due to the normal use case[2] of dumping this
binary data, I need to get the parsed data munged into a form, that can be
put into a namedtuple as single member, even when I have an array of
(typical) six conv.alt values. I'm getting stuck with the limitations of
struct and namedtuple. While inheriting either of them to get the job done,
seem to be feasible, I still feel that I somehow should be able to do
something "list comprehension"-ish, to avoid going there. I just can't see
the light.

(Thank you for reading all the way to here!)


1. Luggage, parcels, shop replenishments ... In short: mostly rectangular
   things that has a barcode attached.

2. Random ordering, not displaying all members, and only a part of the
   arrays.

-- 
//Wegge