Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #26575 > unrolled thread

packed structs

Back to article view | Back to comp.lang.c

Contents

  packed structs JohnF <john@please.see.sig.for.email.com> - 2012-09-22 01:54 +0000
    Re: packed structs Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-09-21 23:22 -0400
      Re: packed structs JohnF <john@please.see.sig.for.email.com> - 2012-09-22 06:37 +0000
        Re: packed structs "BartC" <bc@freeuk.com> - 2012-09-22 13:47 +0100
          Re: packed structs Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-09-22 14:00 +0100
          Re: packed structs JohnF <john@please.see.sig.for.email.com> - 2012-09-22 15:42 +0000
        Re: packed structs Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-09-22 09:13 -0400
        Re: packed structs Johann Klammer <klammerj@NOSPAM.a1.net> - 2012-09-23 03:10 +0200
          Re: packed structs JohnF <john@please.see.sig.for.email.com> - 2012-09-23 02:10 +0000
        Re: packed structs Stephen Sprunk <stephen@sprunk.org> - 2012-09-23 11:44 -0500
          Re: packed structs JohnF <john@please.see.sig.for.email.com> - 2012-09-23 23:23 +0000
            Re: packed structs Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-09-24 01:59 +0100
              Re: packed structs JohnF <john@please.see.sig.for.email.com> - 2012-09-24 02:54 +0000
                Re: packed structs Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-09-24 04:38 +0100
                  Re: packed structs JohnF <john@please.see.sig.for.email.com> - 2012-09-24 04:07 +0000
                    Re: packed structs Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-09-24 12:16 +0100
                      Re: packed structs JohnF <john@please.see.sig.for.email.com> - 2012-09-24 11:45 +0000
                Re: packed structs "BartC" <bc@freeuk.com> - 2012-09-24 10:18 +0100
                  Re: packed structs JohnF <john@please.see.sig.for.email.com> - 2012-09-24 11:04 +0000
                    Re: packed structs Stephen Sprunk <stephen@sprunk.org> - 2012-09-30 14:21 -0500
                      Re: packed structs JohnF <john@please.see.sig.for.email.com> - 2012-10-01 07:34 +0000
                        Re: packed structs Stephen Sprunk <stephen@sprunk.org> - 2012-10-15 00:46 -0500
            Re: packed structs Stephen Sprunk <stephen@sprunk.org> - 2012-09-30 13:52 -0500
      Re: packed structs Nick Keighley <nick_keighley_nospam@hotmail.com> - 2012-09-22 01:31 -0700
        Re: packed structs JohnF <john@please.see.sig.for.email.com> - 2012-09-22 08:53 +0000
          Re: packed structs Jorgen Grahn <grahn+nntp@snipabacken.se> - 2012-09-22 14:17 +0000
            Re: packed structs JohnF <john@please.see.sig.for.email.com> - 2012-09-22 15:33 +0000
              Re: packed structs Jorgen Grahn <grahn+nntp@snipabacken.se> - 2012-09-22 20:43 +0000
              Re: packed structs "BartC" <bc@freeuk.com> - 2012-09-22 22:52 +0100
            Re: packed structs Keith Thompson <kst-u@mib.org> - 2012-09-22 13:47 -0700
              Re: packed structs JohnF <john@forkosh.com.com> - 2012-09-23 00:19 +0000
                Re: packed structs Ian Collins <ian-news@hotmail.com> - 2012-09-23 13:32 +1200
                  Re: packed structs JohnF <john@please.see.sig.for.email.com> - 2012-09-23 02:16 +0000
          Re: packed structs Ian Collins <ian-news@hotmail.com> - 2012-09-23 10:33 +1200
            Re: packed structs Nick Keighley <nick_keighley_nospam@hotmail.com> - 2012-09-23 01:38 -0700
    Re: packed structs The Great Firewall of China Blue <chine.bleu@yahoo.com> - 2012-09-21 21:29 -0700
    Re: packed structs W Karas <wkaras@yahoo.com> - 2012-10-16 09:13 -0700

Page 2 of 2 — ← Prev page 1 [2]

#26920

Stephen Sprunk <stephen@sprunk.org> wrote:
> JohnF wrote:
>> Well, you'd use it however you liked. But for my gif situation,
>> I'd envisioned >>doing away with structs entirely<<, packed or not.
>> The smemf >>format string totally replaces<< the need for any struct.
>> And, of course, I prototyped that for myself, i.e., wrote some
>> pseudocode using the as-yet-uncompleted smemf, just to make sure
>> that idea seems to work.
>>     You can see exhaustively complete comments about the gif block
>> formats at forkosh.com/gifsave89.html by clicking the Listing link
>> along the left-hand side under "Related Pages", and scrolling down
>> to line#493 for the GIFIMAGEDESCRIP struct. My "pseudocode" for that
>> is just one smemf statement that totally replaces the struct,
>> 
>> nbitsinbuffer =  /* whitespace in smemf format string is ignored */
>>   smemf(buffer, " 2C %x  "   /* Image Descriptor identifier is hex 2C */
>>                 "    %2ld"   /* 2-byte little-endian word for X-pos */
>>                 "    %2ld"   /* 2-byte little-endian word for Top */
>>                 "    %2ld"   /* 2-byte little-endian word for Width */
>>                 "    %2ld"   /* 2-byte little-endian word for Height */
>>   /* following is the "Packed" Byte consisting of five bit fields */
>>                 "    %3b "   /* 3-bits #colorbits */
>>                 "  0 %2b "   /* 2-bits "reserved bits" */
>>                 "  0 %1b "   /* 1-bit  local colortable sort flag */
>>                 "  0 %1b "   /* 1-bit  interlace flag */
>>                 "    %1b ",  /* 1-bit  local colortable flag */
>>   col0,row0, ncols,nrows, ncolorbits, (colortable==NULL?0:1) );
>> 
>> And smemf() returns the size, in #bits, of the buffer it constructs.
>> That would usually be a multiple of 8, in which case you can just
>> fwrite(buffer,etc), or do whatever you want with it.
> 
> This example shows how far you have diverged from sprintf().  From such
> a claim, I would have expected something more like this:
> 
> nbitsinbuffer = smemf(buffer,
>    "%1lu"   /* Image Descriptor, 8-bit little-endian unsigned int */
>    "%2lu"   /* X-Pos, 16-bit little-endian unsigned int */
>    "%2lu"   /* Top, 16-bit little-endian unsigned int */
>    "%2lu"   /* Width, 16-bit little-endian unsigned int */
>    "%2lu"   /* Height, 16-bit little-endian unsigned int */
>    /* following is the "Packed" Byte consisting of five bit fields */
>    "%3b"   /* #colorbits, 3-bit field */
>    "%2b"   /* reserved, 2-bit field */
>    "%1b"   /* local colortable sort flag, 1-bit field */
>    "%1b"   /* interlace flag, 1-bit field */
>    "%1b",  /* local colortable flag, 1-bit field */
>    0x2C, col0, row0, ncols, nrows, ncolorbits, 0, 0, 0,
> (colortable==NULL?0:1) );

Thanks for the alternative spec. I'll keep it in mind.
I agree that your format's more sprintf-like. Maybe I
should "deprecate" that literal requirement. More
about why in response to your specific remarks below...

> Notice that whitespace is _not_ ignored and that there are ten arguments
> corresponding to ten format specifiers.

Both of which I'd consider bad news. Certainly, ignoring
whitespace is different. But notice that your format string
has none. And likely never would, because how often is
whitespace going to be part of a binary packet format
specification? So it's basically no use for this particular
purpose.
   And you also don't get constants in your formats.
Instead, you've got that 0x2C and 0,0,0 in the arguments
which "force-feeds" the "%1b" specifiers. And which
the user has to know about. Instead, I'm moving those
same 0x2C and 0's into the format string
preceding the %1b. Different, again, but serves
the purpose better. The user shouldn't have to
remember magic numbers like 0x2C, and where to put
them.
   Instead, maybe some .h file with all the format
strings describing the packets for some particular
protocol/whatever would embed all that info and
hide it from the user. He'd just say something like,
  #include "packetformatstrings.h"
  unsigned char *thispacketbuffer = malloc(whatever);
  ...
  smemf(thispacketbuffer, thispacketformatstring,
    all,the,numbers,I,actually,care,about,and,no,others);
And that would do the entire job, with the user taking
care of his business only, and the binary packet formatting
pretty much entirely handled for him.

> I also used %u rather than %d
> since I'm pretty sure those ints are supposed to be unsigned (but I'm
> not sure that makes a difference here).
> 
> Of course, your use of "%ld" to mean "little-endian word" rather than
> "long signed integer" is a major difference as well, though that's
> forgivable since you obviously need more control over representation
> than sprintf()'s specifiers offer.  When thinking it through, though,
> that was the point at which I decided that trying to reuse those
> specifiers was probably more trouble than it was worth.

Forget the characters %u vs %d and length modifier, flags, etc.
That's easily/trivially changed. In fact, I'd already changed
my mind to %<d or (%<u if you prefer) for little endian,
and the obvious for big, but might change it again.
Just focus on the functionality. The format string syntax,
at least so far as specifiers, flags, modifiers, etc are
concerned, can be decided later. For now, they're just
illustrative -- you have to write something while discussing it.

> S
-- 
John Forkosh  ( mailto:  j@f.com  where j=john and f=forkosh )

[toc] | [prev] | [next] | [standalone]

#27240

On 01-Oct-12 02:34, JohnF wrote:
> Stephen Sprunk <stephen@sprunk.org> wrote:
>> This example shows how far you have diverged from sprintf().  From
>> such a claim, I would have expected something more like this:
>> 
>> ...
> 
> Thanks for the alternative spec. I'll keep it in mind. I agree that
> your format's more sprintf-like. Maybe I should "deprecate" that
> literal requirement. More about why in response to your specific
> remarks below...

Note that I'm not saying that my syntax is better; I'm just saying that
it is more like a sprintf() format string, which may actually turn out
to be worse than coming up with something completely new.

>> Notice that whitespace is _not_ ignored and that there are ten
>> arguments corresponding to ten format specifiers.
> 
> Both of which I'd consider bad news. Certainly, ignoring whitespace
> is different. But notice that your format string has none. And likely
> never would, because how often is whitespace going to be part of a
> binary packet format specification? So it's basically no use for this
> particular purpose.

sprintf() copies everything in the format string (including whitespace)
to its output, replacing format specifiers as it goes.  If you're going
to claim to be similar to sprintf(), then you need to do the same.

> And you also don't get constants in your formats. Instead, you've got
> that 0x2C and 0,0,0 in the arguments which "force-feeds" the "%1b"
> specifiers. And which the user has to know about. Instead, I'm moving
> those same 0x2C and 0's into the format string preceding the %1b.
> Different, again, but serves the purpose better. The user shouldn't
> have to remember magic numbers like 0x2C, and where to put them.

I can see cases where I might want to include literal bytes in the
format string, such as the "GIF89a" at the beginning of GIF files,
without having to use a format specifier.  In some cases, those literal
bytes might happen to include "whitespace".

Instead, rather that doing the same thing as sprintf(), you chose to
have format specifiers modify the preceding literal text in the format
string and treat whitespace specially, which is completely unlike sprintf().

>> I also used %u rather than %d since I'm pretty sure those ints are
>> supposed to be unsigned (but I'm not sure that makes a difference
>> here).
>> 
>> Of course, your use of "%ld" to mean "little-endian word" rather
>> than "long signed integer" is a major difference as well, though
>> that's forgivable since you obviously need more control over
>> representation than sprintf()'s specifiers offer.  When thinking it
>> through, though, that was the point at which I decided that trying
>> to reuse those specifiers was probably more trouble than it was
>> worth.
> 
> Forget the characters %u vs %d and length modifier, flags, etc. 
> That's easily/trivially changed. In fact, I'd already changed my mind
> to %<d or (%<u if you prefer) for little endian, and the obvious for
> big, but might change it again.

I'm curious how you'd specify middle-endian.  Presumably, without one of
those three modifiers, it would use native byte order?

> Just focus on the functionality. The format string syntax, at least
> so far as specifiers, flags, modifiers, etc are concerned, can be
> decided later. For now, they're just illustrative -- you have to
> write something while discussing it.

My point was merely to illustrate why I said your syntax didn't seem to
be similar to sprintf() as you had claimed.  I'm not debating whether it
is a good syntax without such a claim.

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [next] | [standalone]

#26895

On 23-Sep-12 18:23, JohnF wrote:
> Stephen Sprunk <stephen@sprunk.org> wrote:
>> On 22-Sep-12 01:37, JohnF wrote:
>>> I'd actually finished (what I think is) a more elegant solution, 
>>> that I called smemf() that's like memcpy() but under format 
>>> control, including additonal format specifiers for hex, for
>>> bits, and for other stuff. The code actually works fine, but
>>> still uncompleted is 723 lines (though that includes >>many<<
>>> comments), which is somewhat of a tail-wagging-dog situation
>>> which I also want to avoid.
>> 
>> Your smemf() looks interesting, but I'm curious why you went with
>> that syntax (which, despite claiming to be similar to printf,
>> doesn't seem to end up looking much like it) rather than leveraging
>> the syntax of an existing system for the same purpose, eg. Perl's
>> pack/unpack.
> 
> You didn't read all the followups -- I wasn't aware of perl's (or
> python's) pack/unpack, but they were brought to my attention (thanks
> again, guys). Then I indeed said I'd be reading up more carefully
> about them all, and trying to re-spec smemf() (and maybe rename it --
> pack or spackf???) based on the all that stuff, as the best C variant
> I can come up with.

Fair enough.

> By the way, compared with those pack/unpack formats, I think
> smemf()'s format string syntax is a lot more C/sprintf-like (could
> you be more specific?).

Aside from your use of percent signs for format specifiers, I don't see
much in common, and in fact I find your use of percent signs to be quite
misleading since they behave rather differently than printf()'s.

> That's, of course, the trick: access all functionality from a format
> string syntax that's immediately intuitively sensible to both
> (a)people already familiar with perl and/or python pack/unpack and
> maybe C/sprintf, as well as (b)people only familiar with C/sprintf.

I don't see how (a) could have been a goal if you hadn't been aware of
Perl's pack()/unpack() when you developed your smemf() interface.

> Since I'm a b-type myself, and wasn't even aware of the a-types until
> brought to my attention here, the C/sprintf look-alike was my sole
> original goal (which I'd thought was pretty successful, modulo the
> minimal unavoidable syntax differences due to fundamental functional
> requirements differences).

Some differences were unavoidable, but others were not.

For instance, sprintf() copies the format string verbatim to the output
except for format specifiers, which are replaced by formatted arguments.
 smemf() discards whitespace and puts literal arguments in the format
string, which are consumed by subsequent format specifiers.

More detail in my response to your example.

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [next] | [standalone]

#26579

On Sep 22, 4:23 am, Eric Sosman <esos...@ieee-dot-org.invalid> wrote:

serializing structs

>      One way is the function-per-struct-type approach, and
> although it may be "b..o..r..i..n..g" it has advantages
> that should not be dismissed lightly.  Consider that there
> are (most likely) only a handful of structs and hence only
> a handful of functions; writing them won't take enough
> time to b..o..r..e anyone except an ADHD sufferer.

I've dealt with cases where there were considerably more than a
"handful"- communication protocol.

Since I have a low b..o..r..e..d..o..m threshold I resorted to code
generation. The protocol was defined by tables in a PDF document (ug).
Copy-paste turned them into text and perl turned them into something
easily processed. Most of the code was generated by people who didn't
appear to mind writing tons of tedious boring repetitive code. There
was a bit-banging library that did most of the heavy lifting.

[toc] | [prev] | [next] | [standalone]

#26581

Nick Keighley <nick_keighley_nospam@hotmail.com> wrote:
> Eric Sosman <esos...@ieee-dot-org.invalid> wrote:
> 
> serializing structs
> 
>> One way is the function-per-struct-type approach, and
>> although it may be "b..o..r..i..n..g" it has advantages
>> that should not be dismissed lightly. ?Consider that there
>> are (most likely) only a handful of structs and hence only
>> a handful of functions; writing them won't take enough
>> time to b..o..r..e anyone except an ADHD sufferer.
> 
> I've dealt with cases where there were considerably more than a
> "handful"- communication protocol.
> 
> Since I have a low b..o..r..e..d..o..m threshold I resorted to code
> generation. The protocol was defined by tables in a PDF document (ug).
> Copy-paste turned them into text and perl turned them into something
> easily processed. Most of the code was generated by people who didn't
> appear to mind writing tons of tedious boring repetitive code. There
> was a bit-banging library that did most of the heavy lifting.

What do you think of my my smemf()-type of solution in other post?
Not necessarily that particular solution, but the point being that
this has got to be a pretty frequently occurring problem, and the
only available solutions, like yours above, seem to be pretty
awfully ugly. But it ain't rocket science -- there ought to be
a way to deal with these situations elegantly. smemf() solves it
with >>zero<< structs. Instead of a struct, you define the block
or packet format with a sprintf-like format string, and then just
   smemf(buffer_for_block, format_string_describing_block_layout,
   data_for_field_1, data_for_field_2, ...);
And I suppose there are other kinds of ways to deal with this
whole class of problems, which wouldn't exist at all if some
kind of packed structs were C standard. In any case, there
should exist some standard practice for dealing with it.
-- 
John Forkosh  ( mailto:  j@f.com  where j=john and f=forkosh )

[toc] | [prev] | [next] | [standalone]

#26585

On Sat, 2012-09-22, JohnF wrote:
> Nick Keighley <nick_keighley_nospam@hotmail.com> wrote:
>> Eric Sosman <esos...@ieee-dot-org.invalid> wrote:
>> 
>> serializing structs
>> 
>>> One way is the function-per-struct-type approach, and
>>> although it may be "b..o..r..i..n..g" it has advantages
>>> that should not be dismissed lightly. ?Consider that there
>>> are (most likely) only a handful of structs and hence only
>>> a handful of functions; writing them won't take enough
>>> time to b..o..r..e anyone except an ADHD sufferer.
>> 
>> I've dealt with cases where there were considerably more than a
>> "handful"- communication protocol.
>> 
>> Since I have a low b..o..r..e..d..o..m threshold I resorted to code
>> generation. The protocol was defined by tables in a PDF document (ug).
>> Copy-paste turned them into text and perl turned them into something
>> easily processed. Most of the code was generated by people who didn't
>> appear to mind writing tons of tedious boring repetitive code. There
>> was a bit-banging library that did most of the heavy lifting.
>
> What do you think of my my smemf()-type of solution in other post?
> Not necessarily that particular solution, but the point being that
> this has got to be a pretty frequently occurring problem, and the
> only available solutions, like yours above, seem to be pretty
> awfully ugly.

Didn't read very carefully, but it's like the Python 'struct' module,
isn't it?

  http://docs.python.org/library/struct.html

Well, that works well in Python and the only drawbacks I can see in C
are:
- you give up some type safety
- you give up some speed
- your output buffer must be big enough

I don't know why I've never seen this done in C or C++ before.
I'm happy with the b..o..r..i..n..g approach because I've never had
many such message formats (or at least they have had so much common
structure that it was manageable).

...
> And I suppose there are other kinds of ways to deal with this
> whole class of problems, which wouldn't exist at all if some
> kind of packed structs were C standard.

Packed structs would not solve the class of problems. You still have
endianness, you still have variable-sized message formats, and so on.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#26587

Jorgen Grahn <grahn+nntp@snipabacken.se> wrote:
> JohnF wrote:
>> Nick Keighley <nick_keighley_nospam@hotmail.com> wrote:
>>> Eric Sosman <esos...@ieee-dot-org.invalid> wrote:
>>> 
>>> serializing structs
>>> 
>>>> One way is the function-per-struct-type approach, and
>>>> although it may be "b..o..r..i..n..g" it has advantages
>>>> that should not be dismissed lightly. ?Consider that there
>>>> are (most likely) only a handful of structs and hence only
>>>> a handful of functions; writing them won't take enough
>>>> time to b..o..r..e anyone except an ADHD sufferer.
>>> 
>>> I've dealt with cases where there were considerably more than a
>>> "handful"- communication protocol.
>>> 
>>> Since I have a low b..o..r..e..d..o..m threshold I resorted to code
>>> generation. The protocol was defined by tables in a PDF document (ug).
>>> Copy-paste turned them into text and perl turned them into something
>>> easily processed. Most of the code was generated by people who didn't
>>> appear to mind writing tons of tedious boring repetitive code. There
>>> was a bit-banging library that did most of the heavy lifting.
>>
>> What do you think of my my smemf()-type of solution in other post?
>> Not necessarily that particular solution, but the point being that
>> this has got to be a pretty frequently occurring problem, and the
>> only available solutions, like yours above, seem to be pretty
>> awfully ugly.

> Didn't read very carefully, but it's like the Python 'struct' module,
> isn't it?
>     http://docs.python.org/library/struct.html

Thanks, Jorgen, that's >>excellent<<. Quick read suggests it's
exactly like my smemf(), or vice versa since theirs clearly
came first. I guess "great minds think alike". Of course,
I guess idiots probably think alike, too... Okay, I've made
my choice :)
     In any event, I'll read that much more carefully, and
incorporate their inevitable improvements into my "functional
spec", such as it is, and then see if it seems worth re-coding
any changes and finishing it.

> Well, that works well in Python and the only drawbacks I can see in C
> are:
> - you give up some type safety
> - your output buffer must be big enough

Yeah, yeah, programmers ought to be careful.
Not much different than sprintf()'s potential pitfalls.

> - you give up some speed

Hardly a likely problem. This is typically for i/o,
not some compute intensive loop. You'd have to be
formatting god-knows-how-many GB's before anybody
would notice the overhead.

> I don't know why I've never seen this done in C or C++ before.

Well, I'm inclined to re-do smemf()'s specs as close as possible
to python's, except that their format strings are more different
from C's than necessary. In any case, I'm sure their stuff deals
with problems that hadn't occurred to me yet. So it'll be a great
improvement to see precisely what they're doing.

> I'm happy with the b..o..r..i..n..g approach because I've never had
> many such message formats (or at least they have had so much common
> structure that it was manageable).
> ...
>> And I suppose there are other kinds of ways to deal with this
>> whole class of problems, which wouldn't exist at all if some
>> kind of packed structs were C standard.
> 
> Packed structs would not solve the class of problems. You still have
> endianness, you still have variable-sized message formats, and so on.
> /Jorgen

Yeah, I wrote that "packed structs solves all" sentence too quickly,
and it's wrong. smemf() and python pack(fmt,v1,v2,...) both already
handle endianness. And smemf() implements "%*x" to pick up variable
lengths from the arg list (my very quick glance at python doesn't
show that yet, but I'll bet it's there somewheres I'll find later).
You'd have to be more specific about "and so on", but that's also
on my to-do list. Thanks again for the pointer,
-- 
John Forkosh  ( mailto:  j@f.com  where j=john and f=forkosh )

[toc] | [prev] | [next] | [standalone]

#26591

On Sat, 2012-09-22, JohnF wrote:
> Jorgen Grahn <grahn+nntp@snipabacken.se> wrote:
...
>>     http://docs.python.org/library/struct.html

>> - you give up some speed
>
> Hardly a likely problem. This is typically for i/o,
> not some compute intensive loop. You'd have to be
> formatting god-knows-how-many GB's before anybody
> would notice the overhead.

Now that you mention it, yes, it's probably like that.  But I still
suspect that's part of the reason you don't seen that design in real
code.  It's a reflex among C programmers: you don't invent interpreted
mini-languages which require parsing at every call.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#26593

"JohnF" <john@please.see.sig.for.email.com> wrote in message 
news:k3klnr$lq7$1@reader1.panix.com...
> Jorgen Grahn <grahn+nntp@snipabacken.se> wrote:

>> Didn't read very carefully, but it's like the Python 'struct' module,
>> isn't it?
>>     http://docs.python.org/library/struct.html

> Well, I'm inclined to re-do smemf()'s specs as close as possible
> to python's, except that their format strings are more different
> from C's than necessary. In any case, I'm sure their stuff deals
> with problems that hadn't occurred to me yet. So it'll be a great
> improvement to see precisely what they're doing.

The main problem Python has to deal with is that the language doesn't have 
structs.

-- 
Bartc
  

[toc] | [prev] | [next] | [standalone]

#26592

Jorgen Grahn <grahn+nntp@snipabacken.se> writes:
[...]
> Didn't read very carefully, but it's like the Python 'struct' module,
> isn't it?
>
>   http://docs.python.org/library/struct.html

It's also reminiscent of Perl's built-in "pack" and "unpack" functions.

http://perldoc.perl.org/functions/pack.html
http://perldoc.perl.org/functions/unpack.html

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
    Will write code for food.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#26596

Keith Thompson <kst-u@mib.org> wrote:
> Jorgen Grahn <grahn+nntp@snipabacken.se> writes:
> [...]
>> Didn't read very carefully, but it's like the Python 'struct' module,
>> isn't it?
>>
>>  http://docs.python.org/library/struct.html
> 
> It's also reminiscent of Perl's built-in "pack" and "unpack" functions.
>   http://perldoc.perl.org/functions/pack.html
>   http://perldoc.perl.org/functions/unpack.html

Thanks, Keith (and Jorgen again), that's also incredibly useful.
Lots of food for thought to spec out a C variant. And yet another
format/template to peruse. I really should put some effort into
learning these "little languages".
   Despite BartC's remark, "The main problem Python has to deal
with is that the language doesn't have structs", I think this
kind of pack function has valuable uses in C (along with a
scan-like unpack, as suggested by BartC's other remark),
e.g., formatting (and reading) binary blocks/packets/whatever,
which is what brought the idea to my mind. And the fact that
all these little languages have pack/unpack just supports
the notion they're useful funcs. Moreover, I'm sure you can
see they're no great big deal to code. So why not do it
(I'm in the process, but that's no reason others shouldn't
do it differently/better/whatever)? The naysayers can just
ignore it. I'm sure everybody has their favorite C feature
they choose not to use.
   Finally, regarding Ian's remarks, pack/unpack would be
a >>lightweight<< alternative to accomplish this kind of
task. Ian's way involves importing additional tool dependencies,
possibly turning what could be just a few lines of code
into a subtask unto itself. For his large project case,
involving lots and lots of different block formats,
that may be the best approach (I've worked with Swift
at Bankers Trust, albeit a while back, and also with various
ticker feeds, etc, etc, and more etc). But that's no reason
not to have a lightweight alternative.
-- 
John Forkosh  ( mailto:  j@f.com  where j=john and f=forkosh )

[toc] | [prev] | [next] | [standalone]

#26598

On 09/23/12 12:19 PM, JohnF wrote:

>     Finally, regarding Ian's remarks, pack/unpack would be
> a>>lightweight<<  alternative to accomplish this kind of
> task. Ian's way involves importing additional tool dependencies,
> possibly turning what could be just a few lines of code
> into a subtask unto itself.

The up front effort is much the same.  In your case you have to code the 
pack and unpack functions, in mine I had to write a simple code 
generator.  In both cases, once it's done it's done.  While the code 
generator can be simple, mine has evolved to support enum types and 
inheritance, but its still only a couple of hundred lines of code.  A 
colleague on a previous project write all the transforms in XLST, but 
that made my head explode.

In coding, I add a table to an OpenOffice document (part of the project 
documentation, so it's dual use) while you write a format string. 
Probably similar effort although I'd say the table approach is less 
error prone.

At run time, I have optimised code to perform the packing and unpacking 
while your functions have to interpret the format string each time they 
are called.  If you have to handle a lot of traffic, especially on a low 
powered device, that can be a killer.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#26600

Ian Collins <ian-news@hotmail.com> wrote:
> JohnF wrote:
> 
>>     Finally, regarding Ian's remarks, pack/unpack would be
>> a lightweight alternative to accomplish this kind of
>> task. Ian's way involves importing additional tool dependencies,
>> possibly turning what could be just a few lines of code
>> into a subtask unto itself.
> 
> The up front effort is much the same. <<snip>>

Yeah, well, no biggie. There's no reason not to have
two or several different ways to approach the same problem,
and let the programmer decide what he prefers in a
particular situation. This isn't "either/or", it's "both".
-- 
John Forkosh  ( mailto:  j@f.com  where j=john and f=forkosh )

[toc] | [prev] | [next] | [standalone]

#26595

On 09/22/12 08:53 PM, JohnF wrote:
> Nick Keighley<nick_keighley_nospam@hotmail.com>  wrote:
>> Eric Sosman<esos...@ieee-dot-org.invalid>  wrote:
>>
>> serializing structs
>>
>>> One way is the function-per-struct-type approach, and
>>> although it may be "b..o..r..i..n..g" it has advantages
>>> that should not be dismissed lightly. ?Consider that there
>>> are (most likely) only a handful of structs and hence only
>>> a handful of functions; writing them won't take enough
>>> time to b..o..r..e anyone except an ADHD sufferer.
>>
>> I've dealt with cases where there were considerably more than a
>> "handful"- communication protocol.
>>
>> Since I have a low b..o..r..e..d..o..m threshold I resorted to code
>> generation. The protocol was defined by tables in a PDF document (ug).
>> Copy-paste turned them into text and perl turned them into something
>> easily processed. Most of the code was generated by people who didn't
>> appear to mind writing tons of tedious boring repetitive code. There
>> was a bit-banging library that did most of the heavy lifting.
>
> What do you think of my my smemf()-type of solution in other post?
> Not necessarily that particular solution, but the point being that
> this has got to be a pretty frequently occurring problem, and the
> only available solutions, like yours above, seem to be pretty
> awfully ugly. But it ain't rocket science -- there ought to be
> a way to deal with these situations elegantly. smemf() solves it
> with>>zero<<  structs.

having worked on numerous on the wire protocols, I've encountered this 
problem many times.  The most practical solution for all but the most 
trivial cases is to code generate the formatting code.  The source for 
the code generator can either be C (or C++ if inheritance helps) code or 
some other easy to parse format.  This is one case where I prefer XML 
(often in the form of an OpenOffice document) as the "other easy to 
parse format".

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#26605

On Sep 22, 11:33 pm, Ian Collins <ian-n...@hotmail.com> wrote:
> On 09/22/12 08:53 PM, JohnF wrote:
>
>
>
>
>
>
>
>
>
> > Nick Keighley<nick_keighley_nos...@hotmail.com>  wrote:
> >> Eric Sosman<esos...@ieee-dot-org.invalid>  wrote:
>
> >> serializing structs
>
> >>> One way is the function-per-struct-type approach, and
> >>> although it may be "b..o..r..i..n..g" it has advantages
> >>> that should not be dismissed lightly. ?Consider that there
> >>> are (most likely) only a handful of structs and hence only
> >>> a handful of functions; writing them won't take enough
> >>> time to b..o..r..e anyone except an ADHD sufferer.
>
> >> I've dealt with cases where there were considerably more than a
> >> "handful"- communication protocol.
>
> >> Since I have a low b..o..r..e..d..o..m threshold I resorted to code
> >> generation. The protocol was defined by tables in a PDF document (ug).
> >> Copy-paste turned them into text and perl turned them into something
> >> easily processed. Most of the code was generated by people who didn't
> >> appear to mind writing tons of tedious boring repetitive code. There
> >> was a bit-banging library that did most of the heavy lifting.
>
> > What do you think of my my smemf()-type of solution in other post?
> > Not necessarily that particular solution, but the point being that
> > this has got to be a pretty frequently occurring problem, and the
> > only available solutions, like yours above, seem to be pretty
> > awfully ugly. But it ain't rocket science -- there ought to be
> > a way to deal with these situations elegantly. smemf() solves it
> > with>>zero<<  structs.
>
> having worked on numerous on the wire protocols, I've encountered this
> problem many times.  The most practical solution for all but the most
> trivial cases is to code generate the formatting code.  The source for
> the code generator can either be C (or C++ if inheritance helps) code or
> some other easy to parse format.  This is one case where I prefer XML
> (often in the form of an OpenOffice document) as the "other easy to
> parse format".

there's ASN.1 too...

[toc] | [prev] | [next] | [standalone]

#26577

In article <k3j5or$q37$1@reader1.panix.com>,
 JohnF <john@please.see.sig.for.email.com> wrote:

> Any >>portable<< way to accomplish that in c?
> Don't want to use __attribute__((__packed__))
> or #pragma pack, etc, nor #ifdef's to choose
> among whatever alternatives I happen to know
> about. It's a requirement that the code remain
> portable.

Use a byte array instead of a struct and functions to get and put bits
    typedef struct {unsigned char array[n];} Struct;
    
    inline int getsigned(Struct *a, int off, int w) {
        int byte = off/CHAR_BIT;
        int bit = off%CHAR_BIT;
        int value; memcpy(&value, a->array+byte, sizeof(int));
        return (value<<bit)>>(sizeof(int)*CHAR_BIT-w);
    }
    ...

And then define each field
    #define alpha(a)        getsigned(&a, 1, 1)
    #define alphaput(a,v)   putsigned(&a, 1, 1, v)
    #define beta(a)         getsigned(&a, 2, 2)
    #define gamma(a)        getsigned(&a, 4, 2)

You can also use a preprocessor, such as m4,
    packedstructure(Struct,

        packedfield(    ,   1)
        packedfield(alpha,  1)
        packedfield(beta,   2)
        packedfield(gamma,  2)
    )
which expands to the definition of Struct and the fields.

-- 
My name Indigo Montoya.  \\        Annoying Usenet one post at a time.
You flamed my father.     \'         At least I can stay in character.
Prepare to be spanked.   //               When you look into the void,
Stop posting that!      `/  the void looks into you, and fulfills you.

[toc] | [prev] | [next] | [standalone]

#27250

On Friday, September 21, 2012 9:54:36 PM UTC-4, JohnF wrote:
> Any >>portable<< way to accomplish that in c?
> 
> Don't want to use __attribute__((__packed__))
> 
> or #pragma pack, etc, nor #ifdef's to choose
> 
> among whatever alternatives I happen to know
> 
> about. It's a requirement that the code remain
> 
> portable.
> 
> 
> 
> In particular, I'm trying to write blocks that
> 
> conform to a binary file format (gif), and can
> 
> set up structs for them easily enough, but can't
> 
> fwrite(blockstruct,sizeof(blockstruct),1,fileptr),
> 
> or the like, due to blockstruct's inevitable
> 
> padding (which indeed occurs for gif format blocks).
> 
> 
> 
> At the moment, I just have a different func for
> 
> each block type that writes out the members of that
> 
> particular struct individually... b..o..r..i..n..g.
> 
> A generalization of that idea (if portable packing's
> 
> not possible) would also be fine: >>if<< there's some
> 
> way to reference the members of a struct, passed as
> 
> an argument but of unknown (to the func) type, in a
> 
> loop, i.e., for(i=0;i<nmembers;i++)thisstruct->member[i].
> 
> Then I could offsetof() and sizeof() each member, and
> 
> write it out, so just one (much less boring) func
> 
> could handle all the different block type structs.
> 
> 
> 
> But, afaik, I don't think that thisstruct->member[i]
> 
> thing is possible, nor portable packing. So is there
> 
> any "one size fits all" way to handle this problem?
> 
> I'm sure people must come across it frequently enough
> 
> that it's been thought about, and the best possible
> 
> approach (among, possibly, several bad alternatives)
> 
> has been identified.
> 
> -- 
> 
> John Forkosh  ( mailto:  j@f.com  where j=john and f=forkosh )

typedef unsigned char P2b[2];

#define SETP2(PTR, VAL) \
((PTR)[0] = (VAL) >> CHAR_BIT, (PTR)[1] = (VAL) & ((1 << CHAR_BIT) - 1))

#define GETP2(PTR) \
(((PTR)[0] << CHAR_BIT) | (PTR)[0])

Larger "packable ints" are analogous.  My choice of big-endian here is arbitrary.

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.c

csiph-web