Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #30023 > unrolled thread

Data alignment questin, structures

Back to article view | Back to comp.lang.c

Contents

  Data alignment questin, structures mathog <mathog@caltech.edu> - 2013-01-12 10:15 -0800
    Re: Data alignment questin, structures Ben Bacarisse <ben.usenet@bsb.me.uk> - 2013-01-12 18:54 +0000
      Re: Data alignment questin, structures Jorgen Grahn <grahn+nntp@snipabacken.se> - 2013-01-15 20:14 +0000
        Re: Data alignment questin, structures Eric Sosman <esosman@comcast-dot-net.invalid> - 2013-01-15 16:29 -0500
    Re: Data alignment questin, structures Eric Sosman <esosman@comcast-dot-net.invalid> - 2013-01-12 15:41 -0500
      Re: Data alignment questin, structures mathog <dmathog@gmail.com> - 2013-01-18 10:24 -0800
        Re: Data alignment questin, structures James Kuyper <jameskuyper@verizon.net> - 2013-01-18 14:05 -0500
        Re: Data alignment questin, structures Eric Sosman <esosman@comcast-dot-net.invalid> - 2013-01-18 14:46 -0500
        Re: Data alignment questin, structures glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-01-18 21:11 +0000
        Re: Data alignment questin, structures Shao Miller <sha0.miller@gmail.com> - 2013-01-18 17:14 -0500
          Re: Data alignment questin, structures glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-01-18 23:09 +0000
        Re: Data alignment questin, structures Keith Thompson <kst-u@mib.org> - 2013-01-18 16:23 -0800
        Re: Data alignment questin, structures mathog <mathog@caltech.edu> - 2013-01-19 21:02 -0800
          Re: Data alignment questin, structures Shao Miller <sha0.miller@gmail.com> - 2013-01-20 01:25 -0500
          Re: Data alignment questin, structures glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-01-20 07:53 +0000
          Re: Data alignment questin, structures Ben Bacarisse <ben.usenet@bsb.me.uk> - 2013-01-20 14:02 +0000
          Re: Data alignment questin, structures Keith Thompson <kst-u@mib.org> - 2013-01-20 07:35 -0800
            Re: Data alignment questin, structures 88888 Dihedral <dihedral88888@googlemail.com> - 2013-01-20 11:49 -0800
          Re: Data alignment questin, structures James Kuyper <jameskuyper@verizon.net> - 2013-01-20 12:30 -0500
            Re: Data alignment questin, structures mathog <mathog@caltech.edu> - 2013-01-20 10:26 -0800
              Re: Data alignment questin, structures Shao Miller <sha0.miller@gmail.com> - 2013-01-20 15:11 -0500
              Re: Data alignment questin, structures "BartC" <bc@freeuk.com> - 2013-01-20 20:34 +0000
              Re: Data alignment questin, structures James Kuyper <jameskuyper@verizon.net> - 2013-01-20 15:53 -0500
                Re: Data alignment questin, structures 88888 Dihedral <dihedral88888@googlemail.com> - 2013-01-20 13:16 -0800
                Re: Data alignment questin, structures mathog <mathog@caltech.edu> - 2013-01-20 20:27 -0800
                  Re: Data alignment questin, structures Shao Miller <sha0.miller@gmail.com> - 2013-01-21 00:37 -0500
                    Re: Data alignment questin, structures Dr Nick <nospam-4@temporary-address.org.uk> - 2013-01-21 10:07 +0000
                      Re: Data alignment questin, structures mathog <mathog@caltech.edu> - 2013-01-21 10:21 -0800
                        Re: Data alignment questin, structures Tim Rentsch <txr@alumni.caltech.edu> - 2013-01-21 10:34 -0800
                        Re: Data alignment questin, structures Keith Thompson <kst-u@mib.org> - 2013-01-21 13:07 -0800
                          Re: Data alignment questin, structures "BartC" <bc@freeuk.com> - 2013-01-21 22:24 +0000
                          Re: Data alignment questin, structures mathog <dmathog@gmail.com> - 2013-01-22 10:18 -0800
                            Re: Data alignment questin, structures glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-01-22 20:57 +0000
                            Re: Data alignment questin, structures Keith Thompson <kst-u@mib.org> - 2013-01-22 14:51 -0800
                        Re: Data alignment questin, structures Ian Collins <ian-news@hotmail.com> - 2013-01-22 11:12 +1300
                      Re: Data alignment questin, structures glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-01-21 20:53 +0000
                  Re: Data alignment questin, structures Keith Thompson <kst-u@mib.org> - 2013-01-20 22:06 -0800
                    Re: Data alignment questin, structures Philip Lantz <prl@canterey.us> - 2013-01-22 00:27 -0800
                      Re: Data alignment questin, structures James Kuyper <jameskuyper@verizon.net> - 2013-01-22 06:44 -0500
                  Re: Data alignment questin, structures Bart van Ingen Schenau <bart@ingen.ddns.info.invalid> - 2013-01-21 10:19 +0000
              Re: Data alignment questin, structures glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-01-20 22:19 +0000
            Re: Data alignment questin, structures glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-01-20 22:14 +0000
            Re: Data alignment questin, structures "BartC" <bc@freeuk.com> - 2013-01-20 22:48 +0000
              Re: Data alignment questin, structures Les Cargill <lcargill99@comcast.com> - 2013-01-20 18:11 -0600
              Re: Data alignment questin, structures Ian Collins <ian-news@hotmail.com> - 2013-01-21 14:07 +1300
    Re: Data alignment questin, structures Shao Miller <sha0.miller@gmail.com> - 2013-01-12 17:45 -0500
      Re: Data alignment questin, structures mathog <mathog@caltech.edu> - 2013-01-13 10:17 -0800
        Re: Data alignment questin, structures Shao Miller <sha0.miller@gmail.com> - 2013-01-13 17:16 -0500

Page 3 of 3 — ← Prev page 1 2 [3]

#30419

mathog <mathog@caltech.edu> wrote:

(snip)
> which tells the compiler that it needs to know how to copy data in 
> either direction between a specific memstruct and a specific struct. 
> (And that would be illegal if the fields didn't match exactly.)
 
> I am not worried about endian issues or other floating point 
> representations.  Both of these can be dealt with easily enough once the 
> data has been mapped from nonaligned to aligned data structures.  As for 
> bytes that are not 8 bits, does any modern hardware exist where that is 
> not true?  Must we keep the requirement to run on archaic hardware 
> forever going forward with newer versions of the language?

Well, in C a byte is CHAR_BIT bits long, and there are systems where
that isn't 8, but still usually a multiple of 8. 

The 36 and 60 bit machines are pretty much gone.

-- glen

[toc] | [prev] | [next] | [standalone]

#30418

James Kuyper <jameskuyper@verizon.net> wrote:

(snip, someone wrote)
>> it seems not to be safe to pass a "Mystruct4b *" pointer to a function 
>> which references this data at an arbitrary location in memory.  Instead 
>> the only safe method is to pass "char *" pointers and take the data 
>> apart with memcpy() at a very low level, moving it from memory to the 
>> structure, or vice versa.
 
> Correct. Because of these things that are not guaranteed, C structs
> should be considered as suitable only for use for in-memory objects, not
> for reading or writing from files.

Well, you can write them to a file and read them back in the same
program. Also, another program compiled by the same compiler and
run on a similar system should also work.

Padding and alignment can change between versions of a compiler 
on the same system, though usually don't. 

-- glen

[toc] | [prev] | [next] | [standalone]

#30420

"James Kuyper" <jameskuyper@verizon.net> wrote in message 
news:kdh9it$kjm$1@dont-email.me...

> Correct. Because of these things that are not guaranteed, C structs
> should be considered as suitable only for use for in-memory objects, not
> for reading or writing from files.

I've spent quite a few years having to interface with innumerable C 
structures, from another language.

That usually involved the other language being able to exactly match the 
types, sizes, alignments and padding used by a specific C implementation, 
when defining its own structs.

But C itself doesn't seem to have a need to do the same! Yet there are 
plenty of situations where this would be useful (to specify exactly, at 
compile time, how a struct is laid out).

-- 
Bartc 

[toc] | [prev] | [next] | [standalone]

#30424

BartC wrote:
> "James Kuyper" <jameskuyper@verizon.net> wrote in message
> news:kdh9it$kjm$1@dont-email.me...
>
>> Correct. Because of these things that are not guaranteed, C structs
>> should be considered as suitable only for use for in-memory objects, not
>> for reading or writing from files.
>
> I've spent quite a few years having to interface with innumerable C
> structures, from another language.
>
> That usually involved the other language being able to exactly match the
> types, sizes, alignments and padding used by a specific C
> implementation, when defining its own structs.
>
> But C itself doesn't seem to have a need to do the same! Yet there are
> plenty of situations where this would be useful (to specify exactly, at
> compile time, how a struct is laid out).
>


Between #pragma pack() ( where available)  and use of filler, you can
get quite close. 'Course there are still endian issues, etc.

--
Les Cargill

[toc] | [prev] | [next] | [standalone]

#30427

BartC wrote:
> "James Kuyper" <jameskuyper@verizon.net> wrote in message
> news:kdh9it$kjm$1@dont-email.me...
>
>> Correct. Because of these things that are not guaranteed, C structs
>> should be considered as suitable only for use for in-memory objects, not
>> for reading or writing from files.
>
> I've spent quite a few years having to interface with innumerable C
> structures, from another language.
>
> That usually involved the other language being able to exactly match the
> types, sizes, alignments and padding used by a specific C implementation,
> when defining its own structs.
>
> But C itself doesn't seem to have a need to do the same! Yet there are
> plenty of situations where this would be useful (to specify exactly, at
> compile time, how a struct is laid out).

More often than not, where that requirement exists, the means for 
meeting are provided.  I'm thinking primarily of embedded compilers 
here.  I've written many a driver where structs overlay sets of registers.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#30036

On 1/12/2013 13:15, mathog wrote:
> (Apologies if this is a duplicate post, glitch on first attempt.)
>
> Consider the following struct:
>
> typedef struct {
>    uint16_t one;
>    uint16_t twothree[2];
>    uint16_t four;
>    uint16_t five;
> } Mystruct;
>
> It contains data storage which is actually:
>
>    { uint16_t, uint32_t, uint16_t, uint16_t  }
>
> Accessing 'twothree" as a uint32_t directly, for instance with:
>
>    Mystruct instance;
>    printf("twothree is:%d\n",*(uint32_t *)&(instance.twothree[0]);
>
> will be a nonaligned memory access.  That will do nothing untoward on
> x86 CPUs but will blow up on many others.  OK so far?
>
> Now, what happens when this struct is embedded in a binary file stored
> in a character array "buffer[]" such that its position "i" is not
> guaranteed to be aligned on the same boundary that
>
>    Mystract *instance2 = malloc(sizeof(Mystruct));
>
> would have used, but will always be on a multiple of 2.
>
> 1.  Will this always work?
>
>    function1((Mystruct *) &(buffer[i]));
>
> 2.  How about this?
>
>    function2(*(Mystruct *) &(buffer[i]));
>
> Assume that within the function access to the data is by memcpy() to the
> appropriate offset for the uint32_t and by "=" for the uint16_t fields.
>   My guess is that (2) is likely to blow up more often than not, since
> it is trying to pass a Mystruct unaligned.
>
> What does the language standard say should happen in these two cases?

If you have implementation-specific knowledge of what will happen, then 
I'd suggest that you do whatever you want to accomplish your goal.

If you actually care about portability (you seem to), I'd suggest 
formally serializing and deserializing data to and from the file.

You might be interested in modern C, which is C11.  It includes the 
'_Alignof' and '_Alignas' keywords, which allow you to consider and 
choose alignments, respectively.

Prior to C11, you could try to use something like:

   #include <stddef.h> /* For 'offsetof' */
   #define Alignof(type) (offsetof(struct { char c; type t; }, t))

to consider alignment, and could use unions to choose alignment, if you 
know the complete object type whose alignment needs to be satsified:

   union {
       char buffer[512];
       MyStruct as_mystruct;
     } buffer_and_mystruct;

Here, you know that 'buffer_and_mystruct.buffer' will be properly 
aligned for an instance of your structure type.

You can always use 'memcpy' to copy some bytes into a member of an 
instance of your structure type; then it'll be safe to examine that 
member, as long as it hasn't been populated with a trap representation.

-- 
- Shao Miller
--
"Thank you for the kind words; those are the kind of words I like to hear.

Cheerily,"  -- Richard Harter

[toc] | [prev] | [next] | [standalone]

#30070

Shao Miller wrote:

> If you have implementation-specific knowledge of what will happen, then
> I'd suggest that you do whatever you want to accomplish your goal.

That's the point - I want this code to work wherever, without having 
access to all possible platforms to test it first.  Implicitly 
implementation specific is exactly what I am trying to avoid.

>
> If you actually care about portability (you seem to), I'd suggest
> formally serializing and deserializing data to and from the file.

Yeah, I was afraid that was going to be the only standard compliant way 
of doing this.  The key thing that I have learned is that passing a 
pointer to an unaligned structure isn't portable (6.3.2.3p7 cited by 
Eric Sosnan earlier in this thread).  There are some instances of that 
in the code that swaps bytes (big endian to little endian), so those 
routines will need to have their interfaces adjusted.  They are 
otherwise "serial".  Better to do this now, before the code is released, 
than later.  It also seems unavoidable that explicit recordtype_get() 
functions will be needed, with the data passed by void * or char *.

> You might be interested in modern C, which is C11.  It includes the
> '_Alignof' and '_Alignas' keywords, which allow you to consider and
> choose alignments, respectively.

This doesn't help with the struct offset problem, since we do not know 
the offset ahead of time.  In a particular file the struct could be 
anywhere in the buffer, at 100,102,..,500,502,etc.  Or am I 
misunderstanding the purpose of the _Alignof and _Alignas?

Thanks,

David Mathog

[toc] | [prev] | [next] | [standalone]

#30084

On 1/13/2013 13:17, mathog wrote:
> Shao Miller wrote:
>
>> If you have implementation-specific knowledge of what will happen, then
>> I'd suggest that you do whatever you want to accomplish your goal.
>
> That's the point - I want this code to work wherever, without having
> access to all possible platforms to test it first.  Implicitly
> implementation specific is exactly what I am trying to avoid.
>
>>
>> If you actually care about portability (you seem to), I'd suggest
>> formally serializing and deserializing data to and from the file.
>
> Yeah, I was afraid that was going to be the only standard compliant way
> of doing this.  The key thing that I have learned is that passing a
> pointer to an unaligned structure isn't portable (6.3.2.3p7 cited by
> Eric Sosnan

(Sosman)

> earlier in this thread).  There are some instances of that
> in the code that swaps bytes (big endian to little endian), so those
> routines will need to have their interfaces adjusted.  They are
> otherwise "serial".  Better to do this now, before the code is released,
> than later.

Based on what you've typed, I am guessing that the file format is not 
intended to be portable across different platforms.  Is that right? 
That is, copying a saved file that was made on an x86 isn't expected to 
load properly in the same program on PowerPC?  I ask because the padding 
within structures might also be a concern, for you.

By formally serializing/deserializing, you can have a portable file format.

> It also seems unavoidable that explicit recordtype_get()
> functions will be needed, with the data passed by void * or char *.
>
>> You might be interested in modern C, which is C11.  It includes the
>> '_Alignof' and '_Alignas' keywords, which allow you to consider and
>> choose alignments, respectively.
>
> This doesn't help with the struct offset problem, since we do not know
> the offset ahead of time.  In a particular file the struct could be
> anywhere in the buffer, at 100,102,..,500,502,etc.  Or am I
> misunderstanding the purpose of the _Alignof and _Alignas?

Ah, no.  If the structure can appear at any offset in the buffer, then 
these don't help you as much, unless the offset happens to be a multiple 
of the alignment requirement.  But as mentioned, you can 'memcpy' from 
the offset into an aligned location, which you might be interested in 
doing anyway, if a structure can be truncated by being near the end of 
your buffer.

At least you can automate some of the serializing/deserializing...  You 
can have:

   struct s_foo {
       double d;
       int i;
       char c;
     };

   size_t foo_offsets[] = {
       offsetof(struct s_foo, d),
       offsetof(struct s_foo, i),
       offsetof(struct s_foo, c),
     };

   f_serialize * foo_serials[] = {
       serialize_double,
       serialize_int,
       serialize_char,
     };

where 'f_serialize' is a function typedef for a serialization function. 
  Then you can iterate through these arrays with a loop in a generic 
"structure serialization" function, instead of having a monolithic:

   void serialize_foo(struct s_foo * foo) {
       serialize_double(&foo->d);
       serialize_int(&foo->i);
       serialize_char(&foo->c);
     }

for each of your different structure types.

Or, here is one library you might be interested in:

   http://www.leonerd.org.uk/code/libpack/intro.html

-- 
- Shao Miller
--
"Thank you for the kind words; those are the kind of words I like to hear.

Cheerily,"  -- Richard Harter

[toc] | [prev] | [standalone]

Page 3 of 3 — ← Prev page 1 2 [3]

Back to top | Article view | comp.lang.c

csiph-web