Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #30023 > unrolled thread

Data alignment questin, structures

Back to article view | Back to comp.lang.c

Contents

  Data alignment questin, structures mathog <mathog@caltech.edu> - 2013-01-12 10:15 -0800
    Re: Data alignment questin, structures Ben Bacarisse <ben.usenet@bsb.me.uk> - 2013-01-12 18:54 +0000
      Re: Data alignment questin, structures Jorgen Grahn <grahn+nntp@snipabacken.se> - 2013-01-15 20:14 +0000
        Re: Data alignment questin, structures Eric Sosman <esosman@comcast-dot-net.invalid> - 2013-01-15 16:29 -0500
    Re: Data alignment questin, structures Eric Sosman <esosman@comcast-dot-net.invalid> - 2013-01-12 15:41 -0500
      Re: Data alignment questin, structures mathog <dmathog@gmail.com> - 2013-01-18 10:24 -0800
        Re: Data alignment questin, structures James Kuyper <jameskuyper@verizon.net> - 2013-01-18 14:05 -0500
        Re: Data alignment questin, structures Eric Sosman <esosman@comcast-dot-net.invalid> - 2013-01-18 14:46 -0500
        Re: Data alignment questin, structures glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-01-18 21:11 +0000
        Re: Data alignment questin, structures Shao Miller <sha0.miller@gmail.com> - 2013-01-18 17:14 -0500
          Re: Data alignment questin, structures glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-01-18 23:09 +0000
        Re: Data alignment questin, structures Keith Thompson <kst-u@mib.org> - 2013-01-18 16:23 -0800
        Re: Data alignment questin, structures mathog <mathog@caltech.edu> - 2013-01-19 21:02 -0800
          Re: Data alignment questin, structures Shao Miller <sha0.miller@gmail.com> - 2013-01-20 01:25 -0500
          Re: Data alignment questin, structures glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-01-20 07:53 +0000
          Re: Data alignment questin, structures Ben Bacarisse <ben.usenet@bsb.me.uk> - 2013-01-20 14:02 +0000
          Re: Data alignment questin, structures Keith Thompson <kst-u@mib.org> - 2013-01-20 07:35 -0800
            Re: Data alignment questin, structures 88888 Dihedral <dihedral88888@googlemail.com> - 2013-01-20 11:49 -0800
          Re: Data alignment questin, structures James Kuyper <jameskuyper@verizon.net> - 2013-01-20 12:30 -0500
            Re: Data alignment questin, structures mathog <mathog@caltech.edu> - 2013-01-20 10:26 -0800
              Re: Data alignment questin, structures Shao Miller <sha0.miller@gmail.com> - 2013-01-20 15:11 -0500
              Re: Data alignment questin, structures "BartC" <bc@freeuk.com> - 2013-01-20 20:34 +0000
              Re: Data alignment questin, structures James Kuyper <jameskuyper@verizon.net> - 2013-01-20 15:53 -0500
                Re: Data alignment questin, structures 88888 Dihedral <dihedral88888@googlemail.com> - 2013-01-20 13:16 -0800
                Re: Data alignment questin, structures mathog <mathog@caltech.edu> - 2013-01-20 20:27 -0800
                  Re: Data alignment questin, structures Shao Miller <sha0.miller@gmail.com> - 2013-01-21 00:37 -0500
                    Re: Data alignment questin, structures Dr Nick <nospam-4@temporary-address.org.uk> - 2013-01-21 10:07 +0000
                      Re: Data alignment questin, structures mathog <mathog@caltech.edu> - 2013-01-21 10:21 -0800
                        Re: Data alignment questin, structures Tim Rentsch <txr@alumni.caltech.edu> - 2013-01-21 10:34 -0800
                        Re: Data alignment questin, structures Keith Thompson <kst-u@mib.org> - 2013-01-21 13:07 -0800
                          Re: Data alignment questin, structures "BartC" <bc@freeuk.com> - 2013-01-21 22:24 +0000
                          Re: Data alignment questin, structures mathog <dmathog@gmail.com> - 2013-01-22 10:18 -0800
                            Re: Data alignment questin, structures glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-01-22 20:57 +0000
                            Re: Data alignment questin, structures Keith Thompson <kst-u@mib.org> - 2013-01-22 14:51 -0800
                        Re: Data alignment questin, structures Ian Collins <ian-news@hotmail.com> - 2013-01-22 11:12 +1300
                      Re: Data alignment questin, structures glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-01-21 20:53 +0000
                  Re: Data alignment questin, structures Keith Thompson <kst-u@mib.org> - 2013-01-20 22:06 -0800
                    Re: Data alignment questin, structures Philip Lantz <prl@canterey.us> - 2013-01-22 00:27 -0800
                      Re: Data alignment questin, structures James Kuyper <jameskuyper@verizon.net> - 2013-01-22 06:44 -0500
                  Re: Data alignment questin, structures Bart van Ingen Schenau <bart@ingen.ddns.info.invalid> - 2013-01-21 10:19 +0000
              Re: Data alignment questin, structures glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-01-20 22:19 +0000
            Re: Data alignment questin, structures glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-01-20 22:14 +0000
            Re: Data alignment questin, structures "BartC" <bc@freeuk.com> - 2013-01-20 22:48 +0000
              Re: Data alignment questin, structures Les Cargill <lcargill99@comcast.com> - 2013-01-20 18:11 -0600
              Re: Data alignment questin, structures Ian Collins <ian-news@hotmail.com> - 2013-01-21 14:07 +1300
    Re: Data alignment questin, structures Shao Miller <sha0.miller@gmail.com> - 2013-01-12 17:45 -0500
      Re: Data alignment questin, structures mathog <mathog@caltech.edu> - 2013-01-13 10:17 -0800
        Re: Data alignment questin, structures Shao Miller <sha0.miller@gmail.com> - 2013-01-13 17:16 -0500

Page 1 of 3  [1] 2 3  Next page →

#30023 — Data alignment questin, structures

(Apologies if this is a duplicate post, glitch on first attempt.)

Consider the following struct:

typedef struct {
   uint16_t one;
   uint16_t twothree[2];
   uint16_t four;
   uint16_t five;
} Mystruct;

It contains data storage which is actually:

   { uint16_t, uint32_t, uint16_t, uint16_t  }

Accessing 'twothree" as a uint32_t directly, for instance with:

   Mystruct instance;
   printf("twothree is:%d\n",*(uint32_t *)&(instance.twothree[0]);

will be a nonaligned memory access.  That will do nothing untoward on 
x86 CPUs but will blow up on many others.  OK so far?

Now, what happens when this struct is embedded in a binary file stored 
in a character array "buffer[]" such that its position "i" is not 
guaranteed to be aligned on the same boundary that

   Mystract *instance2 = malloc(sizeof(Mystruct));

would have used, but will always be on a multiple of 2.

1.  Will this always work?

   function1((Mystruct *) &(buffer[i]));

2.  How about this?

   function2(*(Mystruct *) &(buffer[i]));

Assume that within the function access to the data is by memcpy() to the 
appropriate offset for the uint32_t and by "=" for the uint16_t fields. 
  My guess is that (2) is likely to blow up more often than not, since 
it is trying to pass a Mystruct unaligned.

What does the language standard say should happen in these two cases?

Thanks,

David Mathog

[toc] | [next] | [standalone]

#30026

mathog <mathog@caltech.edu> writes:

> (Apologies if this is a duplicate post, glitch on first attempt.)
>
> Consider the following struct:
>
> typedef struct {
>   uint16_t one;
>   uint16_t twothree[2];
>   uint16_t four;
>   uint16_t five;
> } Mystruct;
>
> It contains data storage which is actually:
>
>   { uint16_t, uint32_t, uint16_t, uint16_t  }
>
> Accessing 'twothree" as a uint32_t directly, for instance with:
>
>   Mystruct instance;
>   printf("twothree is:%d\n",*(uint32_t *)&(instance.twothree[0]);
>
> will be a nonaligned memory access.  That will do nothing untoward on
> x86 CPUs but will blow up on many others.  OK so far?
>
> Now, what happens when this struct is embedded in a binary file stored
> in a character array "buffer[]" such that its position "i" is not
> guaranteed to be aligned on the same boundary that
>
>   Mystract *instance2 = malloc(sizeof(Mystruct));
>
> would have used, but will always be on a multiple of 2.
>
> 1.  Will this always work?
>
>   function1((Mystruct *) &(buffer[i]));
>
> 2.  How about this?
>
>   function2(*(Mystruct *) &(buffer[i]));
>
> Assume that within the function access to the data is by memcpy() to
> the appropriate offset for the uint32_t and by "=" for the uint16_t
> fields. My guess is that (2) is likely to blow up more often than not,
> since it is trying to pass a Mystruct unaligned.
>
> What does the language standard say should happen in these two cases?

I hope you'll permit a meta-answer.  Having pulled my hair out porting
yards of code like this in the 80s, I am almost certain that there is a
better way to do whatever you are trying to do.  More often than not,
the best way is though a small set of function that extract basic types
from a buffer regardless of alignment and byte order.  Whether you then
choose to use the values directly or to put them into a struct -- whose
members can now be aligned any way the compiler chooses (because you
never play address tricks with them) -- depends on the rest of the code.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#30161

On Sat, 2013-01-12, Ben Bacarisse wrote:
...
> I hope you'll permit a meta-answer.  Having pulled my hair out porting
> yards of code like this in the 80s, I am almost certain that there is a
> better way to do whatever you are trying to do.  More often than not,
> the best way is though a small set of function that extract basic types
> from a buffer regardless of alignment and byte order.  Whether you then
> choose to use the values directly or to put them into a struct -- whose
> members can now be aligned any way the compiler chooses (because you
> never play address tricks with them) -- depends on the rest of the code.

AOL, except in my case "porting yards of code like this in the 80s"
would be "being bogged down in code like this on and off 1996--present".

Everything just becomes so much simpler and safer if you treat octet
buffers as octet buffers, and structs as structs.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#30166

On 1/15/2013 3:14 PM, Jorgen Grahn wrote:
> On Sat, 2013-01-12, Ben Bacarisse wrote:
> ...
>> I hope you'll permit a meta-answer.  Having pulled my hair out porting
>> yards of code like this in the 80s, I am almost certain that there is a
>> better way to do whatever you are trying to do.  More often than not,
>> the best way is though a small set of function that extract basic types
>> from a buffer regardless of alignment and byte order.  Whether you then
>> choose to use the values directly or to put them into a struct -- whose
>> members can now be aligned any way the compiler chooses (because you
>> never play address tricks with them) -- depends on the rest of the code.
>
> AOL, except in my case "porting yards of code like this in the 80s"
> would be "being bogged down in code like this on and off 1996--present".
>
> Everything just becomes so much simpler and safer if you treat octet
> buffers as octet buffers, and structs as structs.

     Let's add "powerful" and "flexible" to the "so much more" list.

     Powerful: It's quite common to want to read and write not just
a struct, but a data structure.  As a simple example, consider
`struct person { char *name; struct person *spouse; }': It will
do you no good at all to send these two pointers to another program.
You need to read and write this data with functions that are aware
of the semantics.  (Also, they can be smart enough to handle "I'm
writing John; John's spouse is Mary so I'll also write Mary; Mary's
spouse is John so I'll also write John; John's spouse is Mary so ...")

     Flexible: So you've settled on a serialized form (in the old
days we had "wire formats"), and along comes someone braying "Ad-hoc
binary formats are s-o-o-o twentieth-century!  Management orders
you to get rid of all that well-tested, highly reliable, blazingly
efficient cruft, and use this shiny new XML DTD instead.  Hop to it!"
If your programs are full of fread() and fwrite() calls you've got a
headache; if they already use functions that separate internal and
external representations you'll have a much easier time.

-- 
Eric Sosman
esosman@comcast-dot-net.invalid

[toc] | [prev] | [next] | [standalone]

#30029

On 1/12/2013 1:15 PM, mathog wrote:
> (Apologies if this is a duplicate post, glitch on first attempt.)
>
> Consider the following struct:
>
> typedef struct {
>    uint16_t one;
>    uint16_t twothree[2];
>    uint16_t four;
>    uint16_t five;
> } Mystruct;
>
> It contains data storage which is actually:
>
>    { uint16_t, uint32_t, uint16_t, uint16_t  }
>
> Accessing 'twothree" as a uint32_t directly, for instance with:
>
>    Mystruct instance;
>    printf("twothree is:%d\n",*(uint32_t *)&(instance.twothree[0]);
>
> will be a nonaligned memory access.  That will do nothing untoward on
> x86 CPUs but will blow up on many others.  OK so far?

     Mostly.  The nature of the "blow up" varies, though: Some
systems ignore offending low-order address bits, and would
print a mixture of instance.one and instance.twothree[0].
(There's also the possibility of padding bytes, although the
possibility appears remote for this particular struct.)

> Now, what happens when this struct is embedded in a binary file stored
> in a character array "buffer[]" such that its position "i" is not
> guaranteed to be aligned on the same boundary that
>
>    Mystract *instance2 = malloc(sizeof(Mystruct));
>
> would have used, but will always be on a multiple of 2.
>
> 1.  Will this always work?
>
>    function1((Mystruct *) &(buffer[i]));

     Not "always," no.  6.3.2.3p7: "A pointer to an object type
may be converted to a pointer to a different object type. If the
resulting pointer is not correctly aligned for the referenced
type, the behavior is undefined. [...]"  There's no guarantee
that the alignment of `buffer' suffices for `Mystruct'.

> 2.  How about this?
>
>    function2(*(Mystruct *) &(buffer[i]));

     Not "always;" same reasoning (with even more force this time,
since the potentially misaligned pointer is not only computed,
but also dereferenced).

> Assume that within the function access to the data is by memcpy() to the
> appropriate offset for the uint32_t and by "=" for the uint16_t fields.
>   My guess is that (2) is likely to blow up more often than not, since
> it is trying to pass a Mystruct unaligned.
>
> What does the language standard say should happen in these two cases?

     "The behavior is undefined."

     What problem are you trying to solve?  As Ben Bacarisse says,
safer and saner approaches are likely to exist.

-- 
Eric Sosman
esosman@comcast-dot-net.invalid

[toc] | [prev] | [next] | [standalone]

#30312

Eric Sosman wrote:
>> Now, what happens when this struct is embedded in a binary file stored
>> in a character array "buffer[]" such that its position "i" is not
>> guaranteed to be aligned on the same boundary that
>>
>>    Mystract *instance2 = malloc(sizeof(Mystruct));
>>
>> would have used, but will always be on a multiple of 2.
>>
>> 1.  Will this always work?
>>
>>    function1((Mystruct *) &(buffer[i]));
>
>      Not "always," no.  6.3.2.3p7: "A pointer to an object type
> may be converted to a pointer to a different object type. If the
> resulting pointer is not correctly aligned for the referenced
> type, the behavior is undefined. [...]"  There's no guarantee
> that the alignment of `buffer' suffices for `Mystruct'.

On further consideration, how does one know what "correctly aligned" 
means for a given struct?  Consider these two examples:

typedef struct {
    uint16_t one;
    uint16_t two;
    uint16_t three;
    uint16_t four;
} Mystruct4;

typedef struct {
    uint16_t one;
    uint16_t two;
    uint16_t three;
    uint16_t four;
    uint16_t five;
} Mystruct5;

The first would presumably be 8 bytes and the second 10.   Is it safe to 
access these in a memory buffer via a pointer so long as that pointer
is aligned on a 2 byte boundary?  I would assume yes because all of the 
data within each struct has that alignment, and I am also assuming that 
this code should always work:

Mystruct5  array[10];  /* valid */
Mystruct5  *aptr;
uint16_t   x;
       array[1].four=20;
       aptr = &array[1];
       /* the next three lines should all print: 1,4 is:20 */
       printf("1,4 is:%u\n",aptr->five);
       somefunction1(aptr);
       somefunction2(*aptr);

...

void somefunction1(Mystruct5 *aptr){
       printf("1,4 is:%u\n",aptr->four);
}
void somefunction2(Mystruct5 aptr){
       printf("1,4 is:%u\n",aptr.four);
}


Or may a compiler make some assumption for Mystruct4 that requires it to 
be aligned on a 4 byte boundary too?  That would not prevent it from 
being used in an array, and would not break the above example (after 
changing Mystruct5 -> Mystruct4).  But it would break both of the two 
function calls if aptr was pointing to a buffer in memory where the data 
was only 2 byte aligned for the structure.  Conversely, adding a 4 byte 
boundary alignment requirement on Mystruct5 would add two pad bytes, for 
no obvious reason, but it would not break the above code example.  I see 
no reason for the compiler to add the 4 byte alignment
requirement for these structures, but is it nevertheless free to do so?

Thanks,

David Mathog

[toc] | [prev] | [next] | [standalone]

#30314

On 01/18/2013 01:24 PM, mathog wrote:
> Eric Sosman wrote:
>>> Now, what happens when this struct is embedded in a binary file stored
>>> in a character array "buffer[]" such that its position "i" is not
>>> guaranteed to be aligned on the same boundary that
>>>
>>>    Mystract *instance2 = malloc(sizeof(Mystruct));
>>>
>>> would have used, but will always be on a multiple of 2.
>>>
>>> 1.  Will this always work?
>>>
>>>    function1((Mystruct *) &(buffer[i]));
>>
>>      Not "always," no.  6.3.2.3p7: "A pointer to an object type
>> may be converted to a pointer to a different object type. If the
>> resulting pointer is not correctly aligned for the referenced
>> type, the behavior is undefined. [...]"  There's no guarantee
>> that the alignment of `buffer' suffices for `Mystruct'.
> 
> On further consideration, how does one know what "correctly aligned" 
> means for a given struct?  Consider these two examples:

In C99 and earlier, there were only a few cases where you could infer
that a pointer was correctly aligned for a given operation. For
instance, a struct must have alignment requirements at least as strict
as those of any of it's members, but it can be stricter.

In C2011, several alignment-oriented features were added, such as
_Alignof(), and it's now possible to determine exactly whether or not
the alignment of any arbitrary type allows an operation.

> typedef struct {
>     uint16_t one;
>     uint16_t two;
>     uint16_t three;
>     uint16_t four;
> } Mystruct4;
> 
> typedef struct {
>     uint16_t one;
>     uint16_t two;
>     uint16_t three;
>     uint16_t four;
>     uint16_t five;
> } Mystruct5;
> 
> The first would presumably be 8 bytes and the second 10.   Is it safe to 
> access these in a memory buffer via a pointer so long as that pointer
> is aligned on a 2 byte boundary?  I would assume yes because all of the 
> data within each struct has that alignment, and I am also assuming that 
> this code should always work:

You've made several assumptions not guaranteed by the standard:
sizeof(uint16_t) == 2
_Alignof(uint16_t) == 2
_Alignof(Mystruct4) == _Alignof(uint16_t)
_Alignof(Mystruct5) == _Alignof(uint16_t)

For implementations where all of those things are true, your conclusion
holds, but it's not necessarily the case that any of those things are true.

> Or may a compiler make some assumption for Mystruct4 that requires it to 
> be aligned on a 4 byte boundary too? ...

Yes.

> ...  I see 
> no reason for the compiler to add the 4 byte alignment
> requirement for these structures, but is it nevertheless free to do so?

Yes.

[toc] | [prev] | [next] | [standalone]

#30317

On 1/18/2013 1:24 PM, mathog wrote:
> Eric Sosman wrote:
>>> Now, what happens when this struct is embedded in a binary file stored
>>> in a character array "buffer[]" such that its position "i" is not
>>> guaranteed to be aligned on the same boundary that
>>>
>>>    Mystract *instance2 = malloc(sizeof(Mystruct));
>>>
>>> would have used, but will always be on a multiple of 2.
>>>
>>> 1.  Will this always work?
>>>
>>>    function1((Mystruct *) &(buffer[i]));
>>
>>      Not "always," no.  6.3.2.3p7: "A pointer to an object type
>> may be converted to a pointer to a different object type. If the
>> resulting pointer is not correctly aligned for the referenced
>> type, the behavior is undefined. [...]"  There's no guarantee
>> that the alignment of `buffer' suffices for `Mystruct'.
>
> On further consideration, how does one know what "correctly aligned"
> means for a given struct?

     As of C11, the _Alignof operator tells you.  In earlier C's
alignment requirements are implementation-defined, and all you
could tell for sure was that the alignment requirement for a
type T had to be an exact divisor of sizeof(T).

> Consider these two examples:
>
> typedef struct {
>     uint16_t one;
>     uint16_t two;
>     uint16_t three;
>     uint16_t four;
> } Mystruct4;
>
> typedef struct {
>     uint16_t one;
>     uint16_t two;
>     uint16_t three;
>     uint16_t four;
>     uint16_t five;
> } Mystruct5;
>
> The first would presumably be 8 bytes and the second 10.

     If uint16_t exists, it follows that CHAR_BIT must be 8 or 16.
Then the minimum sizes of the first struct is 8 or 4, and of the
second 5 or 10.  But these are minima: Padding is permitted after
any struct element (let's ignore bit-fields), for any reason the
implementor finds convincing.

     Even so, I'd be extremely surprised if the size of the first
struct were not 8 or 4.  I'd be only mildly surprised, though, if
the size of the second were 12 or 6.

> Is it safe to
> access these in a memory buffer via a pointer so long as that pointer
> is aligned on a 2 byte boundary?  I would assume yes because all of the
> data within each struct has that alignment,

     The alignment of a struct (or union) is *at least* as strict
as the maximum alignment of any of its elements, but may be stricter.

> and I am also assuming that
> this code should always work:
>
> Mystruct5  array[10];  /* valid */
> Mystruct5  *aptr;
> uint16_t   x;
>        array[1].four=20;
>        aptr = &array[1];
>        /* the next three lines should all print: 1,4 is:20 */
>        printf("1,4 is:%u\n",aptr->five);
>        somefunction1(aptr);
>        somefunction2(*aptr);
>
> ...
>
> void somefunction1(Mystruct5 *aptr){
>        printf("1,4 is:%u\n",aptr->four);
> }
> void somefunction2(Mystruct5 aptr){
>        printf("1,4 is:%u\n",aptr.four);
> }

     Yes, this will always produce the expected output (once the
typo in the first printf() is corrected and suitable scaffolding
is added).  But I don't see what bearing it has on matters of
alignment; could you elucidate?

> Or may a compiler make some assumption for Mystruct4 that requires it to
> be aligned on a 4 byte boundary too?

     Yes.  It might even require 8-byte alignment.  In theory, the
compiler might insert 1016 padding bytes and require 1024-byte
alignment (this is the same theory that predicts the nasal demon).

> That would not prevent it from
> being used in an array, and would not break the above example (after
> changing Mystruct5 -> Mystruct4).  But it would break both of the two
> function calls if aptr was pointing to a buffer in memory where the data
> was only 2 byte aligned for the structure.

     Right.

> Conversely, adding a 4 byte
> boundary alignment requirement on Mystruct5 would add two pad bytes, for
> no obvious reason, but it would not break the above code example.  I see
> no reason for the compiler to add the 4 byte alignment
> requirement for these structures, but is it nevertheless free to do so?

     You and I see no pressing need for padding in these structs,
but the implementor might see one (or imagine he sees one) -- and
he's got the only vote.

-- 
Eric Sosman
esosman@comcast-dot-net.invalid

[toc] | [prev] | [next] | [standalone]

#30324

mathog <dmathog@gmail.com> wrote:
> Eric Sosman wrote:
>>> Now, what happens when this struct is embedded in a binary file stored
>>> in a character array "buffer[]" such that its position "i" is not
>>> guaranteed to be aligned on the same boundary that

>>>    Mystract *instance2 = malloc(sizeof(Mystruct));

>>> would have used, but will always be on a multiple of 2.

(snip)

> On further consideration, how does one know what "correctly aligned" 
> means for a given struct?  Consider these two examples:
 
> typedef struct {
>    uint16_t one;
>    uint16_t two;
>    uint16_t three;
>    uint16_t four;
> } Mystruct4;
 
> typedef struct {
>    uint16_t one;
>    uint16_t two;
>    uint16_t three;
>    uint16_t four;
>    uint16_t five;
> } Mystruct5;
 
> The first would presumably be 8 bytes and the second 10.   Is it safe to 
> access these in a memory buffer via a pointer so long as that pointer
> is aligned on a 2 byte boundary?  

Consider Alpha, where all memory accesss are on four byte boundaries.
Now, there are special instructions to access 16 byte quantities on
two byte, or even (I believe) one byte boundaries, but it seems to me
that the compiler could optimize this case if it knew the structs
was allocated on a four byte boundary, and padded the end, such that
arrays of the struct aligned them on four byte boundaries. (Or, 
similarly, for other boundaries.)

> I would assume yes because all of the 
> data within each struct has that alignment, and I am also assuming that 
> this code should always work:
 
> Mystruct5  array[10];  /* valid */
> Mystruct5  *aptr;
> uint16_t   x;
>       array[1].four=20;
>       aptr = &array[1];
>       /* the next three lines should all print: 1,4 is:20 */
>       printf("1,4 is:%u\n",aptr->five);
>       somefunction1(aptr);
>       somefunction2(*aptr);
 
> ...
 
> void somefunction1(Mystruct5 *aptr){
>       printf("1,4 is:%u\n",aptr->four);
> }
> void somefunction2(Mystruct5 aptr){
>       printf("1,4 is:%u\n",aptr.four);
> }
 
> Or may a compiler make some assumption for Mystruct4 that requires it to 
> be aligned on a 4 byte boundary too?  That would not prevent it from 
> being used in an array, and would not break the above example (after 
> changing Mystruct5 -> Mystruct4).  But it would break both of the two 
> function calls if aptr was pointing to a buffer in memory where the data 
> was only 2 byte aligned for the structure.  Conversely, adding a 4 byte 
> boundary alignment requirement on Mystruct5 would add two pad bytes, for 
> no obvious reason, but it would not break the above code example.  I see 
> no reason for the compiler to add the 4 byte alignment
> requirement for these structures, but is it nevertheless free to do so?

Well, on 32 bit word addressed machines it is somewhat obvious what
might happen. Alpha is byte addressed but, at least until MS got into
it, required access addresses to be multiples of four.A

Given a pointer to an arbitrary (16 bit) short, the compiler has to use
the general case, but it seems to me that it could optimize the struct
access case.

-- glen

[toc] | [prev] | [next] | [standalone]

#30328

On 1/18/2013 13:24, mathog wrote:
> Eric Sosman wrote:
>>> Now, what happens when this struct is embedded in a binary file stored
>>> in a character array "buffer[]" such that its position "i" is not
>>> guaranteed to be aligned on the same boundary that
>>>
>>>    Mystract *instance2 = malloc(sizeof(Mystruct));
>>>
>>> would have used, but will always be on a multiple of 2.
>>>
>>> 1.  Will this always work?
>>>
>>>    function1((Mystruct *) &(buffer[i]));
>>
>>      Not "always," no.  6.3.2.3p7: "A pointer to an object type
>> may be converted to a pointer to a different object type. If the
>> resulting pointer is not correctly aligned for the referenced
>> type, the behavior is undefined. [...]"  There's no guarantee
>> that the alignment of `buffer' suffices for `Mystruct'.
>
> On further consideration, how does one know what "correctly aligned"
> means for a given struct?  Consider these two examples:
>
> typedef struct {
>     uint16_t one;
>     uint16_t two;
>     uint16_t three;
>     uint16_t four;
> } Mystruct4;
>
> typedef struct {
>     uint16_t one;
>     uint16_t two;
>     uint16_t three;
>     uint16_t four;
>     uint16_t five;
> } Mystruct5;
>
> The first would presumably be 8 bytes and the second 10.   Is it safe to
> access these in a memory buffer via a pointer so long as that pointer
> is aligned on a 2 byte boundary?  I would assume yes because all of the
> data within each struct has that alignment, and I am also assuming that
> this code should always work:
>

You can infer that both of these are correctly aligned for a 'uint16_t' 
(the first member of each), if that type exists, which it mightn't.  You 
can convert both a 'Mystruct4 *' and a 'Mystruct5 *' to a 'uint16_t *' 
and back again, but you cannot safely convert a 'uint16_t *' of unknown 
origin to either a 'Mystruct4 *' or a 'Mystruct5 *'.

There can be padding between members and after the last member.  The 
alignment requirement must be an integer factor of the size of the 
structure, and in C11 it must be an integer power of two.  It might be 
argued that this further constraint in C11 suggests that it was always 
the case, or that anyone who was doing something else has no 
representation with Working Group 14. :)

One sometimes sees padding after the penultimate member for the case of 
Flexible Array Members (>= C99).

> Mystruct5  array[10];  /* valid */
> Mystruct5  *aptr;
> uint16_t   x;
>        array[1].four=20;
>        aptr = &array[1];
>        /* the next three lines should all print: 1,4 is:20 */
>        printf("1,4 is:%u\n",aptr->five);

I think you meant 'aptr->four'.

>        somefunction1(aptr);
>        somefunction2(*aptr);
>
> ...
>
> void somefunction1(Mystruct5 *aptr){
>        printf("1,4 is:%u\n",aptr->four);
> }
> void somefunction2(Mystruct5 aptr){
>        printf("1,4 is:%u\n",aptr.four);
> }
>
>
> Or may a compiler make some assumption for Mystruct4 that requires it to
> be aligned on a 4 byte boundary too?  That would not prevent it from
> being used in an array, and would not break the above example (after
> changing Mystruct5 -> Mystruct4).  But it would break both of the two
> function calls if aptr was pointing to a buffer in memory where the data
> was only 2 byte aligned for the structure.  Conversely, adding a 4 byte
> boundary alignment requirement on Mystruct5 would add two pad bytes, for
> no obvious reason, but it would not break the above code example.  I see
> no reason for the compiler to add the 4 byte alignment
> requirement for these structures, but is it nevertheless free to do so?
>

There's no need to assume, is there?  We've already discussed C11 and 
pre-C11 methods for detecting alignment requirements.

The alignment requirement of 'Mystruct5' is unlikely to be 10, pre-C11, 
and not allowed to be 10, for C11.

Given the definition of "common initial sequence" all the way back to 
C89, it is likely that you could cast a 'Mystruct5 *' to a 'Mystruct4 *' 
and back again.  If your code contains:

   #include <stddef.h>

   union {
       Mystruct4 m4;
       Mystruct5 m5;
     } dummy;
   enum e_dummy {
       cv_dummy1 = offsetof(Mystruct4, four),
       cv_dummy2 = offsetof(Mystruct5, four)
     };

then it is guaranteed that these two enum values will compare as equal.

-- 
- Shao Miller
--
"Thank you for the kind words; those are the kind of words I like to hear.

Cheerily,"  -- Richard Harter

[toc] | [prev] | [next] | [standalone]

#30330

Shao Miller <sha0.miller@gmail.com> wrote:

(snip)
> There can be padding between members and after the last member.  The 
> alignment requirement must be an integer factor of the size of the 
> structure, and in C11 it must be an integer power of two.  It might be 
> argued that this further constraint in C11 suggests that it was always 
> the case, or that anyone who was doing something else has no 
> representation with Working Group 14. :)

Machines with decimal addressing are long gone now.

I believe the IBM 650 was one, and maybe a few others from 
that era.

No idea about any alignment requirements, though.

-- glen

[toc] | [prev] | [next] | [standalone]

#30333

mathog <dmathog@gmail.com> writes:
[...]
> On further consideration, how does one know what "correctly aligned" 
> means for a given struct?  Consider these two examples:
>
> typedef struct {
>     uint16_t one;
>     uint16_t two;
>     uint16_t three;
>     uint16_t four;
> } Mystruct4;
>
> typedef struct {
>     uint16_t one;
>     uint16_t two;
>     uint16_t three;
>     uint16_t four;
>     uint16_t five;
> } Mystruct5;
>
> The first would presumably be 8 bytes and the second 10.   Is it safe to 
> access these in a memory buffer via a pointer so long as that pointer
> is aligned on a 2 byte boundary?  I would assume yes because all of the 
> data within each struct has that alignment, and I am also assuming that 
> this code should always work:

The required alignment for a given type is largely up to the whims
of the compiler implementer and the compiler implementer, subject
to the constraints of the standard and common sense.

For the case above, an implementation might require 2-byte alignment
for Mystruct4, or it might require 4-byte or even 8-byte alignment.
Accesses to individual members would only require them to be 2-byte
aligned (or even 1-byte aligned on some implementations), but code
that assigns a whole struct might use 4-byte or 8-byte reads and
writes that might require 4-byte or 8-byte alignment.  A single
64-bit move instruction might be much faster than copying the
struture in pieces, which could be worth the cost in alignment gaps.

For Mystruct5, a compiler could conceivably add 6 bytes of padding
to make the structure as a whole 16 bytes, and then impose a 4-byte,
8-byte, or 16-byte alignment on the structure.  Again, this could
make accesses to entire Mystruct5 objects substantially faster.

[...]

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#30374

>>> Now, what happens when this struct is embedded in a binary file stored
>>> in a character array "buffer[]" such that its position "i" is not
>>> guaranteed to be aligned on the same boundary that
>>>
>>>    Mystract *instance2 = malloc(sizeof(Mystruct));
>>>
>>> would have used, but will always be on a multiple of 2.
>>>
>>> 1.  Will this always work?
>>>
>>>    function1((Mystruct *) &(buffer[i]));
>>
(Eric Sosman wrote:)
>>      Not "always," no.  6.3.2.3p7: "A pointer to an object type
>> may be converted to a pointer to a different object type. If the
>> resulting pointer is not correctly aligned for the referenced
>> type, the behavior is undefined. [...]"  There's no guarantee
>> that the alignment of `buffer' suffices for `Mystruct'.
>
> On further consideration, how does one know what "correctly aligned"
> means for a given struct?  Consider these two examples:
>
> typedef struct {
>     uint16_t one;
>     uint16_t two;
>     uint16_t three;
>     uint16_t four;
> } Mystruct4;
>


Taking into account the various answers in this thread to the preceding
question it would seem that there is still no safe (safe meaning here 
"code that will work on any platform") method to use a C struct
to extract binary data directly from memory, since the compiler can 
throw in padding and alignment requirements whenever it feels like it.
Even if the data is reduced to byte representation, something like this
(I know it isn't the same as Mystruct4 above):

  typedef struct {
      uint8_t one[2];    /* actually uint16_t */
      uint8_t two[4];    /* actually uint32_t */
      int8_t  three[2];  /* etc. */
      int8_t four;
  } Mystruct4b;

it seems not to be safe to pass a "Mystruct4b *" pointer to a function 
which references this data at an arbitrary location in memory.  Instead 
the only safe method is to pass "char *" pointers and take the data 
apart with memcpy() at a very low level, moving it from memory to the 
structure, or vice versa.

What an odd situation.  I would have thought with all of the data that 
does show up in C programming as "a series of bytes in a buffer arranged 
in some particular manner", that is, pretty much any data which is 
passed from machine to machine, C would have developed a method for 
simplifying this sort of code.  Perhaps something along these lines

  /* declare memory organization, no padding, no alignment requirement */
  typedef memstruct {
      uint16_t one;
      uint32_t two;
      int16_t  three;
      uint8_t  four;
  } Mystruct4c;

Specifically so that one could pass a "Mystruct4c *" pointer to a 
function, like so:

    myfunction3a(Mystruct4c *ptr){
       ptr->two = 5;
       printf("value of three:%d\n",ptr->three);
    }

and the compiler would do the "right thing", using memcpy or whatever, 
to hide all of the cruft that is platform specific, up to and including 
loading the magic N bytes of memory (as in the Alpha issue Glen 
mentioned), shifting, and masking and so forth, to handle data types 
smaller than is native for the CPU, without the programmer having to 
ever care about the details.  One might even dream that the compiler
could be induced to do:

   Mystruct4c  native,ncopy;
   Mystruct4b *foreign
   Mystruct4b  acopy;

   /* field names/sizes must match for the following statement */
   alternate_representations {Mystruct4b, Mystruct4c}

   char *buffer;
   /* fill buffer from a file or network */
   foreign = (Mystruct4b *)buffer[123];
   native = *foreign; /* field to field copy, NOT a memcpy*/
   /* change some data in native*/
   *foreign = native; /* field to field copy */
   ncopy  = native;   /* this is a memcpy */
   acopy = *foreign;  /*    as is this   */
   *foreign = acopy;  /*    as is this   */
   ncopy = acopy;     /* field to field copy */

That is, when a memstruct pointer is dereferenced it does not mean quite 
the same thing as when a struct pointer is dereferenced.  A few more 
rules for the compiler, a lot less work for the programmer.

Thanks,

David Mathog

[toc] | [prev] | [next] | [standalone]

#30376

On 1/20/2013 00:02, mathog wrote:
>
> What an odd situation.  I would have thought with all of the data that
> does show up in C programming as "a series of bytes in a buffer arranged
> in some particular manner", that is, pretty much any data which is
> passed from machine to machine, C would have developed a method for
> simplifying this sort of code.
>

There are implementation extensions and libraries that both help with 
this sort of thing.  Some of the rest of your post resembles what's 
available for certain implementations ("packed structures," for 
example).  I don't think it's universal enough to warrant inclusion in 
the Standard, but that's just my opinion.

I know it'd be nice to be able to say, "Any C implementation supports 
the features I desire," but the people who make the implementations 
might disagree. :)

I don't know what your actual goal is, but if it involves communicating 
data between different C implementations, then there's more than 
alignment and padding to worry about.

You appear to wish to work with structures, for some reason.  Why is 
that?  There are other ways to group data values together.

-- 
- Shao Miller
--
"Thank you for the kind words; those are the kind of words I like to hear.

Cheerily,"  -- Richard Harter

[toc] | [prev] | [next] | [standalone]

#30382

mathog <mathog@caltech.edu> wrote:
>>>> Now, what happens when this struct is embedded in a binary file stored
>>>> in a character array "buffer[]" such that its position "i" is not
>>>> guaranteed to be aligned on the same boundary that

>>>>    Mystract *instance2 = malloc(sizeof(Mystruct));
>>>> would have used, but will always be on a multiple of 2.

>>>> 1.  Will this always work?

(snip)
> (Eric Sosman wrote:)
>>>      Not "always," no.  6.3.2.3p7: "A pointer to an object type
>>> may be converted to a pointer to a different object type. If the
>>> resulting pointer is not correctly aligned for the referenced
>>> type, the behavior is undefined. [...]"  There's no guarantee
>>> that the alignment of `buffer' suffices for `Mystruct'.

>> On further consideration, how does one know what "correctly aligned"
>> means for a given struct?  Consider these two examples:

>> typedef struct {
>>     uint16_t one;
>>     uint16_t two;
>>     uint16_t three;
>>     uint16_t four;
>> } Mystruct4;

> Taking into account the various answers in this thread to the preceding
> question it would seem that there is still no safe (safe meaning here 
> "code that will work on any platform") method to use a C struct
> to extract binary data directly from memory, since the compiler can 
> throw in padding and alignment requirements whenever it feels like it.

While I believe that is true, it is still most likely that you can
use it on any appropriate boundary. 

> Even if the data is reduced to byte representation, something like this
> (I know it isn't the same as Mystruct4 above):
 
>  typedef struct {
>      uint8_t one[2];    /* actually uint16_t */
>      uint8_t two[4];    /* actually uint32_t */
>      int8_t  three[2];  /* etc. */
>      int8_t four;
>  } Mystruct4b;
 
> it seems not to be safe to pass a "Mystruct4b *" pointer to a function 
> which references this data at an arbitrary location in memory.  Instead 
> the only safe method is to pass "char *" pointers and take the data 
> apart with memcpy() at a very low level, moving it from memory to the 
> structure, or vice versa.

Yes, that is always the most reliable way. Especially if you have
the possibility of different endianness. (Not to menion all the other
possible different representations.)
 
> What an odd situation.  I would have thought with all of the data that 
> does show up in C programming as "a series of bytes in a buffer arranged 
> in some particular manner", that is, pretty much any data which is 
> passed from machine to machine, C would have developed a method for 
> simplifying this sort of code.  Perhaps something along these lines

Well, pretty much it optimizes for passing data around within the
program. 

For many years Fortran required, or at least it was believed to require,
that COMMON blocks be packed (no padding). On some machine, that was
just a little slow, on others it required run-time trap for the
misaligned access, copy the data, perform the operation, copy the data
back again, and then return. Much much slower.

Most RISC processors require data to be aligned, though some have
special instructions for access to misaligned data. (Faster than a byte
copy and performing the operation on the copy.)
 
>  /* declare memory organization, no padding, no alignment requirement */
>  typedef memstruct {
>      uint16_t one;
>      uint32_t two;
>      int16_t  three;
>      uint8_t  four;
>  } Mystruct4c;

Doesn't help if the actual representation, such as endianness,
is different. 
 
> Specifically so that one could pass a "Mystruct4c *" pointer to a 
> function, like so:
 
>    myfunction3a(Mystruct4c *ptr){
>       ptr->two = 5;
>       printf("value of three:%d\n",ptr->three);
>    }
 
> and the compiler would do the "right thing", using memcpy or whatever, 
> to hide all of the cruft that is platform specific, up to and including 
> loading the magic N bytes of memory (as in the Alpha issue Glen 
> mentioned), shifting, and masking and so forth, to handle data types 
> smaller than is native for the CPU, without the programmer having to 
> ever care about the details.  

In the cases where the compiler knows in advance, it isn't
so bad. But then if you pass a pointer to misaligned data, then it is
referenced in a place that doesn't expect it to be misaligned.
Or, the compiler has to generate slow (maybe much slower) code
for all accesses.

> One might even dream that the compiler could be induced to do:

>   Mystruct4c  native,ncopy;
>   Mystruct4b *foreign
>   Mystruct4b  acopy;
 
>   /* field names/sizes must match for the following statement */
>   alternate_representations {Mystruct4b, Mystruct4c}

Well, when C was new there were still plenty of 36 bit, 48 bit,
and 60 bit machines around, and probably others that I don't know
about. In addition, there is at least (still in the standard) allowance 
for sign magnitude or ones complement representation, and finally
endianness. But yes, it could be done.
 
>   char *buffer;
>   /* fill buffer from a file or network */
>   foreign = (Mystruct4b *)buffer[123];
>   native = *foreign; /* field to field copy, NOT a memcpy*/
>   /* change some data in native*/
>   *foreign = native; /* field to field copy */
>   ncopy  = native;   /* this is a memcpy */
>   acopy = *foreign;  /*    as is this   */
>   *foreign = acopy;  /*    as is this   */
>   ncopy = acopy;     /* field to field copy */

Well, there is XDR http://www.ietf.org/rfc/rfc4506.txt
which will do all the work for any reasonable, and also not
so reasonable representation.
 
> That is, when a memstruct pointer is dereferenced it does not mean quite 
> the same thing as when a struct pointer is dereferenced.  A few more 
> rules for the compiler, a lot less work for the programmer.

Well, as in another thread, consider C a portable assembler.
It helps you, but you still have to do some of the work.

-- glen

[toc] | [prev] | [next] | [standalone]

#30400

mathog <mathog@caltech.edu> writes:
<snip>
> Taking into account the various answers in this thread to the preceding
> question it would seem that there is still no safe (safe meaning here
> "code that will work on any platform") method to use a C struct
> to extract binary data directly from memory, since the compiler can
> throw in padding and alignment requirements whenever it feels like it.
> Even if the data is reduced to byte representation, something like this
> (I know it isn't the same as Mystruct4 above):
>
>  typedef struct {
>      uint8_t one[2];    /* actually uint16_t */
>      uint8_t two[4];    /* actually uint32_t */
>      int8_t  three[2];  /* etc. */
>      int8_t four;
>  } Mystruct4b;
>
> it seems not to be safe to pass a "Mystruct4b *" pointer to a function
> which references this data at an arbitrary location in memory.
> Instead the only safe method is to pass "char *" pointers and take the
> data apart with memcpy() at a very low level, moving it from memory to
> the structure, or vice versa.
>
> What an odd situation.

Maybe it looks odder now than it did in the days C was designed.  In
those days, there was a veritable wild west of machine formats: 36-bit
ints, 9-bit bytes, 24-bit addresses, 60-bit floats, little endian, big
endian and even middle-endian byte orders, etc, etc.  Since you could
not even map a couple of bytes onto an int and be sure you have the
right value, it would have seemed pointless to try to address the other
problems of alignment and packing.

>  I would have thought with all of the data that
> does show up in C programming as "a series of bytes in a buffer
> arranged in some particular manner", that is, pretty much any data
> which is passed from machine to machine, C would have developed a
> method for simplifying this sort of code. Perhaps something along
> these lines
>
>  /* declare memory organization, no padding, no alignment requirement */
>  typedef memstruct {
>      uint16_t one;
>      uint32_t two;
>      int16_t  three;
>      uint8_t  four;
>  } Mystruct4c;

You'd want some syntax to distinguish these access-inefficient structs
from normal ones.  __attribute__((packed)) maybe?

> Specifically so that one could pass a "Mystruct4c *" pointer to a
> function, like so:
>
>    myfunction3a(Mystruct4c *ptr){
>       ptr->two = 5;
>       printf("value of three:%d\n",ptr->three);
>    }
>
> and the compiler would do the "right thing", using memcpy or whatever,
> to hide all of the cruft that is platform specific, up to and
> including loading the magic N bytes of memory (as in the Alpha issue
> Glen mentioned), shifting, and masking and so forth, to handle data
> types smaller than is native for the CPU, without the programmer
> having to ever care about the details.

But you would still not have portable struct to byte array mapping due
to differing byte orders.  What about differing signed integer
representations?  What about floating-point formats?  You could deal
with all these too, of course, but at some point you have to say that
this is not something that should be built in to a programming
language.

<snip>
-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#30402

mathog <mathog@caltech.edu> writes:
[...]
> What an odd situation.  I would have thought with all of the data that 
> does show up in C programming as "a series of bytes in a buffer arranged 
> in some particular manner", that is, pretty much any data which is 
> passed from machine to machine, C would have developed a method for 
> simplifying this sort of code.  Perhaps something along these lines
>
>   /* declare memory organization, no padding, no alignment requirement */
>   typedef memstruct {
>       uint16_t one;
>       uint32_t two;
>       int16_t  three;
>       uint8_t  four;
>   } Mystruct4c;
>
> Specifically so that one could pass a "Mystruct4c *" pointer to a 
> function, like so:
>
>     myfunction3a(Mystruct4c *ptr){
>        ptr->two = 5;
>        printf("value of three:%d\n",ptr->three);
>     }
>
> and the compiler would do the "right thing", using memcpy or whatever, 
> to hide all of the cruft that is platform specific, up to and including 
> loading the magic N bytes of memory (as in the Alpha issue Glen 
> mentioned), shifting, and masking and so forth, to handle data types 
> smaller than is native for the CPU, without the programmer having to 
> ever care about the details.
[...]

Consider this:

    Mystruct4c obj;
    void func(uint32_t *arg);
    func(&obj.two);

Given your definition of what a "memstruct" is, the code that implements
func() would have to allow for the possibility that its argument points
to an unaligned int32_t object.

(Note that gcc's "__attribute__((packed))" doesn't solve this; see my
discussion here: http://stackoverflow.com/q/8568432/827263 and here:
http://stackoverflow.com/a/8568441/827263.)

The alternative, I suppose, would be to forbid taking the address of a
member of a "memstruct", treating its members much like bit fields.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#30408

Keith Thompson於 2013年1月20日星期日UTC+8下午11時35分29秒寫道：
> mathog <mathog@caltech.edu> writes:
> 
> [...]
> 
> > What an odd situation.  I would have thought with all of the data that 
> 
> > does show up in C programming as "a series of bytes in a buffer arranged 
> 
> > in some particular manner", that is, pretty much any data which is 
> 
> > passed from machine to machine, C would have developed a method for 
> 
> > simplifying this sort of code.  Perhaps something along these lines
> 
> >
> 
> >   /* declare memory organization, no padding, no alignment requirement */
> 
> >   typedef memstruct {
> 
> >       uint16_t one;
> 
> >       uint32_t two;
> 
> >       int16_t  three;
> 
> >       uint8_t  four;
> 
> >   } Mystruct4c;
> 
> >
> 
> > Specifically so that one could pass a "Mystruct4c *" pointer to a 
> 
> > function, like so:
> 
> >
> 
> >     myfunction3a(Mystruct4c *ptr){
> 
> >        ptr->two = 5;
> 
> >        printf("value of three:%d\n",ptr->three);
> 
> >     }
> 
> >
> 
> > and the compiler would do the "right thing", using memcpy or whatever, 
> 
> > to hide all of the cruft that is platform specific, up to and including 
> 
> > loading the magic N bytes of memory (as in the Alpha issue Glen 
> 
> > mentioned), shifting, and masking and so forth, to handle data types 
> 
> > smaller than is native for the CPU, without the programmer having to 
> 
> > ever care about the details.
> 
> [...]
> 
> 
> 
> Consider this:
> 
> 
> 
>     Mystruct4c obj;
> 
>     void func(uint32_t *arg);
> 
>     func(&obj.two);
> 
> 
> 
> Given your definition of what a "memstruct" is, the code that implements
> 
> func() would have to allow for the possibility that its argument points
> 
> to an unaligned int32_t object.
> 
> 
> 
> (Note that gcc's "__attribute__((packed))" doesn't solve this; see my
> 
> discussion here: http://stackoverflow.com/q/8568432/827263 and here:
> 
> http://stackoverflow.com/a/8568441/827263.)
> 
> 
> 
> The alternative, I suppose, would be to forbid taking the address of a
> 
> member of a "memstruct", treating its members much like bit fields.
> 
> 
> 
> -- 
> 
> Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
> 
> Working, but not speaking, for JetHead Development, Inc.
> 
> "We must do something.  This is something.  Therefore, we must do this."
> 
>     -- Antony Jay and Jonathan Lynn, "Yes Minister"

The endian problem in C does matter in programming
with a file format which specifies the endian type
independently from the structure in the memory which the endian is specified by the machine or the emulator.

But that is not hard to solve if one got paid well.

[toc] | [prev] | [next] | [standalone]

#30404

On 01/20/2013 12:02 AM, mathog wrote:
>>>> Now, what happens when this struct is embedded in a binary file stored
>>>> in a character array "buffer[]" such that its position "i" is not
>>>> guaranteed to be aligned on the same boundary that
>>>>
>>>>    Mystract *instance2 = malloc(sizeof(Mystruct));
>>>>
>>>> would have used, but will always be on a multiple of 2.
>>>>
>>>> 1.  Will this always work?
>>>>
>>>>    function1((Mystruct *) &(buffer[i]));
>>>
> (Eric Sosman wrote:)
>>>      Not "always," no.  6.3.2.3p7: "A pointer to an object type
>>> may be converted to a pointer to a different object type. If the
>>> resulting pointer is not correctly aligned for the referenced
>>> type, the behavior is undefined. [...]"  There's no guarantee
>>> that the alignment of `buffer' suffices for `Mystruct'.
>>
>> On further consideration, how does one know what "correctly aligned"
>> means for a given struct?  Consider these two examples:
>>
>> typedef struct {
>>     uint16_t one;
>>     uint16_t two;
>>     uint16_t three;
>>     uint16_t four;
>> } Mystruct4;


I've noticed that you routinely fail to provide citations for who wrote
the things that you quoted. That's rather rude to the persons who said
it,, and also inconvenient for the rest of us trying to figure out who
said what. It can also get you into legal trouble in some contexts,
thought usenet posts are so easily forged that it would be hard to
imagine that happening in this context.

I deny permission to any and all authors to quote any text that I write
without appropriate citation. Don't bother responding if you're not
willing to provide it.

> Taking into account the various answers in this thread to the preceding
> question it would seem that there is still no safe (safe meaning here 
> "code that will work on any platform") method to use a C struct
> to extract binary data directly from memory, since the compiler can 
> throw in padding and alignment requirements whenever it feels like it.

Not quite - for instance, padding is not allowed at the beginning, and
the alignment requirement for an aggregate object (a struct or an array)
must be at least as strict as it is for any of the members of the
aggregate. Still, the basic point remains - the standard does not
provide enough guarantees to allow the use such a struct to read date
directly from a file.

> Even if the data is reduced to byte representation, something like this
> (I know it isn't the same as Mystruct4 above):
> 
>   typedef struct {
>       uint8_t one[2];    /* actually uint16_t */
>       uint8_t two[4];    /* actually uint32_t */
>       int8_t  three[2];  /* etc. */
>       int8_t four;
>   } Mystruct4b;
> 
> it seems not to be safe to pass a "Mystruct4b *" pointer to a function 
> which references this data at an arbitrary location in memory.  Instead 
> the only safe method is to pass "char *" pointers and take the data 
> apart with memcpy() at a very low level, moving it from memory to the 
> structure, or vice versa.

Correct. Because of these things that are not guaranteed, C structs
should be considered as suitable only for use for in-memory objects, not
for reading or writing from files.

> What an odd situation.  I would have thought with all of the data that 
> does show up in C programming as "a series of bytes in a buffer arranged 
> in some particular manner", that is, pretty much any data which is 
> passed from machine to machine, C would have developed a method for 
> simplifying this sort of code.  Perhaps something along these lines

I agree that some such feature would be useful. But there are problems
with the various possible alternatives.

>   /* declare memory organization, no padding, no alignment requirement */
>   typedef memstruct {
>       uint16_t one;
>       uint32_t two;
>       int16_t  three;
>       uint8_t  four;
>   } Mystruct4c;
> 
> Specifically so that one could pass a "Mystruct4c *" pointer to a 
> function, like so:
> 
>     myfunction3a(Mystruct4c *ptr){
>        ptr->two = 5;
>        printf("value of three:%d\n",ptr->three);
>     }


The fundamental problem with such a feature is that &ptr->two is not
necessarily correctly aligned for a uint32_t object. This is not a
serious problem on some architectures, where alignment is a mere
efficiency issue. However, there are other platforms where the hardware
imposes very strict alignment requirements.
In code where Mystruct4c is in scope, and where the derivation of the
pointer from an object of that type is perfectly clear, the compiler
could write special code equivalent to using memcpy() to copy from
ptr->two to correctly aligned memory, and then passing the value stored
in that temporary to printf(); for writing to such an object, a pointer
to correctly aligned memory could be used, and them the equivalent of
memcpy() could be used to copy from the temporary object back to the
incorrectly aligned memory. However, consider code like the following:

     uint32_t default = 42;
     uint32_t *p = default < ptr->two ? &default : &ptr->two;
     myfunc(p);

How can myfunc know how to use 'p', when it might be either a pointer to
memory which is correctly aligned for a uint32_t, or a misaligned
pointer? Also, using pointers to temporary correctly-aligned pieces of
memory would mess up comparisons of pointers for equality or relative order.
-- 
James Kuyper

[toc] | [prev] | [next] | [standalone]

#30406

James Kuyper wrote:
> I've noticed that you routinely fail to provide citations for who wrote
> the things that you quoted.
Sorry. When the number of replies is more than 1 or 2 I can't ever 
remember who is >>>> and who is >>>>>> in any case.

That's rather rude to the persons who said
> it,, and also inconvenient for the rest of us trying to figure out who
> said what. It can also get you into legal trouble in some contexts,
> thought usenet posts are so easily forged that it would be hard to
> imagine that happening in this context.
>
> I deny permission to any and all authors to quote any text that I write
> without appropriate citation. Don't bother responding if you're not
> willing to provide it.
>
>> Taking into account the various answers in this thread to the preceding
>> question it would seem that there is still no safe (safe meaning here
>> "code that will work on any platform") method to use a C struct
>> to extract binary data directly from memory, since the compiler can
>> throw in padding and alignment requirements whenever it feels like it.
>
> Not quite - for instance, padding is not allowed at the beginning, and
> the alignment requirement for an aggregate object (a struct or an array)
> must be at least as strict as it is for any of the members of the
> aggregate. Still, the basic point remains - the standard does not
> provide enough guarantees to allow the use such a struct to read date
> directly from a file.
>
>> Even if the data is reduced to byte representation, something like this
>> (I know it isn't the same as Mystruct4 above):
>>
>>    typedef struct {
>>        uint8_t one[2];    /* actually uint16_t */
>>        uint8_t two[4];    /* actually uint32_t */
>>        int8_t  three[2];  /* etc. */
>>        int8_t four;
>>    } Mystruct4b;
>>
>> it seems not to be safe to pass a "Mystruct4b *" pointer to a function
>> which references this data at an arbitrary location in memory.  Instead
>> the only safe method is to pass "char *" pointers and take the data
>> apart with memcpy() at a very low level, moving it from memory to the
>> structure, or vice versa.
>
> Correct. Because of these things that are not guaranteed, C structs
> should be considered as suitable only for use for in-memory objects, not
> for reading or writing from files.
>
>> What an odd situation.  I would have thought with all of the data that
>> does show up in C programming as "a series of bytes in a buffer arranged
>> in some particular manner", that is, pretty much any data which is
>> passed from machine to machine, C would have developed a method for
>> simplifying this sort of code.  Perhaps something along these lines
>
> I agree that some such feature would be useful. But there are problems
> with the various possible alternatives.
>
>>    /* declare memory organization, no padding, no alignment requirement */
>>    typedef memstruct {
>>        uint16_t one;
>>        uint32_t two;
>>        int16_t  three;
>>        uint8_t  four;
>>    } Mystruct4c;
>>
>> Specifically so that one could pass a "Mystruct4c *" pointer to a
>> function, like so:
>>
>>      myfunction3a(Mystruct4c *ptr){
>>         ptr->two = 5;
>>         printf("value of three:%d\n",ptr->three);
>>      }
>
>
> The fundamental problem with such a feature is that &ptr->two is not
> necessarily correctly aligned for a uint32_t object.

Exactly.  memstruct != struct.  A memstruct is a memory structure 
designed for accessing nonaligned data, the field organization is 
verbatim and cannot be distorted by the compiler to add padding or 
require alignments.  In a memstruct "ptr->two" means some equivalent of:

    uint32_t tmp4;
    memcpy(&tmp4,ptr + offsetof(Mystruct4c,two),4);

especially including the cases where tmp4 is in a register, "memcpy" is 
some assembler operation(s) to copy from a random point in memory into 
that register, so that

   &(ptr->two)

would not be allowed.  Just like this:

   register uint32_t tmp4;
   printf("Pointer to:%p\n",&tmp4);

is not valid. Presumably casting a memstruct to a struct and vice versa 
would also be forbidden.  That was the point of:

 >  Mystruct4c  native;
 >  Mystruct4b *foreign
 >  /* field names/sizes must match for the following statement */
 >  alternate_representations {Mystruct4b, Mystruct4c}

which tells the compiler that it needs to know how to copy data in 
either direction between a specific memstruct and a specific struct. 
(And that would be illegal if the fields didn't match exactly.)

I am not worried about endian issues or other floating point 
representations.  Both of these can be dealt with easily enough once the 
data has been mapped from nonaligned to aligned data structures.  As for 
bytes that are not 8 bits, does any modern hardware exist where that is 
not true?  Must we keep the requirement to run on archaic hardware 
forever going forward with newer versions of the language?

Regards,

David Mathog

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | comp.lang.c

csiph-web