Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #106277 > unrolled thread

Parsing struct memory?

Back to article view | Back to comp.lang.c

Contents

  Parsing struct memory? Thiago Adams <thiago.adams@gmail.com> - 2017-03-16 12:58 -0700
    Re: Parsing struct memory? Thiago Adams <thiago.adams@gmail.com> - 2017-03-16 13:21 -0700
      Re: Parsing struct memory? Ian Collins <ian-news@hotmail.com> - 2017-03-17 11:16 +1300
        Re: Parsing struct memory? James Kuyper <jameskuyper@verizon.net> - 2017-03-16 22:09 -0400
          Re: Parsing struct memory? Ian Collins <ian-news@hotmail.com> - 2017-03-17 15:23 +1300
          Re: Parsing struct memory? David Brown <david.brown@hesbynett.no> - 2017-03-17 10:11 +0100
            Re: Parsing struct memory? "James R. Kuyper" <jameskuyper@verizon.net> - 2017-03-17 11:35 -0400
      Re: Parsing struct memory? gordonb.7zilv@burditt.org (Gordon Burditt) - 2017-03-16 20:17 -0500
        Re: Parsing struct memory? James Kuyper <jameskuyper@verizon.net> - 2017-03-16 21:56 -0400
    Re: Parsing struct memory? supercat@casperkitty.com - 2017-03-16 13:51 -0700
      Re: Parsing struct memory? Lowell Gilbert <lgusenet@be-well.ilk.org> - 2017-03-16 19:39 -0400
        Re: Parsing struct memory? BartC <bc@freeuk.com> - 2017-03-17 12:16 +0000
        Re: Parsing struct memory? "Peter \"Shaggy\" Haywood" <phaywood@alphalink.com.au> - 2017-03-24 18:06 +1100
          Re: Parsing struct memory? supercat@casperkitty.com - 2017-03-24 14:35 -0700
    Re: Parsing struct memory? Keith Thompson <kst-u@mib.org> - 2017-03-16 19:14 -0700
      Re: Parsing struct memory? scott@slp53.sl.home (Scott Lurndal) - 2017-03-17 12:45 +0000
    Re: Parsing struct memory? scott@slp53.sl.home (Scott Lurndal) - 2017-03-17 12:41 +0000
    Re: Parsing struct memory? Thiago Adams <thiago.adams@gmail.com> - 2017-03-17 08:17 -0700
    Re: Parsing struct memory? Thiago Adams <thiago.adams@gmail.com> - 2017-03-20 05:07 -0700

#106277 — Parsing struct memory?

Having some struct;

struct {
  char c;
  void* p;
  double d; 
};

Is it possible to parse it just knowing each type?
Something similar of var_arg but for structs.

char c = arg(pointer, char);
void* p = arg(pointer, void*);
double d = arg(pointer, double);

I was trying https://en.wikipedia.org/wiki/Data_structure_alignment

for x86 this is working on visual studio to return the next pointer.

#define NEXT(n)   ( (sizeof(n) + sizeof(int) - 1) & ~(sizeof(int) - 1) )

For x64 not. Changing int for __int64 doesn't work as well.

[toc] | [next] | [standalone]

#106282

On Thursday, March 16, 2017 at 4:58:24 PM UTC-3, Thiago Adams wrote:
> Having some struct;
> 
> struct {
>   char c;
>   void* p;
>   double d; 
> };
> 
> Is it possible to parse it just knowing each type?
> Something similar of var_arg but for structs.
> 
> char c = arg(pointer, char);
> void* p = arg(pointer, void*);
> double d = arg(pointer, double);
> 
> I was trying https://en.wikipedia.org/wiki/Data_structure_alignment
> 
> for x86 this is working on visual studio to return the next pointer.
> 
> #define NEXT(n)   ( (sizeof(n) + sizeof(int) - 1) & ~(sizeof(int) - 1) )
> 
> For x64 not. Changing int for __int64 doesn't work as well.

Something I also would like to understand is how malloc knows the alignment if it doesn't know the types?
Does it always align with pointer size?

And how about aligned_alloc? Why do we need it?

[toc] | [prev] | [next] | [standalone]

#106291

On 03/17/17 09:21 AM, Thiago Adams wrote:
> On Thursday, March 16, 2017 at 4:58:24 PM UTC-3, Thiago Adams wrote:
>> Having some struct;
>>
>> struct {
>>    char c;
>>    void* p;
>>    double d;
>> };
>>
>> Is it possible to parse it just knowing each type?
>> Something similar of var_arg but for structs.
>>
>> char c = arg(pointer, char);
>> void* p = arg(pointer, void*);
>> double d = arg(pointer, double);
>>
>> I was trying https://en.wikipedia.org/wiki/Data_structure_alignment
>>
>> for x86 this is working on visual studio to return the next pointer.
>>
>> #define NEXT(n)   ( (sizeof(n) + sizeof(int) - 1) & ~(sizeof(int) - 1) )
>>
>> For x64 not. Changing int for __int64 doesn't work as well.
>
> Something I also would like to understand is how malloc knows the
> alignment if it doesn't know the types? Does it always align with
> pointer size?

malloc is required to return an address "suitable for any object type". 
  C11 added max_align_t as an alias for the largest type used to 
determine this alignment..

> And how about aligned_alloc? Why do we need it?

To allocate a block on some externally specified boundary, greater than 
alignof(max_align_t).  Unix (and no doubt windows) systems already have 
system specific functions for doing this, such as getting a block of 
memory aligned to a page boundary.

-- 
Ian

[toc] | [prev] | [next] | [standalone]

#106309

On 03/16/2017 06:16 PM, Ian Collins wrote:
> On 03/17/17 09:21 AM, Thiago Adams wrote:
...
>> And how about aligned_alloc? Why do we need it?
> 
> To allocate a block on some externally specified boundary, greater than 
> alignof(max_align_t).  Unix (and no doubt windows) systems already have 
> system specific functions for doing this, such as getting a block of 
> memory aligned to a page boundary.

"The value of alignment shall be a valid alignment supported by the
implementation and the value of size shall be an integral multiple of
alignment." (7.22.3.1p2)

"Valid alignments include only those values returned by an _Alignof
expression for fundamental types, plus an additional
implementation-defined set of values, which may be empty." (6.2.8p4)

By definition, for every type, _Alignof(type) <= _Alignof(max_align_t).
Thus, all valid alignments greater than _Alignof(max_align_t) are
implementation-defined, and there need not be any. If there are
implementation-defined valid alignments greater _Alignof(max_align_t),
aligned_alloc() certainly could be used for the purpose you describe.
However, since there need not be any such alignments, I think it's more
reasonable to assume that the primary purpose of aligned_alloc() is to
allow the allocation of memory that is LESS strictly aligned than the
memory allocated by malloc().

[toc] | [prev] | [next] | [standalone]

#106311

On 03/17/17 03:09 PM, James Kuyper wrote:
> On 03/16/2017 06:16 PM, Ian Collins wrote:
>> On 03/17/17 09:21 AM, Thiago Adams wrote:
> ...
>>> And how about aligned_alloc? Why do we need it?
>>
>> To allocate a block on some externally specified boundary, greater than
>> alignof(max_align_t).  Unix (and no doubt windows) systems already have
>> system specific functions for doing this, such as getting a block of
>> memory aligned to a page boundary.
>
> "The value of alignment shall be a valid alignment supported by the
> implementation and the value of size shall be an integral multiple of
> alignment." (7.22.3.1p2)
>
> "Valid alignments include only those values returned by an _Alignof
> expression for fundamental types, plus an additional
> implementation-defined set of values, which may be empty." (6.2.8p4)
>
> By definition, for every type, _Alignof(type) <= _Alignof(max_align_t).
> Thus, all valid alignments greater than _Alignof(max_align_t) are
> implementation-defined, and there need not be any. If there are
> implementation-defined valid alignments greater _Alignof(max_align_t),
> aligned_alloc() certainly could be used for the purpose you describe.
> However, since there need not be any such alignments, I think it's more
> reasonable to assume that the primary purpose of aligned_alloc() is to
> allow the allocation of memory that is LESS strictly aligned than the
> memory allocated by malloc().

I have never used the existing Posix/Solaris memalign functions in that 
way, but I concede they can be.  On systems that support them, the are 
only 1 (on 64bit) and 2 (on 32bit) smaller alignments than 
alignof(max_align_t) where there are many more greater!  I have 
typically used page size alignment.

-- 
Ian

[toc] | [prev] | [next] | [standalone]

#106326

On 17/03/17 03:09, James Kuyper wrote:
> On 03/16/2017 06:16 PM, Ian Collins wrote:
>> On 03/17/17 09:21 AM, Thiago Adams wrote:
> ...
>>> And how about aligned_alloc? Why do we need it?
>>
>> To allocate a block on some externally specified boundary, greater than 
>> alignof(max_align_t).  Unix (and no doubt windows) systems already have 
>> system specific functions for doing this, such as getting a block of 
>> memory aligned to a page boundary.
> 
> "The value of alignment shall be a valid alignment supported by the
> implementation and the value of size shall be an integral multiple of
> alignment." (7.22.3.1p2)
> 
> "Valid alignments include only those values returned by an _Alignof
> expression for fundamental types, plus an additional
> implementation-defined set of values, which may be empty." (6.2.8p4)
> 
> By definition, for every type, _Alignof(type) <= _Alignof(max_align_t).
> Thus, all valid alignments greater than _Alignof(max_align_t) are
> implementation-defined, and there need not be any. If there are
> implementation-defined valid alignments greater _Alignof(max_align_t),
> aligned_alloc() certainly could be used for the purpose you describe.
> However, since there need not be any such alignments, I think it's more
> reasonable to assume that the primary purpose of aligned_alloc() is to
> allow the allocation of memory that is LESS strictly aligned than the
> memory allocated by malloc().
> 

It is perhaps /possible/ that aligned_alloc() could be used for getting
less strictly aligned memory, but it seems unlikely to me.  The only
real benefit would be to reduce memory consumption - maybe you want to
allocate a 32-bit "int" without wasting the space needed for alignment
if max_align_t is 64-bit.  However, such allocations usually already
have a good deal more than 32 bits of overhead to handle heap
management, "free", etc. - trying to reduce the alignment for an
allocation is going to give very little benefit unless you have a highly
specialised allocator.

Also, the way I read 7.22.3 and 7.22.3.1 of C11, the memory returned by
aligned_alloc() must be aligned according to its argument, /and/ to "any
type of object with a fundamental alignment requirement", just like
malloc().

On the other hand, there are many reasons you might want an alignment
that is larger than the fundamental alignment for a type.  Modern cpus
often have SIMD and vector instructions that can handle multiple
elements at a time, but require greater alignment for their data.
Aligning to cache lines can have a significant effect on speed.
Circular buffers can be implemented more efficiently with greater
alignment - on many devices, DMA hardware requires alignment for such
buffers.  Memory protection mechanisms (MMU or MPU) will often need
large alignments (these things are usually handled by the OS and system
calls - but /someone/ has to write that OS!).

[toc] | [prev] | [next] | [standalone]

#106351

On 03/17/2017 05:11 AM, David Brown wrote:
> On 17/03/17 03:09, James Kuyper wrote:
...
>> "The value of alignment shall be a valid alignment supported by the
>> implementation and the value of size shall be an integral multiple of
>> alignment." (7.22.3.1p2)
>>
>> "Valid alignments include only those values returned by an _Alignof
>> expression for fundamental types, plus an additional
>> implementation-defined set of values, which may be empty." (6.2.8p4)
>>
>> By definition, for every type, _Alignof(type) <= _Alignof(max_align_t).
>> Thus, all valid alignments greater than _Alignof(max_align_t) are
>> implementation-defined, and there need not be any. If there are
>> implementation-defined valid alignments greater _Alignof(max_align_t),
>> aligned_alloc() certainly could be used for the purpose you describe.
>> However, since there need not be any such alignments, I think it's more
>> reasonable to assume that the primary purpose of aligned_alloc() is to
>> allow the allocation of memory that is LESS strictly aligned than the
>> memory allocated by malloc().
...
> Also, the way I read 7.22.3 and 7.22.3.1 of C11, the memory returned by
> aligned_alloc() must be aligned according to its argument, /and/ to "any
> type of object with a fundamental alignment requirement", just like
> malloc().

You're right: I concentrated my attention on 7.22.3.1, without paying 
attention to 7.22.3. So, as far as strictly conforming code is 
concerned, aligned_alloc() is equivalent to malloc() - the only 
alignments that require it to behave differently are implementation-defined.

[toc] | [prev] | [next] | [standalone]

#106306

> Something I also would like to understand is how malloc knows the
> alignment if it doesn't know the types?

It uses the worst-case alignment of all possible types.  For example
on a wierd implementation like this:

	Shorts align on a multiple of 3.
	Pointers align on a multiple of 7.
	Longs align on a multiple of 8.
	Doubles align on a multiple of 9.

then a struct that *might* contain each of these can be aligned on
the Least Common Multiple of 3, 7, 8, and 9, which is aligning on
a multiple of 504.  malloc() can always get it right by aligning
to a multiple of 504 bytes even if it doesn't know what the types
are.

In actual practice, alignments are on a power of two on most machines
(since addresses are represented in binary, there's no excuse for
requiring alignment other than a power of 2), so you just take the
most restrictive alignment and use it.  Requiring an alignment of
4 or 8 on a 64-bit CPU doesn't waste nearly as much memory as
aligning to a multiple of 504.  Some compilers will align all
structures to the strictest alignment even though, for example,
character arrays do not *HAVE* to be aligned at all, for speed reasons.

Note:  alignment of a type is not necessarily a multiple of
sizeof(type), but it's probably not more strict than that.  It might
be less strict than that.  A 64-bit machine does not necessarily
have an 64-bit-wide data bus (it might be 32 bits wide) so a 64-bit
long may only need to be on a multiple of 4 bytes, not 8).  The
reason the width of the data bus matters is that when fetching data,
the CPU needs to either have circuitry to re-arrange the bytes, or
have permission to reject the request.  The circuitry comes with a
cost: more silicon, more power to run it, and more gate delays on
the bus.  Some machines don't require any alignment stricter than
being on a multiple of 1 byte.

> Does it always align with pointer size?

No.  But the largest basic objects:  pointers, integer types, and
floating types, tend to determine the worst-case alignment.  long
double, which on some machines takes up 10 bytes or 12 bytes with
only 10 actually used, typically has an alignment of at most 4.

> And how about aligned_alloc? Why do we need it?

What is that?  Not part of any standard I can think of?

Some parts of the architecture may require alignment greater than
what the CPU requires, like page alignment for DMA operations.  (A
page on an x86 architecture is typically 4096 bytes, but the MMU
also supports 4 MiB pages.) It may also be related to design
restrictions, such as "all shared memory segments must begin on a
page boundary", because the MMU only maps whole pages.

[toc] | [prev] | [next] | [standalone]

#106308

Note: as I've previously told you, permission to quote anything written
by me without proper attribution is denied.

On 03/16/2017 09:17 PM, Gordon Burditt wrote:
>> Something I also would like to understand is how malloc knows the
>> alignment if it doesn't know the types?
> 
> It uses the worst-case alignment of all possible types.  For example
> on a wierd implementation like this:
> 
> 	Shorts align on a multiple of 3.
> 	Pointers align on a multiple of 7.
> 	Longs align on a multiple of 8.
> 	Doubles align on a multiple of 9.
> 
> then a struct that *might* contain each of these can be aligned on
> the Least Common Multiple of 3, 7, 8, and 9, which is aligning on
> a multiple of 504.  malloc() can always get it right by aligning
> to a multiple of 504 bytes even if it doesn't know what the types
> are.
> 
> In actual practice, alignments are on a power of two on most machines
> (since addresses are represented in binary, there's no excuse for
> requiring alignment other than a power of 2), so you just take the

Your example is consistent with C99 and earlier, but not with C2011:
"Every valid alignment value shall be a nonnegative integral power of
two." (6.2.8p4)

...
> Note:  alignment of a type is not necessarily a multiple of
> sizeof(type), but it's probably not more strict than that.  It might

Because of the way arrays work, sizeof(type) must be an integer multiple
of _Alignof(type). The only way that _Alignof(type) could be an integer
multiple of sizeof(type) would be if that multiple is 1. Note: since
your other comments indicate a lack of familiarity with C2011, I'll
mention that _Alignof() is new in that version of the standard. However,
the above comment relies on _Alignof() only to simplify the writing.
Replace _Alignof(type) with "the alignment requirement of type", and
that statement is equally true of every version of the standard.

...
>> And how about aligned_alloc? Why do we need it?

It allows memory to be allocated with an alignment less strict than that
mandated for malloc().

> What is that?  Not part of any standard I can think of?

If so, then your understanding of the standard is at least 6 years out
of date. It's described in 7.22.3.1 of C2011. It's new in that version
of the standard.

[toc] | [prev] | [next] | [standalone]

#106284

On Thursday, March 16, 2017 at 2:58:24 PM UTC-5, Thiago Adams wrote:
> Having some struct;
> 
> struct {
>   char c;
>   void* p;
>   double d; 
> };
> 
> Is it possible to parse it just knowing each type?
> Something similar of var_arg but for structs.
> 
> char c = arg(pointer, char);
> void* p = arg(pointer, void*);
> double d = arg(pointer, double);

Nothing in the C Standard would impose sufficient restrictions on structure
layout to allow such a construct to be implemented in "portable" code.  It
could be implemented in a fashion that would work with the vast majority of
compilers by determining the required alignment for each type and adjusting
the "current scan" pointer according to that alignment, make a note of the
resulting address, adding the size of the return type to that pointer, and
then returning the object stored at the previous address.

Such code would fail of course on implementations which employ unusual layout
rules, but that could be guarded against via "offsetof" checks.  It might
also fail occasionally on implementations which assume that code will never
do any "interesting" kinds of address computations.  There are proposals to
add a concept of "pointer provenance" to C, which would essentially invite
a compiler to assume that pointers to certain objects will only be formed
in certain ways the compiler knows about.  Since some application fields have
no need for tricky address computations, and since such an assumption could
allow useful optimizations in cases where it holds, it would make sense for
the Standard to allow implementations targeting those application fields to
make such assumptions.  That wouldn't mean programs that do field address
computations should be viewed as "broken", but compiler writers may very
well view them that way anyhow.

[toc] | [prev] | [next] | [standalone]

#106300

supercat@casperkitty.com writes:

> On Thursday, March 16, 2017 at 2:58:24 PM UTC-5, Thiago Adams wrote:
>> Having some struct;
>> 
>> struct {
>>   char c;
>>   void* p;
>>   double d; 
>> };
>> 
>> Is it possible to parse it just knowing each type?
>> Something similar of var_arg but for structs.
>> 
>> char c = arg(pointer, char);
>> void* p = arg(pointer, void*);
>> double d = arg(pointer, double);
>
> Nothing in the C Standard would impose sufficient restrictions on structure
> layout to allow such a construct to be implemented in "portable" code.  It
> could be implemented in a fashion that would work with the vast majority of
> compilers by determining the required alignment for each type and adjusting
> the "current scan" pointer according to that alignment, make a note of the
> resulting address, adding the size of the return type to that pointer, and
> then returning the object stored at the previous address.
>
> Such code would fail of course on implementations which employ unusual layout
> rules, but that could be guarded against via "offsetof" checks.  It might
> also fail occasionally on implementations which assume that code will never
> do any "interesting" kinds of address computations.  There are proposals to
> add a concept of "pointer provenance" to C, which would essentially invite
> a compiler to assume that pointers to certain objects will only be formed
> in certain ways the compiler knows about.  Since some application fields have
> no need for tricky address computations, and since such an assumption could
> allow useful optimizations in cases where it holds, it would make sense for
> the Standard to allow implementations targeting those application fields to
> make such assumptions.  That wouldn't mean programs that do field address
> computations should be viewed as "broken", but compiler writers may very
> well view them that way anyhow.

I may be missing something in your argument, but I don't think it's
right. C requires that there be an alignment suitable for any object.

-- 
Lowell Gilbert, embedded/networking software engineer
	 http://be-well.ilk.org/~lowell/

[toc] | [prev] | [next] | [standalone]

#106332

On 17/03/2017 01:19, Stefan Ram wrote:
> Lowell Gilbert <lgusenet@be-well.ilk.org> writes:
>> I may be missing something in your argument, but I don't think it's
>> right. C requires that there be an alignment suitable for any object.
>
>   There is a speech by Antony Polukhin "C++14 Reflections -
>   Without Macros, Markup nor External Tooling", where he shows
>
> #include <iostream>
> #include "magic_get.hpp"
>
> struct complicated_struct
> { int i;
>   short s;
>   double d;
>   unsigned u; };
>
> int main()
> { using namespace pod_ops;
>
>   complicated_struct s { 1, 2, 3.0, 4 };
>
>   ::std::cout << "s == " << s << ::std::endl; }
>
>   . C++ usually /cannot/ print arbitrary structures, just like C.
>   But Antony somehow manages to make this program print

My little scripting language can do this [http://pastebin.com/qxXvYBd1]. 
Such things are not hard if a language decides to have them built in. 
C++ has possibly chosen the hard way.

> s == {1, 2, 3.0, 4}
>
>   , because he manages to "parse the struct". Just watch the
>   video in the web!
>
>   It uses template metaprogramming IIRC,

OK, it /has/ chosen the hard way...

> so it might not be
>   possible in C.

For structs consisting of basic types, then just a table of offsets and 
types are needed. A C compiler will already have this information; it's 
a shame it doesn't make it available to the program (the interface 
doesn't need to be elegant, but it can be as simple as the name of an 
extern table eg. __table_complicated_struct in some specified layout).

The same goes for lists of functions and their addresses.

It gets more fiddly for unions, nested anonymous structs, arrays, and 
pointers because a type representation scheme is needed.

Unions are a problem because a general purpose routine won't know which 
member is the 'active' one.

But some things can be done externally as I think has been mentioned.

-- 
bartc

[toc] | [prev] | [next] | [standalone]

#106999

Please excuse the lateness of this followup. Also, although this is
ostensibly replying to Lowell's post, it's actually directed to
supercat and Thiago.

Groovy hepcat Lowell Gilbert was jivin' in comp.lang.c on Fri, 17 Mar
2017 10:39 am. It's a cool scene! Dig it.

> supercat@casperkitty.com writes:
> 
>> On Thursday, March 16, 2017 at 2:58:24 PM UTC-5, Thiago Adams wrote:
>>> Having some struct;
>>> 
>>> struct {
>>>   char c;
>>>   void* p;
>>>   double d;
>>> };

  Of course you need a name for the struct, either a tag or a typedef
name. Otherwise there's not a great deal you can do with it.

>>> Is it possible to parse it just knowing each type?
>>> Something similar of var_arg but for structs.
>>> 
>>> char c = arg(pointer, char);
>>> void* p = arg(pointer, void*);
>>> double d = arg(pointer, double);
>>
>> Nothing in the C Standard would impose sufficient restrictions on
>> structure
>> layout to allow such a construct to be implemented in "portable"
>> code.

  Oh?

#include <stdio.h>
#include <string.h>
#include <assert.h>

struct foo
{
  char c;
  void *p;
  double d;
};

void *lookup_foo_member(struct foo *p, const char *type)
{
  void *r = NULL;

  if(0 == (strcmp(type, "char")))
    r = &(p->c);
  else if(0 == (strcmp(type, "void*")))
    r = &(p->p);
  else if(0 == (strcmp(type, "double")))
    r = &(p->d);

  assert(r != NULL);

  return r;
}

#define arg(p, t) (*(t*)lookup_foo_member(p, #t))

int main(void)
{
  struct foo x = {'a', NULL, 1.234};
  char c = arg(&x, char);
  void *p = arg(&x, void*);
  double d = arg(&x, double);

  printf("c = %c, p = %p, d = %f\n", c, p, d);
  return 0;
}

"Sufficient restrictions on structure layout" (whatever that means) has
nothing to do with it. Presumably you mean that the layout of struct
members plays some part in achieving the stated goal of determining
which struct member to access by its type. But, as I've just shown,
nothing could be further from the truth.

>>       It could be implemented in a fashion that would work with the 
>> vast majority of compilers by determining the required alignment for
>> each type and adjusting the "current scan" pointer according to that
>> alignment, make a note of the resulting address, adding the size of
>> the return type to that pointer, and then returning the object stored
>> at the previous address.

  Um..., what?

-- 


----- Dig the NEW and IMPROVED news sig!! -----


-------------- Shaggy was here! ---------------
              Ain't I'm a dawg!!

[toc] | [prev] | [next] | [standalone]

#107006

On Friday, March 24, 2017 at 3:40:55 PM UTC-5, Peter "Shaggy" Haywood wrote:
> Please excuse the lateness of this followup. Also, although this is
> ostensibly replying to Lowell's post, it's actually directed to
> supercat and Thiago.

[supercat said]
> >> Nothing in the C Standard would impose sufficient restrictions on
> >> structure
> >> layout to allow such a construct to be implemented in "portable"
> >> code.
> 
>   Oh?
> 
> void *lookup_foo_member(struct foo *p, const char *type)
> {
>   void *r = NULL;
> 
>   if(0 == (strcmp(type, "char")))
>     r = &(p->c);
>   else if(0 == (strcmp(type, "void*")))
>     r = &(p->p);
>   else if(0 == (strcmp(type, "double")))
>     r = &(p->d);
> 
>   assert(r != NULL);
> 
>   return r;
> }

That construct is hard-coded for the fact that "foo" has one "char" named
c, one "void*" named "p", and one "double" named "d".  The whole point of
using the aforementioned technique in general is that it makes it practical
to have code read and write structures of arbitrary types to some kind of
stream, without the reading/writing code having to know anything about the
structures being read or written.  The source or recipient of the streams
would have to know about the structure layout of course, but the code that
does the I/O wouldn't.

> "Sufficient restrictions on structure layout" (whatever that means) has
> nothing to do with it. Presumably you mean that the layout of struct
> members plays some part in achieving the stated goal of determining
> which struct member to access by its type. But, as I've just shown,
> nothing could be further from the truth.

Given its type, and the order and types of all preceding struct members.

> >>       It could be implemented in a fashion that would work with the 
> >> vast majority of compilers by determining the required alignment for
> >> each type and adjusting the "current scan" pointer according to that
> >> alignment, make a note of the resulting address, adding the size of
> >> the return type to that pointer, and then returning the object stored
> >> at the previous address.
> 
>   Um..., what?

For compilers that layout structures in commonplace fashion, the following
will allow code to compute the offset of each struct member, if the sizes
and alignments of that member and all previous members are known.

    size_t get_ofs(size_t *scanOfs, size_t size, unsigned align)
    {
      size_t ofs;
      ofs = (*scanOfs + (align-1)) & ~(align-1); // Find next aligned offset
      *scanOfs = ofs+size;
      return ofs;
    }

Start by initializing a size_t called "scanOfs" to zero.  Then, for each
member of the structure, call get_ofs() passing the address of scanOfs,
the size of the item, and its required alignment.  The return value from
each call will be the offset of that item within the structure.  Note that
the code doesn't need to know anything about the type of the structure
other than the sizes and alignments of its members, in order.

The Standard doesn't require that compilers lay out structures in such a
way that a method could retrieve struct fields *without hard-coded logic
for every possible sequence of nested types*, so there would be no way to
adapt anything like the above in a way that would work on a compiler
designed to thwart such efforts.

[toc] | [prev] | [next] | [standalone]

#106310

Thiago Adams <thiago.adams@gmail.com> writes:
> Having some struct;
>
> struct {
>   char c;
>   void* p;
>   double d; 
> };
>
> Is it possible to parse it just knowing each type?
> Something similar of var_arg but for structs.
>
> char c = arg(pointer, char);
> void* p = arg(pointer, void*);
> double d = arg(pointer, double);

Not easily or portably.

Once upon a time, I worked on a project that parsed C declarations to
generate C code that would print out the sizes and offsets of all the
members of all the structures in a set of C headers.  The intent was to
use that information to generate code that could access struct members
by name at run time from raw data transmitted to a system with a
possibly different architecture.  (I don't have any of the source code.)

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#106336

Keith Thompson <kst-u@mib.org> writes:

>Once upon a time, I worked on a project that parsed C declarations to
>generate C code that would print out the sizes and offsets of all the
>members of all the structures in a set of C headers.  The intent was to
>use that information to generate code that could access struct members
>by name at run time from raw data transmitted to a system with a
>possibly different architecture.  (I don't have any of the source code.)

On x86/x86_64 systems (running Linux), we have the pahole(1) utility which
will produce a nice report showing the sizes and offsets of all the members
of all structures (and classes for C++) along with hints showing alignment
holes and regions where packing the struct may reduce the memory footprint.

For example:

struct lconv {
        char *                     decimal_point;        /*     0     8 */
        char *                     thousands_sep;        /*     8     8 */
        char *                     grouping;             /*    16     8 */
        char *                     int_curr_symbol;      /*    24     8 */
        char *                     currency_symbol;      /*    32     8 */
        char *                     mon_decimal_point;    /*    40     8 */
        char *                     mon_thousands_sep;    /*    48     8 */
        char *                     mon_grouping;         /*    56     8 */
        /* --- cacheline 1 boundary (64 bytes) --- */
        char *                     positive_sign;        /*    64     8 */
        char *                     negative_sign;        /*    72     8 */
        char                       int_frac_digits;      /*    80     1 */
        char                       frac_digits;          /*    81     1 */
        char                       p_cs_precedes;        /*    82     1 */
        char                       p_sep_by_space;       /*    83     1 */
        char                       n_cs_precedes;        /*    84     1 */
        char                       n_sep_by_space;       /*    85     1 */
        char                       p_sign_posn;          /*    86     1 */
        char                       n_sign_posn;          /*    87     1 */
        char                       int_p_cs_precedes;    /*    88     1 */
        char                       int_p_sep_by_space;   /*    89     1 */
        char                       int_n_cs_precedes;    /*    90     1 */
        char                       int_n_sep_by_space;   /*    91     1 */
        char                       int_p_sign_posn;      /*    92     1 */
        char                       int_n_sign_posn;      /*    93     1 */

        /* size: 96, cachelines: 2, members: 24 */
        /* padding: 2 */
        /* last cacheline: 32 bytes */
};

Quite useful when micro-optimizing for optimal cache line usage (by moving
frequently accessed fields together such that they occupy a single cache-line).

[toc] | [prev] | [next] | [standalone]

#106335

Thiago Adams <thiago.adams@gmail.com> writes:
>Having some struct;
>
>struct {
>  char c;
>  void* p;
>  double d; 
>};
>
>Is it possible to parse it just knowing each type?
>Something similar of var_arg but for structs.
>
>char c = arg(pointer, char);
>void* p = arg(pointer, void*);
>double d = arg(pointer, double);
>
>I was trying https://en.wikipedia.org/wiki/Data_structure_alignment
>
>for x86 this is working on visual studio to return the next pointer.
>
>#define NEXT(n)   ( (sizeof(n) + sizeof(int) - 1) & ~(sizeof(int) - 1) )
>
>For x64 not. Changing int for __int64 doesn't work as well.
>
>

Generally fields are aligned on the natural boundary.    A pointer is
64-bits on x86_64, so therefore, the pointer is aligned on a 64-bit
boundary, not a 32-bit boundary which your NEXT macro assumes.

[toc] | [prev] | [next] | [standalone]

#106347

On Thursday, March 16, 2017 at 4:58:24 PM UTC-3, Thiago Adams wrote:
> Having some struct;
> 
> struct {
>   char c;
>   void* p;
>   double d; 
> };
> 
> Is it possible to parse it just knowing each type?
> Something similar of var_arg but for structs.
> 
> char c = arg(pointer, char);
> void* p = arg(pointer, void*);
> double d = arg(pointer, double);
> 
> I was trying https://en.wikipedia.org/wiki/Data_structure_alignment
> 
> for x86 this is working on visual studio to return the next pointer.
> 
> #define NEXT(n)   ( (sizeof(n) + sizeof(int) - 1) & ~(sizeof(int) - 1) )
> 
> For x64 not. Changing int for __int64 doesn't work as well.


This one is working on x86 and x64. 

The previous macro was copied from va_arg x86 and I think on the stack the alignment was always the same as int.

I am comparing the offset with the calculated one.

#include <stdlib.h>
#include <stdio.h>
#include <stddef.h>

#define NEW_OFFSET(offset, type)  (((offset) + sizeof(type) - 1) & ~(sizeof(type) - 1))


int main()
{
	struct X {  

		char a1;
		char a2;
		void* a3;
		char a4;
	} x;

	int i = 0;

	i = offsetof(struct X, a1);
	printf("%d %d\n", i, (int) NEW_OFFSET(i,char));

	i = offsetof(struct X, a2);
	printf("%d %d\n", i, (int)NEW_OFFSET(i, char));

	i = offsetof(struct X, a3);
	printf("%d %d\n", i, (int)NEW_OFFSET(i, void*));

	i = offsetof(struct X, a4);
	printf("%d %d\n", i, (int)NEW_OFFSET(i, char));


	printf("\n");

    return 0;
}

For my specific problem, I can build the struct. (build the memory) and doing this probably the program is very portable.

I need to copy values to heap and then later to copy from heap to stack.

I am not sure if I must used the correct alignment or if it is optional.
To cast some memory for double for instance, does it need to be aligned?

[toc] | [prev] | [next] | [standalone]

#106548

On Thursday, March 16, 2017 at 4:58:24 PM UTC-3, Thiago Adams wrote:
> Having some struct;
> 
> struct {
>   char c;
>   void* p;
>   double d; 
> };
> 
> Is it possible to parse it just knowing each type?
> Something similar of var_arg but for structs.

In case someone need, this is the code that is working x64 x86.
It's based on va_args I found in VC++ headers for arm.
(they look more generic)

#include <stdio.h>
#include <stdlib.h>


typedef struct X {
	char a1;
	char a2;
	double a3;
	void* a4;
	char a5;
} X;
#define X_INIT {'a', 'b', 1.2, (void*)3 , 'c'}

#define _SLOTSIZEOF2(t)  ((sizeof(t) + sizeof(t) - 1) & ~(sizeof(t) - 1))
#define _APALIGN2(t,ap)  (((char*)0 - (ap)) & (sizeof(t) - 1))
#define arg(ap, t) (*(t*)((ap += _SLOTSIZEOF2(t) + _APALIGN2(t,ap)) - _SLOTSIZEOF2(t)))

void Use(void* pv)
{
	char* args = pv;
	char a1 = arg(args, char);
	char a2 = arg(args, char);
	double a3 = arg(args, double);
	void* a4 = arg(args, void*);
	char a5  = arg(args, char);
	printf("\n%c\n", (int)a5);
}

int main()
{
  X x = X_INIT;
  Use(&x);
  return 0;
}

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c

csiph-web