Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #35578 > unrolled thread

Struct with unaligned fields

Back to article view | Back to comp.lang.c

Contents

  Struct with unaligned fields "James Harris" <james.harris.1@gmail.com> - 2013-08-22 19:06 +0100
    Re: Struct with unaligned fields glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-08-22 18:18 +0000
    Re: Struct with unaligned fields James Kuyper <jameskuyper@verizon.net> - 2013-08-22 14:57 -0400
      Re: Struct with unaligned fields "James Harris" <james.harris.1@gmail.com> - 2013-08-22 21:05 +0100
        Re: Struct with unaligned fields James Kuyper <jameskuyper@verizon.net> - 2013-08-22 16:45 -0400
        Re: Struct with unaligned fields Keith Thompson <kst-u@mib.org> - 2013-08-22 15:12 -0700
      Re: Struct with unaligned fields glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-08-22 22:22 +0000
      Re: Struct with unaligned fields Les Cargill <lcargill99@comcast.com> - 2013-08-22 22:07 -0500
    Re: Struct with unaligned fields Stephen Sprunk <stephen@sprunk.org> - 2013-08-22 14:13 -0500
      Re: Struct with unaligned fields Les Cargill <lcargill99@comcast.com> - 2013-08-22 22:08 -0500
        Re: Struct with unaligned fields "James Harris" <james.harris.1@gmail.com> - 2013-08-23 09:36 +0100
          Re: Struct with unaligned fields Les Cargill <lcargill99@comcast.com> - 2013-08-23 12:48 -0500
      Re: Struct with unaligned fields "James Harris" <james.harris.1@gmail.com> - 2013-08-23 09:34 +0100
        Re: Struct with unaligned fields Stephen Sprunk <stephen@sprunk.org> - 2013-08-23 04:47 -0500
          Re: Struct with unaligned fields "James Harris" <james.harris.1@gmail.com> - 2013-08-23 15:31 +0100
            Re: Struct with unaligned fields Stephen Sprunk <stephen@sprunk.org> - 2013-08-23 13:31 -0500
              Re: Struct with unaligned fields "James Harris" <james.harris.1@gmail.com> - 2013-08-23 20:46 +0100
              Re: Struct with unaligned fields glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-08-23 20:48 +0000
    Re: Struct with unaligned fields Jorgen Grahn <grahn+nntp@snipabacken.se> - 2013-08-22 20:01 +0000
      Re: Struct with unaligned fields "James Harris" <james.harris.1@gmail.com> - 2013-08-23 12:54 +0100
        Re: Struct with unaligned fields Eric Sosman <esosman@comcast-dot-net.invalid> - 2013-08-23 08:17 -0400
        Re: Struct with unaligned fields Jorgen Grahn <grahn+nntp@snipabacken.se> - 2013-08-23 20:00 +0000
        Re: Struct with unaligned fields Tim Rentsch <txr@alumni.caltech.edu> - 2013-08-24 16:50 -0700
    Re: Struct with unaligned fields "BartC" <bc@freeuk.com> - 2013-08-22 21:59 +0100
      Re: Struct with unaligned fields Keith Thompson <kst-u@mib.org> - 2013-08-22 15:02 -0700
      Re: Struct with unaligned fields Tim Rentsch <txr@alumni.caltech.edu> - 2013-08-24 16:42 -0700
    Re: Struct with unaligned fields Siri Cruise <chine.bleu@yahoo.com> - 2013-08-22 14:21 -0700
    Re: Struct with unaligned fields Eric Sosman <esosman@comcast-dot-net.invalid> - 2013-08-22 17:23 -0400
      Re: Struct with unaligned fields "James Harris" <james.harris.1@gmail.com> - 2013-08-23 13:03 +0100
        Re: Struct with unaligned fields Eric Sosman <esosman@comcast-dot-net.invalid> - 2013-08-23 08:43 -0400
          Re: Struct with unaligned fields "James Harris" <james.harris.1@gmail.com> - 2013-08-23 20:20 +0100
            Re: Struct with unaligned fields James Kuyper <jameskuyper@verizon.net> - 2013-08-23 16:15 -0400
            Re: Struct with unaligned fields Ian Collins <ian-news@hotmail.com> - 2013-08-24 11:47 +1200
              Re: Struct with unaligned fields "James Harris" <james.harris.1@gmail.com> - 2013-08-24 14:01 +0100
                Re: Struct with unaligned fields Robert Wessel <robertwessel2@yahoo.com> - 2013-08-24 14:26 -0500
                  Re: Struct with unaligned fields Ian Collins <ian-news@hotmail.com> - 2013-08-25 08:55 +1200
                    Re: Struct with unaligned fields Robert Wessel <robertwessel2@yahoo.com> - 2013-08-25 04:17 -0500
                      Re: Struct with unaligned fields Ian Collins <ian-news@hotmail.com> - 2013-08-25 21:22 +1200
                  Re: Struct with unaligned fields Stephen Sprunk <stephen@sprunk.org> - 2013-08-24 16:16 -0500
                    Re: Struct with unaligned fields Robert Wessel <robertwessel2@yahoo.com> - 2013-08-25 04:43 -0500
                    Re: Struct with unaligned fields David Thompson <dave.thompson2@verizon.net> - 2013-08-28 22:27 -0400
                      Re: Struct with unaligned fields Stephen Sprunk <stephen@sprunk.org> - 2013-08-29 14:30 -0500
                        Re: Struct with unaligned fields Robert Wessel <robertwessel2@yahoo.com> - 2013-08-29 16:41 -0500
                          Re: Struct with unaligned fields Stephen Sprunk <stephen@sprunk.org> - 2013-09-01 23:13 -0500
                            Re: Struct with unaligned fields Richard Damon <Richard@Damon-Family.org> - 2013-09-02 09:48 -0400
                          Re: Struct with unaligned fields David Thompson <dave.thompson2@verizon.net> - 2013-09-04 00:01 -0400
        Re: Struct with unaligned fields "BartC" <bc@freeuk.com> - 2013-08-23 23:55 +0100
    Re: Struct with unaligned fields Les Cargill <lcargill99@comcast.com> - 2013-08-22 22:05 -0500
      Re: Struct with unaligned fields Stephen Sprunk <stephen@sprunk.org> - 2013-08-23 00:29 -0500
        Re: Struct with unaligned fields Les Cargill <lcargill99@comcast.com> - 2013-08-23 12:53 -0500
          Re: Struct with unaligned fields Stephen Sprunk <stephen@sprunk.org> - 2013-08-23 15:15 -0500
            Re: Struct with unaligned fields Les Cargill <lcargill99@comcast.com> - 2013-08-23 22:14 -0500
              Re: Struct with unaligned fields Ian Collins <ian-news@hotmail.com> - 2013-08-24 16:24 +1200
                Re: Struct with unaligned fields Les Cargill <lcargill99@comcast.com> - 2013-08-23 23:47 -0500
              Re: Struct with unaligned fields Keith Thompson <kst-u@mib.org> - 2013-08-24 01:16 -0700
                Re: Struct with unaligned fields Les Cargill <lcargill99@comcast.com> - 2013-08-24 09:45 -0500
                  Re: Struct with unaligned fields Keith Thompson <kst-u@mib.org> - 2013-08-24 13:52 -0700
                    Re: Struct with unaligned fields Tim Prince <tprince@computer.org> - 2013-08-24 17:07 -0400
                    Re: Struct with unaligned fields Les Cargill <lcargill99@comcast.com> - 2013-08-24 16:35 -0500
                      Re: Struct with unaligned fields Ian Collins <ian-news@hotmail.com> - 2013-08-25 09:51 +1200
                      Re: Struct with unaligned fields Keith Thompson <kst-u@mib.org> - 2013-08-24 16:04 -0700
                        Re: Struct with unaligned fields Les Cargill <lcargill99@comcast.com> - 2013-08-24 21:03 -0500
            Re: Struct with unaligned fields Eric Sosman <esosman@comcast-dot-net.invalid> - 2013-08-24 08:22 -0400
    Re: Struct with unaligned fields Nobody <nobody@nowhere.com> - 2013-08-23 17:51 +0100
    Re: Struct with unaligned fields falk@rahul.net (Edward A. Falk) - 2013-08-24 01:43 +0000

Page 3 of 4 — ← Prev page 1 2 [3] 4  Next page →

#36056

On Sat, 24 Aug 2013 16:16:08 -0500, Stephen Sprunk
<stephen@sprunk.org> wrote:

> On 24-Aug-13 14:26, Robert Wessel wrote:

> > class nbou4     // Unsigned, four bytes
> >     {
> >   private:
> >     unsigned char a, b, c, d;
> >   public:
> >     operator unsigned long()     <snip << and | >
> >     nbou4& operator=(unsigned long l)     <snip >> and &>
> >     };
> 
> Ah, I figured this had to be possible but didn't think of overloading
> those particular operators; that's clever.
> 
> > This depends on eight bits bytes, four byte longs,
> 
> Why not make that dependency explicit by using uint8_t and uint32_t
> instead of unsigned char and unsigned long?  That also has a side
> benefit of dealing with machines that have a 64-bit unsigned long.
> 
Yes and yes.

> > and that the compiler not insert any alignment between the items,
> > so that the individual chars within the item are actually contiguous
> > with the preceding and following items.
> > 
> > Certainly not 100% portable, but it wil work on many implementations.
> 
> Lack of padding is not guaranteed, true, but alignon(nbou4) is almost
> certainly 1, so why would an implementation insert padding?  I've never
> heard of one that would.
> 
The original 16-bit Tandem NonStop, and I believe its predecessor
HP3000, had different format pointers to byte (char) and everything
larger (word=2bytes or more). They used 'word' pointers for structs,
so all structs had to be word = 2-byte aligned. This only mattered for
struct-in-struct, because all top-level variables were word-aligned
anyway. When they extended to 32-bit NonStop II aka NS2 they added
32-bit all-byte pointers, but included the original instructions and
pointers, retronymed NS1+ (the plus was some marketroid's invention,
nothing was actually added), so to allow pointer conversions to work
as long as you stayed in the 16-bit addressable area (essentially the
stack and "low" static and heap, the compiler default and thus an easy
rule to follow) structs still had to be 2-byte aligned.

"NS1+" and NS2 hardware is gone now, but TNS was mostly used for
enterprise-critical systems like airline reservations and bank ATMs.
The owners of these systems are reluctant and slow to make changes
whose failure might cost billions of dollars, so successor hardware
today -- commodity byte RISC -- can still run 30-year-old NS2 and even
NS1 code, nearly matching IBM's better-known S/360 clan.

[toc] | [prev] | [next] | [standalone]

#36089

On 28-Aug-13 21:27, David Thompson wrote:
> On Sat, 24 Aug 2013 16:16:08 -0500, Stephen Sprunk 
> <stephen@sprunk.org> wrote:
>> On 24-Aug-13 14:26, Robert Wessel wrote:
>>> and that the compiler not insert any alignment between the
>>> items, so that the individual chars within the item are actually
>>> contiguous with the preceding and following items.
>>> 
>>> Certainly not 100% portable, but it wil work on many
>>> implementations.
>> 
>> Lack of padding is not guaranteed, true, but alignon(nbou4) is
>> almost certainly 1, so why would an implementation insert padding?
>> I've never heard of one that would.
>
> The original 16-bit Tandem NonStop, and I believe its predecessor 
> HP3000, had different format pointers to byte (char) and everything 
> larger (word=2bytes or more). They used 'word' pointers for structs, 
> so all structs had to be word = 2-byte aligned. This only mattered
> for struct-in-struct, because all top-level variables were
> word-aligned anyway. [snip later history]

Ah; that's interesting.  Does the same hold for other word-addressed
systems, or is it peculiar to the above example?

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [next] | [standalone]

#36094

On Thu, 29 Aug 2013 14:30:30 -0500, Stephen Sprunk
<stephen@sprunk.org> wrote:

>On 28-Aug-13 21:27, David Thompson wrote:
>> On Sat, 24 Aug 2013 16:16:08 -0500, Stephen Sprunk 
>> <stephen@sprunk.org> wrote:
>>> On 24-Aug-13 14:26, Robert Wessel wrote:
>>>> and that the compiler not insert any alignment between the
>>>> items, so that the individual chars within the item are actually
>>>> contiguous with the preceding and following items.
>>>> 
>>>> Certainly not 100% portable, but it wil work on many
>>>> implementations.
>>> 
>>> Lack of padding is not guaranteed, true, but alignon(nbou4) is
>>> almost certainly 1, so why would an implementation insert padding?
>>> I've never heard of one that would.
>>
>> The original 16-bit Tandem NonStop, and I believe its predecessor 
>> HP3000, had different format pointers to byte (char) and everything 
>> larger (word=2bytes or more). They used 'word' pointers for structs, 
>> so all structs had to be word = 2-byte aligned. This only mattered
>> for struct-in-struct, because all top-level variables were
>> word-aligned anyway. [snip later history]
>
>Ah; that's interesting.  Does the same hold for other word-addressed
>systems, or is it peculiar to the above example?


Some word addressed machines used the same size pointer for chars, but
with a different internal format.  There were examples of both char
pointers having the character offset stuffed in at both ends of the
address.  IOW, both (8*WordAddress+CharOffset) and
(WordAddress+(CharOffset<<29)) happened (obviously assuming eight byte
words for this example), and the compiler would generate the code to
deal with that as necessary.

Pretty much all modern "word addressed" machines address words via
their effective byte addresses, IOW, the first word is at zero, the
second at 8, the third at 16, and the low bits are not used.  That
makes the char pointer format the same as other pointers, although
dereferencing them still requires extra work.

[toc] | [prev] | [next] | [standalone]

#36208

On 29-Aug-13 16:41, Robert Wessel wrote:
> On Thu, 29 Aug 2013 14:30:30 -0500, Stephen Sprunk 
> <stephen@sprunk.org> wrote:
>> On 28-Aug-13 21:27, David Thompson wrote:
>>> On Sat, 24 Aug 2013 16:16:08 -0500, Stephen Sprunk 
>>> <stephen@sprunk.org> wrote:
>>>> Lack of padding is not guaranteed, true, but alignon(nbou4) is 
>>>> almost certainly 1, so why would an implementation insert
>>>> padding? I've never heard of one that would.
>>> 
>>> The original 16-bit Tandem NonStop, and I believe its predecessor
>>> HP3000, had different format pointers to byte (char) and
>>> everything larger (word=2bytes or more). They used 'word'
>>> pointers for structs, so all structs had to be word = 2-byte
>>> aligned. This only mattered for struct-in-struct, because all
>>> top-level variables were word-aligned anyway. [snip later
>>> history]
>> 
>> Ah; that's interesting.  Does the same hold for other
>> word-addressed systems, or is it peculiar to the above example?
> 
> Some word addressed machines used the same size pointer for chars,
> but with a different internal format.  There were examples of both
> char pointers having the character offset stuffed in at both ends of
> the address. ...

I was aware of special char-offset pointers; I just hadn't heard of
structs (even ones with only char members) having to be aligned on a
word boundary.

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [next] | [standalone]

#36226

On 9/2/13 12:13 AM, Stephen Sprunk wrote:

> I was aware of special char-offset pointers; I just hadn't heard of
> structs (even ones with only char members) having to be aligned on a
> word boundary.
> 
> S
> 

The problem is that you can have a pointer to a struct that you don't
know the details of, so if the size of a pointer is a function of the
alignment of the struct, you need to either make all pointers the worse
case size or force all structs to have stricter alignment.

[toc] | [prev] | [next] | [standalone]

#36266

On Thu, 29 Aug 2013 16:41:54 -0500, Robert Wessel
<robertwessel2@yahoo.com> wrote:

> On Thu, 29 Aug 2013 14:30:30 -0500, Stephen Sprunk
> <stephen@sprunk.org> wrote:
> 
> >On 28-Aug-13 21:27, David Thompson wrote:
> >> On Sat, 24 Aug 2013 16:16:08 -0500, Stephen Sprunk 
> >> <stephen@sprunk.org> wrote:
> >>> On 24-Aug-13 14:26, Robert Wessel wrote:
> >>>> and that the compiler not insert any alignment between the
> >>>> items, so that the individual chars within the item are actually
> >>>> contiguous with the preceding and following items.
> >>>> 
> >>>> Certainly not 100% portable, but it wil work on many
> >>>> implementations.
> >>> 
> >>> Lack of padding is not guaranteed, true, but alignon(nbou4) is
> >>> almost certainly 1, so why would an implementation insert padding?
> >>> I've never heard of one that would.
> >>
> >> The original 16-bit Tandem NonStop, and I believe its predecessor 
> >> HP3000, had different format pointers to byte (char) and everything 
> >> larger (word=2bytes or more). They used 'word' pointers for structs, 
> >> so all structs had to be word = 2-byte aligned. This only mattered
> >> for struct-in-struct, because all top-level variables were
> >> word-aligned anyway. [snip later history]
> >
To be clear re the below: TNS1 pointers were the same *size*, 16 bits.
Byte pointer was high 15 bits select word and low bit select byte,
word pointer was plain 16 bits, but the compilers+linker put variables
and the hardware put the (upward!) stack in 00000-77777; 100000-177777
was used by the language runtimes (things like FILE blocks in C) and
if needed (rarely) could also be manually allocated/managed.

> >Ah; that's interesting.  Does the same hold for other word-addressed
> >systems, or is it peculiar to the above example?
> 
> 
> Some word addressed machines used the same size pointer for chars, but
> with a different internal format.  There were examples of both char
> pointers having the character offset stuffed in at both ends of the
> address.  IOW, both (8*WordAddress+CharOffset) and
> (WordAddress+(CharOffset<<29)) happened (obviously assuming eight byte
> words for this example), and the compiler would generate the code to
> deal with that as necessary.
> 
PDP-10 (and PDP-6) had an interesting variant on this. It had 36-bit
word and most instructions accessed a word using a 18-bit address
(optionally with index/offset and indirect); one special group of
instructions accessed the left or right half-word (selected by the
instruction, at compile time); and one special group of instructions
could access *any* bit-field from 1 bit up to the whole word, using a
'byte pointer' with the word address in the right half like a normal
pointer, and the bit offset and bit width in the left half. Since C
requires that chars be at least 8 bits, fixed, and a u-char array
access all bits of memory, it could only use 9, 12, or 18 bit bytes.
AFAIK there was no C for PDP-10 back in the 1970s (when C and even
Unix were mostly viewed as a curious experiment); there was talk a few
years ago in alt.sys.pdp10 about some people doing a gcc target (there
are still a few hardware machines going, and also good simulators) and
I'm pretty sure they used 9.

> Pretty much all modern "word addressed" machines address words via
> their effective byte addresses, IOW, the first word is at zero, the
> second at 8, the third at 16, and the low bits are not used.  That
> makes the char pointer format the same as other pointers, although
> dereferencing them still requires extra work.

[toc] | [prev] | [next] | [standalone]

#35669

"James Harris" <james.harris.1@gmail.com> wrote in message 
news:kv7j0j$a96$1@dont-email.me...

> 2. The structure could well come from a ramdisk.

Even with ramdisk, there will be extra overheads accessing such data, 
compared with normal memory, through having to use a file system, looking up 
filenames, locating the sectors, and, usually, transferring the data of each 
sector or cluster. In addition, the application might be doing its own 
accesses to the final data.

So the overheads of unpacking a few bytes from the FAT should in most cases 
be insignificant.

> 3. A FAT header is just a particular example. There are other memory 
> layouts that would need to be treated in the same way. For instance, the 
> BIOS Data Area which is present on every PC at boot time is a design from 
> the same era and has some simlilarly badly aligned elements (not that I 
> need to access the BDA).

That sounds like something else that doesn't need to be accessed that often. 
But since it is specific to a PC (and hence x86) the misalignments can be 
tolerated.

-- 
Bartc 

[toc] | [prev] | [next] | [standalone]

#35603

James Harris wrote:
> A file allocation table is an on-disk structure that was designed in the
> days before people took much care with field alignment. FAT file systems are
> still prevalent today. I need to write some code to work with such
> structures so have to deal with fields that will not be aligned.
>
> To illustrate, the FAT header starts like this:
>
>    8-byte BS_OEMName
>    2-byte BPB_BytsPerSec
>    1-byte BPB_SecPerClus
>    2-byte BPB_RsvdSecCnt
>    ...
>
> As you can see, it is impossible to align the last of those fields,
> BPB_RsvdSecCnt.

It is not impossible.

> The same is true of some of the fields that follow. (For
> anyone who is interested the full list of fields can be seen in a document
> called fatgen103. Copies are freely available on the internet.)
>
> I've seen the c faq at http://c-faq.com/struct/padding.html and know that
> there are no ideal solutions. This post is to ask for suggestions as to how
> to address this well and portably.
>
> I could do something basic and slow but what makes this an interesting
> challenge is that: 1) many of the fields *will* be naturally aligned so
> don't need any special access mechanism and 2) some of the machines the code
> will run on will support unaligned accessed and some will not.

GNU in general seems to always support #pragma pack. This
includes a push/pop  semantic that would be ideal for bracketing struct
declarations with.

http://gcc.gnu.org/onlinedocs/gcc/Structure_002dPacking-Pragmas.html

There still may be endian issues I haven't thought through.

> It would be
> good to use the same C source to run on any type of machine and get the
> compiler to emit the faster code where it can.
>
> Anyone already addressed the same sort of issue? I have some ideas but is
> there a standard approach?
>

I have yet to see a case where I could not control packing
properly, so long as the compiler has furniture for it.

Still, it may be better to make something table driven. You would
possibly need a new table for each architecture. And I would not
be above a general serialization routine to do the right memmove()
and memcpy() operations to pack an unpacked struct from a chunk of bytes.

> James
>
>

--
Les Cargill

[toc] | [prev] | [next] | [standalone]

#35608

On 22-Aug-13 22:05, Les Cargill wrote:
> James Harris wrote:
>> I've seen the c faq at http://c-faq.com/struct/padding.html and
>> know that there are no ideal solutions. This post is to ask for
>> suggestions as to how to address this well and portably.
>> 
>> I could do something basic and slow but what makes this an
>> interesting challenge is that: 1) many of the fields *will* be
>> naturally aligned so don't need any special access mechanism and 2)
>> some of the machines the code will run on will support unaligned
>> accessed and some will not.
> 
> GNU in general seems to always support #pragma pack. This includes a
> push/pop  semantic that would be ideal for bracketing struct 
> declarations with.
> 
> http://gcc.gnu.org/onlinedocs/gcc/Structure_002dPacking-Pragmas.html

That may not solve the unaligned access issue, though.

> There still may be endian issues I haven't thought through.

FAT originated on x86, so all multibyte fields are little-endian.

While POSIX provides convenient functions to convert to/from big-endian
form, which one can count on to be highly optimized (e.g. to no-ops on
big-endian machines), I'm not aware of any corresponding standardized
functions for little-endian form.  Performance is unlikely to be a real
problem in light of glacial disk I/O speeds, though.

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [next] | [standalone]

#35648

Stephen Sprunk wrote:
> On 22-Aug-13 22:05, Les Cargill wrote:
>> James Harris wrote:
>>> I've seen the c faq at http://c-faq.com/struct/padding.html and
>>> know that there are no ideal solutions. This post is to ask for
>>> suggestions as to how to address this well and portably.
>>>
>>> I could do something basic and slow but what makes this an
>>> interesting challenge is that: 1) many of the fields *will* be
>>> naturally aligned so don't need any special access mechanism and 2)
>>> some of the machines the code will run on will support unaligned
>>> accessed and some will not.
>>
>> GNU in general seems to always support #pragma pack. This includes a
>> push/pop  semantic that would be ideal for bracketing struct
>> declarations with.
>>
>> http://gcc.gnu.org/onlinedocs/gcc/Structure_002dPacking-Pragmas.html
>
> That may not solve the unaligned access issue, though.
>

I am rather desperately trying to figure out why that's a problem. :)
Even if it's a problem, assign the unaligned struct member to an
aligned copy and pass the pointer to that. You can even be lazy and do 
"crash directed development" :) to decide where this is needed.

You're just using the struct as an interface to the disk layout,
not anything else. It's for assigning from and assigning to,
nothing else.

>> There still may be endian issues I haven't thought through.
>
> FAT originated on x86, so all multibyte fields are little-endian.
>

Right. Which means there's a context in which GNU is used where that's a 
problem.

> While POSIX provides convenient functions to convert to/from big-endian
> form, which one can count on to be highly optimized (e.g. to no-ops on
> big-endian machines), I'm not aware of any corresponding standardized
> functions for little-endian form.  Performance is unlikely to be a real
> problem in light of glacial disk I/O speeds, though.
>
> S
>

--
Les Cargill

[toc] | [prev] | [next] | [standalone]

#35660

On 23-Aug-13 12:53, Les Cargill wrote:
> Stephen Sprunk wrote:
>> On 22-Aug-13 22:05, Les Cargill wrote:
>>> GNU in general seems to always support #pragma pack. This
>>> includes a push/pop  semantic that would be ideal for bracketing
>>> struct declarations with.
>>> 
>>> http://gcc.gnu.org/onlinedocs/gcc/Structure_002dPacking-Pragmas.html
>>
>> That may not solve the unaligned access issue, though.
> 
> I am rather desperately trying to figure out why that's a problem.
> :)

Some CPUs (e.g. most RISCs) don't allow unaligned accesses at all; if
you try it, the program crashes with a bus error.

Systems that merely have slow unaligned accesses aren't be worth
worrying about, unless you are doing it in a tight loop.

> Even if it's a problem, assign the unaligned struct member to an 
> aligned copy and pass the pointer to that.

That won't work, for the same reason that direct access to the unaligned
member won't work.  At some point, you need code that deals with the
unaligned accesses.

If you can afford an unpacked shadow copy, you could simply memcpy()
individual members between the two, but that assumes the original copy
won't change out from under you (is this a live disk?) _and_ there are
enough shadow copy accesses to make it worth the effort.

If you're doing an occasional access now and then to a live disk
structure, you're better off with accessor functions that realign the
data on the fly--one per field.

> You can even be lazy and do "crash directed development" :) to
> decide where this is needed.

Provided you have a system that actually does crash--and unit tests that
access every single field.

S

-- 
Stephen Sprunk         "God does not play dice."  --Albert Einstein
CCIE #3723         "God is an inveterate gambler, and He throws the
K5SSS        dice at every possible opportunity." --Stephen Hawking

[toc] | [prev] | [next] | [standalone]

#35673

Stephen Sprunk wrote:
> On 23-Aug-13 12:53, Les Cargill wrote:
>> Stephen Sprunk wrote:
>>> On 22-Aug-13 22:05, Les Cargill wrote:
>>>> GNU in general seems to always support #pragma pack. This
>>>> includes a push/pop  semantic that would be ideal for bracketing
>>>> struct declarations with.
>>>>
>>>> http://gcc.gnu.org/onlinedocs/gcc/Structure_002dPacking-Pragmas.html
>>>
>>> That may not solve the unaligned access issue, though.
>>
>> I am rather desperately trying to figure out why that's a problem.
>> :)
>
> Some CPUs (e.g. most RISCs) don't allow unaligned accesses at all; if
> you try it, the program crashes with a bus error.
>

Perhaps I have lived a charmed life, then.
I hate to be that guy, but can I have a concrete example?

please remember that even primitive 'C'  objects are an abstraction,
and that development tools can take some measure of pain to make
them more comfortable for us.

I have done this on ARM{7,9,10}, Coldfire, Intel from 16
through 64 bit, some 32 bit AVR or other and several MIPS ,
and it worked just fine.

I do not recall whether the compiler masked and shifted for you, but I 
expect it did, if any of those had this difficulty.

I would be somewhat disappointed in a GNU implementation that allowed
that pragma but the code crashed because of it. It is part of the GNU 
standard. I could see it #error-ing out if it could not be or was not
supported. But crashing would be very disappointing.

I have never seen a case where I could not develop a solution
that used bit fields and packed structs and demonstrate that to other 
team members.

So while I accept that you're not alone in this claim, I'd
have to see an actual example to believe it. And because this is a
much more civilized thing than legions of shifts and masks littering
actual code ( even as macros) :) , I'd start there anyway.


> Systems that merely have slow unaligned accesses aren't be worth
> worrying about, unless you are doing it in a tight loop.
>

Right.

>> Even if it's a problem, assign the unaligned struct member to an
>> aligned copy and pass the pointer to that.
>
> That won't work, for the same reason that direct access to the unaligned
> member won't work.

Then I would expect a compiler diagnostic from the "#pragma 
pack(push,1)" And see below how I tested this.

> At some point, you need code that deals with the
> unaligned accesses.
>

I'd rather get that from the toolset if I can. And - surprise - it's
always worked out.

> If you can afford an unpacked shadow copy, you could simply memcpy()
> individual members between the two, but that assumes the original copy
> won't change out from under you (is this a live disk?) _and_ there are
> enough shadow copy accesses to make it worth the effort.
>

Right. You'd only do it when you needed to.

> If you're doing an occasional access now and then to a live disk
> structure, you're better off with accessor functions that realign the
> data on the fly--one per field.
>

I can't disagree too much there. I've done that, too. If that were the 
case, I would then lean on something akin to C++ or make it all table 
driven.

I have also used scripting languages to generate test code and header
files to manage this.

>> You can even be lazy and do "crash directed development" :) to
>> decide where this is needed.
>
> Provided you have a system that actually does crash--and unit tests that
> access every single field.
>

This is not difficult. memmove() a memory pattern to an instance of the 
struct, then printf) every value in the struct.


I'd even capture the output, and diff that with the last
output. This was executed from the makefile. I don't trust
this, either - it's just worked out for me.

> S
>

--
Les Cargill

[toc] | [prev] | [next] | [standalone]

#35674

Les Cargill wrote:
> Stephen Sprunk wrote:
>> On 23-Aug-13 12:53, Les Cargill wrote:
>>> Stephen Sprunk wrote:
>>>> On 22-Aug-13 22:05, Les Cargill wrote:
>>>>> GNU in general seems to always support #pragma pack. This
>>>>> includes a push/pop  semantic that would be ideal for bracketing
>>>>> struct declarations with.
>>>>>
>>>>> http://gcc.gnu.org/onlinedocs/gcc/Structure_002dPacking-Pragmas.html
>>>>
>>>> That may not solve the unaligned access issue, though.
>>>
>>> I am rather desperately trying to figure out why that's a problem.
>>> :)
>>
>> Some CPUs (e.g. most RISCs) don't allow unaligned accesses at all; if
>> you try it, the program crashes with a bus error.
>>
>
> Perhaps I have lived a charmed life, then.
> I hate to be that guy, but can I have a concrete example?

SPARC.

I had to get some 68K code that relied on packed structs to run on a Sun 
box and it wasn't a trivial task.  The Sun compiler did have a 
misaligned option (which uses traps to call functions that use multiple 
reads, shits and masks) , but using cased the code to run like a three 
legged dog with a sore foot.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#35675

Ian Collins wrote:
> Les Cargill wrote:
>> Stephen Sprunk wrote:
>>> On 23-Aug-13 12:53, Les Cargill wrote:
>>>> Stephen Sprunk wrote:
>>>>> On 22-Aug-13 22:05, Les Cargill wrote:
>>>>>> GNU in general seems to always support #pragma pack. This
>>>>>> includes a push/pop  semantic that would be ideal for bracketing
>>>>>> struct declarations with.
>>>>>>
>>>>>> http://gcc.gnu.org/onlinedocs/gcc/Structure_002dPacking-Pragmas.html
>>>>>
>>>>> That may not solve the unaligned access issue, though.
>>>>
>>>> I am rather desperately trying to figure out why that's a problem.
>>>> :)
>>>
>>> Some CPUs (e.g. most RISCs) don't allow unaligned accesses at all; if
>>> you try it, the program crashes with a bus error.
>>>
>>
>> Perhaps I have lived a charmed life, then.
>> I hate to be that guy, but can I have a concrete example?
>
> SPARC.
>

Okay. I feel your pain, bro. It's over, man. It's over.

> I had to get some 68K code that relied on packed structs to run on a Sun
> box and it wasn't a trivial task.  The Sun compiler did have a
> misaligned option (which uses traps to call functions that use multiple
> reads, shits and masks) , but using cased the code to run like a three
> legged dog with a sore foot.
>

Now never let us speak of this again :)

--
Les Cargill

[toc] | [prev] | [next] | [standalone]

#35678

Les Cargill <lcargill99@comcast.com> writes:
> Stephen Sprunk wrote:
>> On 23-Aug-13 12:53, Les Cargill wrote:
>>> Stephen Sprunk wrote:
>>>> On 22-Aug-13 22:05, Les Cargill wrote:
>>>>> GNU in general seems to always support #pragma pack. This
>>>>> includes a push/pop  semantic that would be ideal for bracketing
>>>>> struct declarations with.
>>>>>
>>>>> http://gcc.gnu.org/onlinedocs/gcc/Structure_002dPacking-Pragmas.html
>>>>
>>>> That may not solve the unaligned access issue, though.
>>>
>>> I am rather desperately trying to figure out why that's a problem.
>>> :)
>>
>> Some CPUs (e.g. most RISCs) don't allow unaligned accesses at all; if
>> you try it, the program crashes with a bus error.
>
> Perhaps I have lived a charmed life, then.
> I hate to be that guy, but can I have a concrete example?
[...]

On a SPARC, this program:

#include <stdio.h>
int main(void)
{   
    struct foo {
        char c;
        int x;
    } __attribute__((packed));
    union u {
        struct foo f;
        long double align;
    };
    union u obj = { { 'a', 10 } };
    int *ptr = &obj.f.x;
    printf("obj.f.x = %d\n", obj.f.x);
    fflush(stdout);
    printf("*ptr = %d\n", *ptr);
    return 0;
}

produces this output:

ptr  = ffbff351
obj.f.x = 10
Bus error

(I needed the union to force the `struct foo` object to be word-aligned;
without it, the compiler places it at an odd address, causing the `int x`
member to be word-aligned.)

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#35688

Keith Thompson wrote:
> Les Cargill <lcargill99@comcast.com> writes:
>> Stephen Sprunk wrote:
>>> On 23-Aug-13 12:53, Les Cargill wrote:
>>>> Stephen Sprunk wrote:
>>>>> On 22-Aug-13 22:05, Les Cargill wrote:
>>>>>> GNU in general seems to always support #pragma pack. This
>>>>>> includes a push/pop  semantic that would be ideal for bracketing
>>>>>> struct declarations with.
>>>>>>
>>>>>> http://gcc.gnu.org/onlinedocs/gcc/Structure_002dPacking-Pragmas.html
>>>>>
>>>>> That may not solve the unaligned access issue, though.
>>>>
>>>> I am rather desperately trying to figure out why that's a problem.
>>>> :)
>>>
>>> Some CPUs (e.g. most RISCs) don't allow unaligned accesses at all; if
>>> you try it, the program crashes with a bus error.
>>
>> Perhaps I have lived a charmed life, then.
>> I hate to be that guy, but can I have a concrete example?
> [...]
>
> On a SPARC, this program:
>
> #include <stdio.h>
> int main(void)
> {
>      struct foo {
>          char c;
>          int x;
>      } __attribute__((packed));
>      union u {
>          struct foo f;
>          long double align;
>      };
>      union u obj = { { 'a', 10 } };
>      int *ptr = &obj.f.x;
>      printf("obj.f.x = %d\n", obj.f.x);
>      fflush(stdout);
>      printf("*ptr = %d\n", *ptr);
>      return 0;
> }
>
> produces this output:
>
> ptr  = ffbff351
> obj.f.x = 10
> Bus error
>
> (I needed the union to force the `struct foo` object to be word-aligned;
> without it, the compiler places it at an odd address, causing the `int x`
> member to be word-aligned.)
>

Thanks, Keith. Such sadistic hardware!

--
Les Cargill

[toc] | [prev] | [next] | [standalone]

#35705

Les Cargill <lcargill99@comcast.com> writes:
> Keith Thompson wrote:
>> Les Cargill <lcargill99@comcast.com> writes:
>>> Stephen Sprunk wrote:
[...]       
>>>> Some CPUs (e.g. most RISCs) don't allow unaligned accesses at all; if
>>>> you try it, the program crashes with a bus error.
>>>
>>> Perhaps I have lived a charmed life, then.
>>> I hate to be that guy, but can I have a concrete example?
>> [...]
>>
>> On a SPARC, this program:
>>
[snip]
>>
>> produces this output:
>>
>> ptr  = ffbff351
>> obj.f.x = 10
>> Bus error
>>
>> (I needed the union to force the `struct foo` object to be word-aligned;
>> without it, the compiler places it at an odd address, causing the `int x`
>> member to be word-aligned.)
>
> Thanks, Keith. Such sadistic hardware!

Sadistic?  I don't think so.

My understanding is that supporting misaligned access requires
extra work in designing the CPU hardware.  Not supporting it left
more resources available, both in design effort and chip area,
to support other features (lots of registers, for example) and
better performance.

And 90+% of the time, keeping operands properly aligned isn't much
of a problem anyway; the compiler takes care of it for you.

Perhaps the tradeoffs have changed since SPARC was designed, and
supporting unaligned access makes more sense now.  But you can see
from my (snipped) example how convoluted your code has to be to
trigger a problem.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#35707

On 8/24/2013 4:52 PM, Keith Thompson wrote:
> Les Cargill <lcargill99@comcast.com> writes:
>> Keith Thompson wrote:
>>> Les Cargill <lcargill99@comcast.com> writes:
>>>> Stephen Sprunk wrote:
> [...]
>>>>> Some CPUs (e.g. most RISCs) don't allow unaligned accesses at all; if
>>>>> you try it, the program crashes with a bus error.
>>>>
>>>> Perhaps I have lived a charmed life, then.
>>>> I hate to be that guy, but can I have a concrete example?
>>> [...]
>>>
>>> On a SPARC, this program:
>>>
> [snip]
>>>
>>> produces this output:
>>>
>>> ptr  = ffbff351
>>> obj.f.x = 10
>>> Bus error
>>>
>>> (I needed the union to force the `struct foo` object to be word-aligned;
>>> without it, the compiler places it at an odd address, causing the `int x`
>>> member to be word-aligned.)
>>
>> Thanks, Keith. Such sadistic hardware!
>
> Sadistic?  I don't think so.
>
> My understanding is that supporting misaligned access requires
> extra work in designing the CPU hardware.  Not supporting it left
> more resources available, both in design effort and chip area,
> to support other features (lots of registers, for example) and
> better performance.
>
> And 90+% of the time, keeping operands properly aligned isn't much
> of a problem anyway; the compiler takes care of it for you.
>
> Perhaps the tradeoffs have changed since SPARC was designed, and
> supporting unaligned access makes more sense now.  But you can see
> from my (snipped) example how convoluted your code has to be to
> trigger a problem.
>
Several current compilers use split moves to handle what appear to be 
mis-aligned access, in case the hardware support may be even worse than 
using multiple instructions, even though neither choice will fail outright.
For example, initial implementations of AVX-256 perform better when 
mis-aligned 256-bit moves are split into 128-bit move instructions. It's 
impractical to sort out whether a newer hardware platform is running 
which would perform better with a 256-bit unaligned move.
Current 512-bit wide architectures support mis-alignment only by the 
compiler choosing split moves, at a performance penalty.

-- 
Tim Prince

[toc] | [prev] | [next] | [standalone]

#35710

Keith Thompson wrote:
> Les Cargill <lcargill99@comcast.com> writes:
>> Keith Thompson wrote:
>>> Les Cargill <lcargill99@comcast.com> writes:
>>>> Stephen Sprunk wrote:
> [...]
>>>>> Some CPUs (e.g. most RISCs) don't allow unaligned accesses at all; if
>>>>> you try it, the program crashes with a bus error.
>>>>
>>>> Perhaps I have lived a charmed life, then.
>>>> I hate to be that guy, but can I have a concrete example?
>>> [...]
>>>
>>> On a SPARC, this program:
>>>
> [snip]
>>>
>>> produces this output:
>>>
>>> ptr  = ffbff351
>>> obj.f.x = 10
>>> Bus error
>>>
>>> (I needed the union to force the `struct foo` object to be word-aligned;
>>> without it, the compiler places it at an odd address, causing the `int x`
>>> member to be word-aligned.)
>>
>> Thanks, Keith. Such sadistic hardware!
>
> Sadistic?  I don't think so.
>

I forgot to smiley. Here: :)

> My understanding is that supporting misaligned access requires
> extra work in designing the CPU hardware.

So they push this off to the toolchain. But the toolchain
doesn't support it...

> Not supporting it left
> more resources available, both in design effort and chip area,
> to support other features (lots of registers, for example) and
> better performance.
>
> And 90+% of the time, keeping operands properly aligned isn't much
> of a problem anyway; the compiler takes care of it for you.
>

I believe that was the original point - that GNU offers
"#prgama pack" and that I've see it work. Seems the best approach,
if it was available.

> Perhaps the tradeoffs have changed since SPARC was designed, and
> supporting unaligned access makes more sense now.  But you can see
> from my (snipped) example how convoluted your code has to be to
> trigger a problem.
>

Certainly. In this case, it appears to be a vestige of formats
developed under 8-bit machines.

--
Les Cargill

[toc] | [prev] | [next] | [standalone]

#35713

Les Cargill wrote:
> Keith Thompson wrote:
>> Les Cargill <lcargill99@comcast.com> writes:
>>> Keith Thompson wrote:
>>>> Les Cargill <lcargill99@comcast.com> writes:
>>>>> Stephen Sprunk wrote:
>> [...]
>>>>>> Some CPUs (e.g. most RISCs) don't allow unaligned accesses at all; if
>>>>>> you try it, the program crashes with a bus error.
>>>>>
>>>>> Perhaps I have lived a charmed life, then.
>>>>> I hate to be that guy, but can I have a concrete example?
>>>> [...]
>>>>
>>>> On a SPARC, this program:
>>>>
>> [snip]
>>>>
>>>> produces this output:
>>>>
>>>> ptr  = ffbff351
>>>> obj.f.x = 10
>>>> Bus error
>>>>
>>>> (I needed the union to force the `struct foo` object to be word-aligned;
>>>> without it, the compiler places it at an odd address, causing the `int x`
>>>> member to be word-aligned.)
>>>
>>> Thanks, Keith. Such sadistic hardware!
>>
>> Sadistic?  I don't think so.
>>
>
> I forgot to smiley. Here: :)
>
>> My understanding is that supporting misaligned access requires
>> extra work in designing the CPU hardware.
>
> So they push this off to the toolchain. But the toolchain
> doesn't support it...

In the case of SPARC, yes it does, but at a significant performance 
cost.  Even with the appropriate hardware support, misaligned access is 
still a performance killer.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

Page 3 of 4 — ← Prev page 1 2 [3] 4  Next page →

Back to top | Article view | comp.lang.c

csiph-web