Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #69929 > unrolled thread

[QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding

Back to article view | Back to comp.lang.c++

Contents

  [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-20 19:55 +0100
    Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-20 20:12 +0100
      Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Keith Thompson <kst-u@mib.org> - 2019-11-20 22:33 -0800
    Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-20 20:22 +0100
    Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Jorgen Grahn <grahn+nntp@snipabacken.se> - 2019-11-20 19:34 +0000
      Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-21 00:54 +0100
        Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Öö Tiib <ootiib@hot.ee> - 2019-11-20 16:27 -0800
        Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Manfred <invalid@add.invalid> - 2019-11-21 01:48 +0100
          Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-21 04:09 +0100
        Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding James Kuyper <jameskuyper@alumni.caltech.edu> - 2019-11-20 17:44 -0800
          Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-21 04:14 +0100
    Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding David Brown <david.brown@hesbynett.no> - 2019-11-20 22:03 +0100
      Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-21 01:00 +0100
        Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding David Brown <david.brown@hesbynett.no> - 2019-11-21 13:22 +0100
          Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Keith Thompson <kst-u@mib.org> - 2019-11-21 11:30 -0800
            Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding David Brown <david.brown@hesbynett.no> - 2019-11-21 22:54 +0100
    Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Paavo Helde <myfirstname@osa.pri.ee> - 2019-11-20 23:28 +0200
    Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding James Kuyper <jameskuyper@alumni.caltech.edu> - 2019-11-20 15:48 -0800
    Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Reinhardt Behm <rbehm@hushmail.com> - 2019-11-21 10:33 +0800
      Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding David Brown <david.brown@hesbynett.no> - 2019-11-21 13:25 +0100
    Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Keith Thompson <kst-u@mib.org> - 2019-11-20 22:32 -0800
      Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Keith Thompson <kst-u@mib.org> - 2019-11-20 22:43 -0800
        Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-21 12:53 +0100
          Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Paavo Helde <myfirstname@osa.pri.ee> - 2019-11-21 18:20 +0200
            Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-21 18:07 +0100
            Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> - 2019-11-21 17:34 +0000
            Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Christian Gollwitzer <auriocus@gmx.de> - 2019-11-22 07:26 +0100
              Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Keith Thompson <kst-u@mib.org> - 2019-11-22 00:37 -0800
                Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding David Brown <david.brown@hesbynett.no> - 2019-11-22 10:10 +0100
                  Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Öö Tiib <ootiib@hot.ee> - 2019-11-22 02:23 -0800
                    Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding David Brown <david.brown@hesbynett.no> - 2019-11-22 12:38 +0100
                      Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Manfred <noname@add.invalid> - 2019-11-22 17:13 +0100
                        Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding David Brown <david.brown@hesbynett.no> - 2019-11-22 18:48 +0100
                  Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding James Kuyper <jameskuyper@alumni.caltech.edu> - 2019-11-22 10:49 -0500
                    Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding James Kuyper <jameskuyper@alumni.caltech.edu> - 2019-11-22 10:52 -0500
                      Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Tim Rentsch <tr.17687@z991.linuxsc.com> - 2019-11-28 12:51 -0800
                Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-22 11:20 +0100
                  Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Keith Thompson <kst-u@mib.org> - 2019-11-22 10:00 -0800
              Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding David Brown <david.brown@hesbynett.no> - 2019-11-22 09:37 +0100
                Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Öö Tiib <ootiib@hot.ee> - 2019-11-22 01:37 -0800
                  Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding David Brown <david.brown@hesbynett.no> - 2019-11-22 11:40 +0100
                    Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Öö Tiib <ootiib@hot.ee> - 2019-11-22 04:55 -0800
                      Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding David Brown <david.brown@hesbynett.no> - 2019-11-22 15:18 +0100
                        Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Öö Tiib <ootiib@hot.ee> - 2019-11-22 07:23 -0800
                          Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding David Brown <david.brown@hesbynett.no> - 2019-11-22 16:34 +0100
                  Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding scott@slp53.sl.home (Scott Lurndal) - 2019-11-22 15:16 +0000
                    Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Öö Tiib <ootiib@hot.ee> - 2019-11-23 05:11 -0800
                  Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding James Kuyper <jameskuyper@alumni.caltech.edu> - 2019-11-22 10:56 -0500
                    Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Öö Tiib <ootiib@hot.ee> - 2019-11-23 07:03 -0800
                Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-22 11:31 +0100
                  Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding David Brown <david.brown@hesbynett.no> - 2019-11-22 12:23 +0100
                    Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-22 13:14 +0100
                      Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding David Brown <david.brown@hesbynett.no> - 2019-11-22 15:04 +0100
                        Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-22 19:35 +0100
                          Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding James Kuyper <jameskuyper@alumni.caltech.edu> - 2019-11-22 14:11 -0500
                            Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-23 00:25 +0100
                              Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding James Kuyper <jameskuyper@alumni.caltech.edu> - 2019-11-22 23:56 -0500
                                Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-23 11:12 +0100
                                  Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Paavo Helde <myfirstname@osa.pri.ee> - 2019-11-23 14:51 +0200
                                  Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding James Kuyper <jameskuyper@alumni.caltech.edu> - 2019-11-25 11:45 -0500
                          Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Keith Thompson <kst-u@mib.org> - 2019-11-22 22:04 -0800
                            Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-23 11:26 +0100
                              Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Keith Thompson <kst-u@mib.org> - 2019-11-23 12:27 -0800
                                Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-24 00:54 +0100
                                  Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Keith Thompson <kst-u@mib.org> - 2019-11-23 17:06 -0800
                                    Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-24 12:10 +0100
    Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Tim Rentsch <tr.17687@z991.linuxsc.com> - 2019-11-28 13:21 -0800

Page 3 of 4 — ← Prev page 1 2 [3] 4  Next page →

#69975

On 22/11/2019 10:37, Öö Tiib wrote:
> On Friday, 22 November 2019 10:37:59 UTC+2, David Brown  wrote:
>>
>> When you use "pragma pack", you are making code that is non-standard,
>> non-portable, and can cause trouble interacting with other code.  (What
>> happens when you mix pointers to a packed version of your struct, and a
>> non-packed version?  Are they compatible?  Will your compiler warn you
>> of mixups?  My guess is you don't know, and that is not a good thing.)
> 
> Both gcc and clang have -Waddress-of-packed-member but AFAIK you
> won't get that warning when compiling for platform on what it does not
> matter.

I wasn't saying that /I/ don't know the answer to these questions, or
that other more experienced developers don't know.  I was saying that
that folks suggesting "pack" here don't know the answers.  And that is
what matters - they are recommending a technique that they do not fully
understand.  (If they did understand it fully, they would understand
that it is completely unnecessary in this situation.)

For the gcc warnings:

"-Waddress-of-packed-member" warns when you take the address of a packed
member, regardless of alignment and regardless of whether the target
supports unaligned data.  It is enabled by default.

"-Wpacked" warns if you have made a struct "packed" but it does not
affect the size or layout of the struct.  ("packed" attribute will still
affect the alignment of the struct.)  It would trigger for code like the
the packed "pixel" struct, since the "packed" attribute does not affect
the structure.

Another useful warning here is "-Wpadded", which will tell you if a
struct has padding for alignment of members or the struct itself.  If
"-Wpadded" is not triggered for your structs, making them "packed" is
not going to be at all useful (unless you need to change their alignment).

> 
>> They can lead to inefficiencies - on some compilers and targets, packed
>> fields are accessed by byte.  Even when the fields are accessed by their
>> full size, packed means you can get misalignments, which can have a very
>> significant impact on efficiency in some situations.
> 
> Huh? Packed structs usually lead to raise of storage efficiency.

No, they don't - not usually, not significantly, and not helpfully.
/Occasionally/ they can help, but more often you can get close to the
same efficiency by simply re-arranging the members more sensibly.  (And
if you don't have enough members to make this work, you don't have
enough data for the memory efficiency to be relevant.)

> Say there is struct containing a char and double. Normally it is 16 bytes
> but packed it is  9 bytes. 43.75% of storage saved!
> Where we have to process large array of those there it is often impossible
> to predict without profiling on target platform if that will cause performance
> gain or loss.
> 

If memory efficiency or performance here is important, you would be
better off having two arrays - one of doubles, one of chars.  Sure, that
would involve a little extra code - but C++ excels in letting you write
wrappers so that you have a good clean interface while keeping the
internal structures intact.  And now you have all your doubles aligned,
giving significantly better cache usage and access speeds, and allowing
the possibility of SIMD vector instructions for massive improvements.
And since a structure like this is typically for a "valid, value" pair,
you now have 8 times the efficiency in dealing with invalid data over
the array since you only need to access the big doubles when the data is
valid.  And if you only actually needed a single bit (bool) of the char,
you could potentially get another factor of 8 here (depending on the
ratio of valid to invalid values).

Packed /can/ be useful, but very often it is used when it is unnecessary
or directly detrimental, and often there are better ways to deal with
the situation.

[toc] | [prev] | [next] | [standalone]

#69980

On Friday, 22 November 2019 12:40:21 UTC+2, David Brown  wrote:
> On 22/11/2019 10:37, Öö Tiib wrote:
> > On Friday, 22 November 2019 10:37:59 UTC+2, David Brown  wrote:
> > 
> >> They can lead to inefficiencies - on some compilers and targets, packed
> >> fields are accessed by byte.  Even when the fields are accessed by their
> >> full size, packed means you can get misalignments, which can have a very
> >> significant impact on efficiency in some situations.
> > 
> > Huh? Packed structs usually lead to raise of storage efficiency.
> 
> No, they don't - not usually, not significantly, and not helpfully.
> /Occasionally/ they can help, but more often you can get close to the
> same efficiency by simply re-arranging the members more sensibly.  (And
> if you don't have enough members to make this work, you don't have
> enough data for the memory efficiency to be relevant.)

But I did nowhere say that the effect of raised storage efficiency is
guaranteed to be significant or helpful anyhow? I only said that
there usually is that effect. Period. Lack of that effect is less usual.
Careful design and arrangement of data in way that padding is
minimal is effort and so is rarely invested. But what you say about
lot of members clearly contradicts with example that I brought in
next sentence.

> > Say there is struct containing a char and double. Normally it is 16 bytes
> > but packed it is  9 bytes. 43.75% of storage saved!
> > Where we have to process large array of those there it is often impossible
> > to predict without profiling on target platform if that will cause performance
> > gain or loss.
> > 
> 
> If memory efficiency or performance here is important, you would be
> better off having two arrays - one of doubles, one of chars.

It is third option that will save memory like packing on this example
It is worth to try out as well but unfortunately it is also not a silver
bullet.
It can even sometimes perform worse than packed structs depending
on platform it runs on, on context and what kind of processing is
performed on that data. Trying that alternative out may involve way
more significant refactorings than adding two pragmas.
 
> would involve a little extra code - but C++ excels in letting you write
> wrappers so that you have a good clean interface while keeping the
> internal structures intact.  And now you have all your doubles aligned,
> giving significantly better cache usage and access speeds, and allowing
> the possibility of SIMD vector instructions for massive improvements.

Those are also available only on some (I agree that common) platforms
and usefulness of those assumes that by "processing" we did mean
(I also agree that those are common) sequential for loops.

> And since a structure like this is typically for a "valid, value" pair,
> you now have 8 times the efficiency in dealing with invalid data over
> the array since you only need to access the big doubles when the data is
> valid.  And if you only actually needed a single bit (bool) of the char,
> you could potentially get another factor of 8 here (depending on the
> ratio of valid to invalid values).

Validity is more commonly encoded into values of doubles (in typical
64-bit IEEE 754 doubles there is huge supply of NaNs plus two Infs)
so I assumed that the char under question carries more than one bit
of information in it.

> Packed /can/ be useful, but very often it is used when it is unnecessary
> or directly detrimental, and often there are better ways to deal with
> the situation.

Correct. I wrote nowhere something that contradicts with it.
I only said that it is often impossible to predict without profiling on
target platform if the effect of those is positive or negative as
answer to your claim that these can lead only to significant
negative effects. Also it is often cheapest alternative to try
and test what the two pragmas added to code cause.

[toc] | [prev] | [next] | [standalone]

#69983

On 22/11/2019 13:55, Öö Tiib wrote:
> On Friday, 22 November 2019 12:40:21 UTC+2, David Brown  wrote:
>> On 22/11/2019 10:37, Öö Tiib wrote:
>>> On Friday, 22 November 2019 10:37:59 UTC+2, David Brown  wrote:
>>>
>>>> They can lead to inefficiencies - on some compilers and targets, packed
>>>> fields are accessed by byte.  Even when the fields are accessed by their
>>>> full size, packed means you can get misalignments, which can have a very
>>>> significant impact on efficiency in some situations.
>>>
>>> Huh? Packed structs usually lead to raise of storage efficiency.
>>
>> No, they don't - not usually, not significantly, and not helpfully.
>> /Occasionally/ they can help, but more often you can get close to the
>> same efficiency by simply re-arranging the members more sensibly.  (And
>> if you don't have enough members to make this work, you don't have
>> enough data for the memory efficiency to be relevant.)
> 
> But I did nowhere say that the effect of raised storage efficiency is
> guaranteed to be significant or helpful anyhow? I only said that
> there usually is that effect. Period. Lack of that effect is less usual.

Fair enough.

> Careful design and arrangement of data in way that padding is
> minimal is effort and so is rarely invested. 

I would expect anyone who cares about the storage efficiency to put in
that effort.

> But what you say about
> lot of members clearly contradicts with example that I brought in
> next sentence.

Yes, but see below about that one.

> 
>>> Say there is struct containing a char and double. Normally it is 16 bytes
>>> but packed it is  9 bytes. 43.75% of storage saved!
>>> Where we have to process large array of those there it is often impossible
>>> to predict without profiling on target platform if that will cause performance
>>> gain or loss.
>>>
>>
>> If memory efficiency or performance here is important, you would be
>> better off having two arrays - one of doubles, one of chars.
> 
> It is third option that will save memory like packing on this example
> It is worth to try out as well but unfortunately it is also not a silver
> bullet.

If programming was full of silver bullet solutions, it would not be
nearly as much fun :-)

> It can even sometimes perform worse than packed structs depending
> on platform it runs on, on context and what kind of processing is
> performed on that data. Trying that alternative out may involve way
> more significant refactorings than adding two pragmas.
>  
>> would involve a little extra code - but C++ excels in letting you write
>> wrappers so that you have a good clean interface while keeping the
>> internal structures intact.  And now you have all your doubles aligned,
>> giving significantly better cache usage and access speeds, and allowing
>> the possibility of SIMD vector instructions for massive improvements.
> 
> Those are also available only on some (I agree that common) platforms
> and usefulness of those assumes that by "processing" we did mean
> (I also agree that those are common) sequential for loops.

Of course.  It is only a possibility, not something that is always useful.

> 
>> And since a structure like this is typically for a "valid, value" pair,
>> you now have 8 times the efficiency in dealing with invalid data over
>> the array since you only need to access the big doubles when the data is
>> valid.  And if you only actually needed a single bit (bool) of the char,
>> you could potentially get another factor of 8 here (depending on the
>> ratio of valid to invalid values).
> 
> Validity is more commonly encoded into values of doubles (in typical
> 64-bit IEEE 754 doubles there is huge supply of NaNs plus two Infs)
> so I assumed that the char under question carries more than one bit
> of information in it.

I was thinking of something roughly like a std::optional<double>, but
where you want greater storage efficiency.

Of course there are endless use-cases for structures similar to this,
and many ways to handle it.  I am just showing some ideas that can often
be better than throwing "packed" into the mix (while still accepting
that /sometimes/ "packed" is the best choice).

> 
>> Packed /can/ be useful, but very often it is used when it is unnecessary
>> or directly detrimental, and often there are better ways to deal with
>> the situation.
> 
> Correct. I wrote nowhere something that contradicts with it.
> I only said that it is often impossible to predict without profiling on
> target platform if the effect of those is positive or negative as
> answer to your claim that these can lead only to significant
> negative effects. Also it is often cheapest alternative to try
> and test what the two pragmas added to code cause.
> 

It might be that there has been a little mixup about which claims
referred to which uses of "packed" (without saying who has mixed things
up, as it was likely to be me).  In general, I agree that "packed" may
have positive effects, negative effects, or no effects, and any effects
may be significant or insignificant.  I say there are usually, but not
always, alternatives that could do better.  But I agree that "packed" is
often an easy choice from the development viewpoint.

However, I would say it is almost always a bad choice to start off with
a design that /requires/ "packed" - the OP has no reason at all for
doing that.

The case where I said there could never be any positive effects, but
might be negative effects, was a poster who suggested "packing" a struct
consisting of 4 uint8_t members.  There, I think we can agree, it could
never lead to increased storage or runtime efficiency on any real
platform.  (There is, I believe, a very hypothetical possibility of the
struct having extra padding.)  And it is certainly possible that on some
platforms with some compilers, there could be negative effects.

[toc] | [prev] | [next] | [standalone]

#69987

On Friday, 22 November 2019 16:18:15 UTC+2, David Brown  wrote:
> On 22/11/2019 13:55, Öö Tiib wrote:
> > On Friday, 22 November 2019 12:40:21 UTC+2, David Brown  wrote:
> > 
> > Validity is more commonly encoded into values of doubles (in typical
> > 64-bit IEEE 754 doubles there is huge supply of NaNs plus two Infs)
> > so I assumed that the char under question carries more than one bit
> > of information in it.
> 
> I was thinking of something roughly like a std::optional<double>, but
> where you want greater storage efficiency.

There we slightly mixed up in our thinkings ... I myself imagined
something like array of "quantities" where double is value and char
selects its magnitude (or physical unit) from limited set of such.

> Of course there are endless use-cases for structures similar to this,
> and many ways to handle it.  I am just showing some ideas that can often
> be better than throwing "packed" into the mix (while still accepting
> that /sometimes/ "packed" is the best choice).

Yes, and splitting struct between different arrays can be OK for
storage optimization. But it can also increase cache misses
because locality of members of object is now gone ... needs to
be profiled.

> >> Packed /can/ be useful, but very often it is used when it is unnecessary
> >> or directly detrimental, and often there are better ways to deal with
> >> the situation.
> > 
> > Correct. I wrote nowhere something that contradicts with it.
> > I only said that it is often impossible to predict without profiling on
> > target platform if the effect of those is positive or negative as
> > answer to your claim that these can lead only to significant
> > negative effects. Also it is often cheapest alternative to try
> > and test what the two pragmas added to code cause.
> > 
> 
> It might be that there has been a little mixup about which claims
> referred to which uses of "packed" (without saying who has mixed things
> up, as it was likely to be me).  In general, I agree that "packed" may
> have positive effects, negative effects, or no effects, and any effects
> may be significant or insignificant.  I say there are usually, but not
> always, alternatives that could do better.  But I agree that "packed" is
> often an easy choice from the development viewpoint.
> 
> However, I would say it is almost always a bad choice to start off with
> a design that /requires/ "packed" - the OP has no reason at all for
> doing that.

I also replied to OP that his unsigned bitfields in anonymous unions in
packed structs can only harm compared to good old uint8_t.
 
> The case where I said there could never be any positive effects, but
> might be negative effects, was a poster who suggested "packing" a struct
> consisting of 4 uint8_t members.  There, I think we can agree, it could
> never lead to increased storage or runtime efficiency on any real
> platform.  (There is, I believe, a very hypothetical possibility of the
> struct having extra padding.)  And it is certainly possible that on some
> platforms with some compilers, there could be negative effects.

True. Benefits are only fictional but real bad effects
can emerge in practice. Such code looks confused and so
experienced reader becomes careful ... hmm I bet here
will be major crocodiles somewhere. ;)
  
 

[toc] | [prev] | [next] | [standalone]

#69989

On 22/11/2019 16:23, Öö Tiib wrote:
> On Friday, 22 November 2019 16:18:15 UTC+2, David Brown  wrote:
>> On 22/11/2019 13:55, Öö Tiib wrote:
>>> On Friday, 22 November 2019 12:40:21 UTC+2, David Brown  wrote:
>>>
>>> Validity is more commonly encoded into values of doubles (in typical
>>> 64-bit IEEE 754 doubles there is huge supply of NaNs plus two Infs)
>>> so I assumed that the char under question carries more than one bit
>>> of information in it.
>>
>> I was thinking of something roughly like a std::optional<double>, but
>> where you want greater storage efficiency.
> 
> There we slightly mixed up in our thinkings ... I myself imagined
> something like array of "quantities" where double is value and char
> selects its magnitude (or physical unit) from limited set of such.

OK.  The gains from having these as separate arrays (rather than packed
structs) might not be as much, but they could still be relevant.

> 
>> Of course there are endless use-cases for structures similar to this,
>> and many ways to handle it.  I am just showing some ideas that can often
>> be better than throwing "packed" into the mix (while still accepting
>> that /sometimes/ "packed" is the best choice).
> 
> Yes, and splitting struct between different arrays can be OK for
> storage optimization. But it can also increase cache misses
> because locality of members of object is now gone ... needs to
> be profiled.

You still have as much locality, especially when iterating over the
arrays, just split into two blocks.  Random access might be worse - you
will always have to pull in two cache lines, rather than only sometimes
needing to do so.  (But if you are doing mainly random access and speed
is important, unpacked structs will be best.)  Misalignment penalties
for the packed struct can vary from nothing to a great deal, depending
on the target platform.

Yes - you need to profile and test.

> 
>>>> Packed /can/ be useful, but very often it is used when it is unnecessary
>>>> or directly detrimental, and often there are better ways to deal with
>>>> the situation.
>>>
>>> Correct. I wrote nowhere something that contradicts with it.
>>> I only said that it is often impossible to predict without profiling on
>>> target platform if the effect of those is positive or negative as
>>> answer to your claim that these can lead only to significant
>>> negative effects. Also it is often cheapest alternative to try
>>> and test what the two pragmas added to code cause.
>>>
>>
>> It might be that there has been a little mixup about which claims
>> referred to which uses of "packed" (without saying who has mixed things
>> up, as it was likely to be me).  In general, I agree that "packed" may
>> have positive effects, negative effects, or no effects, and any effects
>> may be significant or insignificant.  I say there are usually, but not
>> always, alternatives that could do better.  But I agree that "packed" is
>> often an easy choice from the development viewpoint.
>>
>> However, I would say it is almost always a bad choice to start off with
>> a design that /requires/ "packed" - the OP has no reason at all for
>> doing that.
> 
> I also replied to OP that his unsigned bitfields in anonymous unions in
> packed structs can only harm compared to good old uint8_t.
>  
>> The case where I said there could never be any positive effects, but
>> might be negative effects, was a poster who suggested "packing" a struct
>> consisting of 4 uint8_t members.  There, I think we can agree, it could
>> never lead to increased storage or runtime efficiency on any real
>> platform.  (There is, I believe, a very hypothetical possibility of the
>> struct having extra padding.)  And it is certainly possible that on some
>> platforms with some compilers, there could be negative effects.
> 
> True. Benefits are only fictional but real bad effects
> can emerge in practice. Such code looks confused and so
> experienced reader becomes careful ... hmm I bet here
> will be major crocodiles somewhere. ;)
>   

[toc] | [prev] | [next] | [standalone]

#69985

=?UTF-8?B?w5bDtiBUaWli?= <ootiib@hot.ee> writes:
>On Friday, 22 November 2019 10:37:59 UTC+2, David Brown  wrote:
>> 
>> When you use "pragma pack", you are making code that is non-standard,
>> non-portable, and can cause trouble interacting with other code.  (What
>> happens when you mix pointers to a packed version of your struct, and a
>> non-packed version?  Are they compatible?  Will your compiler warn you
>> of mixups?  My guess is you don't know, and that is not a good thing.)
>
>Both gcc and clang have -Waddress-of-packed-member but AFAIK you
>won't get that warning when compiling for platform on what it does not
>matter.
>
>> They can lead to inefficiencies - on some compilers and targets, packed
>> fields are accessed by byte.  Even when the fields are accessed by their
>> full size, packed means you can get misalignments, which can have a very
>> significant impact on efficiency in some situations.
>
>Huh? Packed structs usually lead to raise of storage efficiency.
>Say there is struct containing a char and double. Normally it is 16 bytes
>but packed it is  9 bytes. 43.75% of storage saved!

Of course, accesses become less efficient (particularly if the
unaligned accesses trap to the kernel for resolution).

[toc] | [prev] | [next] | [standalone]

#70015

On Friday, 22 November 2019 17:17:07 UTC+2, Scott Lurndal  wrote:
> =?UTF-8?B?w5bDtiBUaWli?= <ootiib@hot.ee> writes:
> >On Friday, 22 November 2019 10:37:59 UTC+2, David Brown  wrote:
> >> 
> >> When you use "pragma pack", you are making code that is non-standard,
> >> non-portable, and can cause trouble interacting with other code.  (What
> >> happens when you mix pointers to a packed version of your struct, and a
> >> non-packed version?  Are they compatible?  Will your compiler warn you
> >> of mixups?  My guess is you don't know, and that is not a good thing.)
> >
> >Both gcc and clang have -Waddress-of-packed-member but AFAIK you
> >won't get that warning when compiling for platform on what it does not
> >matter.
> >
> >> They can lead to inefficiencies - on some compilers and targets, packed
> >> fields are accessed by byte.  Even when the fields are accessed by their
> >> full size, packed means you can get misalignments, which can have a very
> >> significant impact on efficiency in some situations.
> >
> >Huh? Packed structs usually lead to raise of storage efficiency.
> >Say there is struct containing a char and double. Normally it is 16 bytes
> >but packed it is  9 bytes. 43.75% of storage saved!
> 
> Of course, accesses become less efficient (particularly if the
> unaligned accesses trap to the kernel for resolution).

On such platform the compilers usually generate more code for
accessing packed structs ... something that they already do for
bit-fields. But yes, that is also factor, code may grow.

[toc] | [prev] | [next] | [standalone]

#69992

On 11/22/19 4:37 AM, Öö Tiib wrote:
> On Friday, 22 November 2019 10:37:59 UTC+2, David Brown  wrote:
...
>> They can lead to inefficiencies - on some compilers and targets, packed
>> fields are accessed by byte.  Even when the fields are accessed by their
>> full size, packed means you can get misalignments, which can have a very
>> significant impact on efficiency in some situations.
> 
> Huh? Packed structs usually lead to raise of storage efficiency.

He said efficiency, not "storage efficiency". If there's any difference
between the packed and unpacked version of a struct, the packed version
usually requires slower access. If that weren't the case, there'd be no
motive for the implementation to make the unpacked version different.

[toc] | [prev] | [next] | [standalone]

#70016

On Friday, 22 November 2019 17:56:32 UTC+2, James Kuyper  wrote:
> On 11/22/19 4:37 AM, Öö Tiib wrote:
> > On Friday, 22 November 2019 10:37:59 UTC+2, David Brown  wrote:
> ...
> >> They can lead to inefficiencies - on some compilers and targets, packed
> >> fields are accessed by byte.  Even when the fields are accessed by their
> >> full size, packed means you can get misalignments, which can have a very
> >> significant impact on efficiency in some situations.
> > 
> > Huh? Packed structs usually lead to raise of storage efficiency.
> 
> He said efficiency, not "storage efficiency". If there's any difference
> between the packed and unpacked version of a struct, the packed version
> usually requires slower access. If that weren't the case, there'd be no
> motive for the implementation to make the unpacked version different.

I agree that accessing packed versions is always slower. 
Accessing code may be bigger and slower and difference can be
significant or insignificant depending on platform and situation.

However here is slight controversy. Same data can be laid out
so it is more efficient for storage and communications but
different layout can be more efficient for processing.
On lot of cases it does not matter and best layout is what
is for people most easy to reason about.
However where it matters there it can be worth to transform
data between the layouts or it can be worth to use one
or other (or even some third) layout for it all time since
transformations have some cost too. Sometimes only profiling
of fully functional product can show if it affects overall
performance and how. 

[toc] | [prev] | [next] | [standalone]

#69974

Il 22/11/19 09:37, David Brown ha scritto:
> On 22/11/2019 07:26, Christian Gollwitzer wrote:
>> Am 21.11.19 um 17:20 schrieb Paavo Helde:
>>> On 21.11.2019 13:53, Soviet_Mario wrote:
>>>> My only requirement is that the plain native type BYTE used in gambas
>>>> code that calls the C++ static library and that expects a 8-bit wide
>>>> unsigned integral, find its matching type in the C++ code, as they need
>>>> to share bitmaps and transparency maps.
>>>
>>> This is easy, use unsigned char or std::uint8_t.
>>>
>>>> The bitmaps use 32-bit packed colors+alpha channel, so I was assuming no
>>>> difficulty in exchange.
>>>
>>> This is more tricky because the size of int or long varies much more
>>> than char. For example, long is 32 bits in 64-bit MSVC and 64 bits
>>> with 64-bit gcc on Linux.
>>>
>>> However, for your purposes std::uint32_t should work fine.
>>
>> Why not
>>
>> #pragma pack(1)
>> struct pixel {
>>      uint8_t red, green, blue, alpha;
>> }
>>
>> ? That would make more sense to me than uint32_t and bit-twiddling to
>> get the channel values.
>>
>> Christian
> 
> If you are using "pragma pack" (or gcc "packed" attribute), you are
> probably doing something wrong.  It can have some uses when dealing with
> file formats 

it is exactly this second case : two programs have to 
communicate and store picture files in a compact way (and if 
I'll manage to, even exchange such pictures loaded in 
memory) upon a single big binary "fread"  from disk or save 
with a single raw "fwrite" to disk.
I can't find it acceptable that the DISK layout and RAM 
layout may differ as to packing aspects.
Non only for the waste of space (a large waste as pictures 
could be some tens megapixels), but as in such case I would 
have to read/write only single elements at a time. 
Unacceptable (*).
I must be allowed to load and save fast and in a single 
shot, so 1:1 mapping RAM<=>DISK.
This requirement is underogatable, even if I agree with all 
other remarks you make afterwards ...


(*) well actually I do not make strictly a single load/save 
as stated. First I save/load a very small descriptor with 
WIDTH/HEIGHT stored as 16 bit integers, then, aware of the 
array size of the buffer, alloc and fread or fwrite down.

It's a very "stupid" non standard format, but I don't need 
to export this directly. When in gambas I save using its 
native functions. The C++ engine is just supposed to 
transform pictures and save in raw format.


> or network packets with unaligned fields.  But most of the
> cases I see - such as this one (and the OP's first suggestion) are
> misunderstandings and misuses.
> 
> When you use "pragma pack", you are making code that is non-standard,
> non-portable, and can cause trouble interacting with other code.  (What
> happens when you mix pointers to a packed version of your struct, and a
> non-packed version?  Are they compatible?  Will your compiler warn you
> of mixups?  My guess is you don't know, and that is not a good thing.)
> 
> They can lead to inefficiencies - on some compilers and targets, packed
> fields are accessed by byte.  Even when the fields are accessed by their
> full size, packed means you can get misalignments, which can have a very
> significant impact on efficiency in some situations.
> 
> They can lead to restrictions in the code.  For some tools, you can't
> take the address of a packed field, or you might see problems using it
> because the compiler assumes pointers are properly aligned and that
> might not be the case for packed fields.
> 
> In short, "packed" can lead to a lot of extra complications and
> muck-ups.  Perhaps the most insidious aspect is that /usually/ you don't
> see problems with them.  Things will work while you are writing /this/
> code, and testing it with /this/ compiler - the problems will turn up
> later when someone else gets trouble using the code in a different
> project, and won't know why.
> 
> So use "packed" only when you /really/ need to - when it is clearly the
> best solution to the problem.  And only when you /really/ understand
> what it does, and why it might be useful.
> 
> And clearly, in this case, it is utterly pointless.  Just write the
> struct directly - it is impossible for "pack" to be of any help here.
> And if you don't understand that, you certainly should not be using
> "pack".  (And you can ask here for more help in understanding, if this
> post is not sufficient.)
> 
> 


-- 
1) Resistere, resistere, resistere.
2) Se tutti pagano le tasse, le tasse le pagano tutti
Soviet_Mario - (aka Gatto_Vizzato)

[toc] | [prev] | [next] | [standalone]

#69977

On 22/11/2019 11:31, Soviet_Mario wrote:
> Il 22/11/19 09:37, David Brown ha scritto:
>> On 22/11/2019 07:26, Christian Gollwitzer wrote:
>>> Am 21.11.19 um 17:20 schrieb Paavo Helde:
>>>> On 21.11.2019 13:53, Soviet_Mario wrote:
>>>>> My only requirement is that the plain native type BYTE used in gambas
>>>>> code that calls the C++ static library and that expects a 8-bit wide
>>>>> unsigned integral, find its matching type in the C++ code, as they
>>>>> need
>>>>> to share bitmaps and transparency maps.
>>>>
>>>> This is easy, use unsigned char or std::uint8_t.
>>>>
>>>>> The bitmaps use 32-bit packed colors+alpha channel, so I was
>>>>> assuming no
>>>>> difficulty in exchange.
>>>>
>>>> This is more tricky because the size of int or long varies much more
>>>> than char. For example, long is 32 bits in 64-bit MSVC and 64 bits
>>>> with 64-bit gcc on Linux.
>>>>
>>>> However, for your purposes std::uint32_t should work fine.
>>>
>>> Why not
>>>
>>> #pragma pack(1)
>>> struct pixel {
>>>      uint8_t red, green, blue, alpha;
>>> }
>>>
>>> ? That would make more sense to me than uint32_t and bit-twiddling to
>>> get the channel values.
>>>
>>> Christian
>>
>> If you are using "pragma pack" (or gcc "packed" attribute), you are
>> probably doing something wrong.  It can have some uses when dealing with
>> file formats 
> 
> it is exactly this second case : two programs have to communicate and
> store picture files in a compact way (and if I'll manage to, even
> exchange such pictures loaded in memory) upon a single big binary
> "fread"  from disk or save with a single raw "fwrite" to disk.
> I can't find it acceptable that the DISK layout and RAM layout may
> differ as to packing aspects.
> Non only for the waste of space (a large waste as pictures could be some
> tens megapixels), but as in such case I would have to read/write only
> single elements at a time. Unacceptable (*).
> I must be allowed to load and save fast and in a single shot, so 1:1
> mapping RAM<=>DISK.
> This requirement is underogatable, even if I agree with all other
> remarks you make afterwards ...

You have a struct with byte data - making it packed is /useless/.  There
is /never/ any advantage in that.

If the structure contains data of different sizes, then first check if
there really are padding or alignment issues.  Most structures used for
file formats are "naturally aligned" - the people who designed the
structures are smart enough to make sure that every 2-byte data item is
at an even address, every 4-byte data item is at a multiple of 4
address, and so on.  There are exceptions, especially for some older
formats from the days of 16-bit cpus, but it is often the case.

And if it is the case here, then making it "packed" is useless at best,
and can easily be a significant disadvantage.

All of what you have written here is "premature optimisation" - it is
knee-jerk thinking "I've got a file structure, I must use packed" rather
than looking at what is actually relevant or helpful.  At the moment,
you haven't the faintest clue as to whether using a packed struct makes
a real and measurable difference that is worth the cost - even if we
assume the file format details are not aligned.  You haven't done any
timings of real-world usage of your system to see if this is a
bottleneck.  You haven't tested alternatives like "mmap", or existing
libraries.  You haven't compared the time it takes to load or save the
file with the time it takes to do your processing.  You haven't compared
the time it takes to pack or unpack the format to the time it takes to
read or write it from the disk.  You haven't compared the time the
processing takes for a "packed" struct to how it would run with an
unpacked struct.

(Yes, I realise I have made a lot of claims here about what you have and
haven't done.  From your posts, I think they are almost certainly true.
 If I am wrong there, let me know.)


If you want to tell me that using "pack" makes your development process
simpler and the code easier, that's fine - I'll believe that, and it is
a perfectly good reason for using it.  But if you try to tell me you are
doing it for run-time speed reasons, then you are wrong.  Measure first,
find out if it really is too slow, find your /real/ bottlenecks, /then/
start looking for ways to improve the throughput.

> 
> 
> (*) well actually I do not make strictly a single load/save as stated.
> First I save/load a very small descriptor with WIDTH/HEIGHT stored as 16
> bit integers, then, aware of the array size of the buffer, alloc and
> fread or fwrite down.
> 
> It's a very "stupid" non standard format, but I don't need to export
> this directly. When in gambas I save using its native functions. The C++
> engine is just supposed to transform pictures and save in raw format.
> 

So don't make stupid formats.  Make /simple/ formats, and non-standard
formats, if that is what is best for the job.  But don't make a stupid
format.  If the fields in the struct are not properly ("naturally")
aligned, change the format so that they are.  That will help the gambas
side, help the C++ side, and mean you don't have to faff around with
"pack" extensions /or/ with extra packing and unpacking stages.

[toc] | [prev] | [next] | [standalone]

#69979

Il 22/11/19 12:23, David Brown ha scritto:
> On 22/11/2019 11:31, Soviet_Mario wrote:
>> Il 22/11/19 09:37, David Brown ha scritto:
>>> On 22/11/2019 07:26, Christian Gollwitzer wrote:
>>>> Am 21.11.19 um 17:20 schrieb Paavo Helde:
>>>>> On 21.11.2019 13:53, Soviet_Mario wrote:
>>>>>> My only requirement is that the plain native type BYTE used in gambas
>>>>>> code that calls the C++ static library and that expects a 8-bit wide
>>>>>> unsigned integral, find its matching type in the C++ code, as they
>>>>>> need
>>>>>> to share bitmaps and transparency maps.
>>>>>
>>>>> This is easy, use unsigned char or std::uint8_t.
>>>>>
>>>>>> The bitmaps use 32-bit packed colors+alpha channel, so I was
>>>>>> assuming no
>>>>>> difficulty in exchange.
>>>>>
>>>>> This is more tricky because the size of int or long varies much more
>>>>> than char. For example, long is 32 bits in 64-bit MSVC and 64 bits
>>>>> with 64-bit gcc on Linux.
>>>>>
>>>>> However, for your purposes std::uint32_t should work fine.
>>>>
>>>> Why not
>>>>
>>>> #pragma pack(1)
>>>> struct pixel {
>>>>       uint8_t red, green, blue, alpha;
>>>> }
>>>>
>>>> ? That would make more sense to me than uint32_t and bit-twiddling to
>>>> get the channel values.
>>>>
>>>> Christian
>>>
>>> If you are using "pragma pack" (or gcc "packed" attribute), you are
>>> probably doing something wrong.  It can have some uses when dealing with
>>> file formats
>>
>> it is exactly this second case : two programs have to communicate and
>> store picture files in a compact way (and if I'll manage to, even
>> exchange such pictures loaded in memory) upon a single big binary
>> "fread"  from disk or save with a single raw "fwrite" to disk.
>> I can't find it acceptable that the DISK layout and RAM layout may
>> differ as to packing aspects.
>> Non only for the waste of space (a large waste as pictures could be some
>> tens megapixels), but as in such case I would have to read/write only
>> single elements at a time. Unacceptable (*).
>> I must be allowed to load and save fast and in a single shot, so 1:1
>> mapping RAM<=>DISK.
>> This requirement is underogatable, even if I agree with all other
>> remarks you make afterwards ...
> 
> You have a struct with byte data - making it packed is /useless/.  There
> is /never/ any advantage in that.
> 
> If the structure contains data of different sizes, then first check if
> there really are padding or alignment issues.  Most structures used for
> file formats are "naturally aligned" - the people who designed the
> structures are smart enough to make sure that every 2-byte data item is
> at an even address, every 4-byte data item is at a multiple of 4
> address, and so on.  There are exceptions, especially for some older
> formats from the days of 16-bit cpus, but it is often the case.
> 
> And if it is the case here, then making it "packed" is useless at best,
> and can easily be a significant disadvantage.
> 
> All of what you have written here is "premature optimisation" - it is
> knee-jerk thinking "I've got a file structure, I must use packed" rather
> than looking at what is actually relevant or helpful.  At the moment,
> you haven't the faintest clue as to whether using a packed struct makes
> a real and measurable difference that is worth the cost -

sure I have no idea, I put the pragma there just for 
"precaution"
what cost is that supposed to have ?


> even if we
> assume the file format details are not aligned.  You haven't done any
> timings of real-world usage of your system to see if this is a
> bottleneck.  You haven't tested alternatives like "mmap", or existing
> libraries. 

I'm writing from scratch almost, but It will be a very 
simple library

> You haven't compared the time it takes to load or save the
> file with the time it takes to do your processing. 

sure, but I'll have to do both in any case :)
It's a non-sense comparison though
I just thought of
read/write all-at-once vs pixel by pixel (and choose I)

no use in knowing if I spend more time applying 
transformations on the loaded picture vs saving or loading 
it. They are sequential mandatory operations


> You haven't compared
> the time it takes to pack or unpack the format to the time it takes to

chosen not to pack/unpack anything actually, as I'm hoping 
the RAM / disk layout are the very same raw binary.
I was asking help in making sure this was (not if it was 
useful or not)

> read or write it from the disk.  You haven't compared the time the
> processing takes for a "packed" struct to how it would run with an
> unpacked struct.

forgive me : how to read an all packed file in an array of 
UNpacked data of different layout, in a single read 
operation ? I can't figure out this

> 
> (Yes, I realise I have made a lot of claims here about what you have and
> haven't done.  From your posts, I think they are almost certainly true.
>   If I am wrong there, let me know.)

no problem, most are right.
Some are irrelevant imho

> 
> 
> If you want to tell me that using "pack" makes your development process
> simpler and the code easier, that's fine

I don't know. Just searching a way to do binary read/write 
in one single shot directly to/from RAM without any conversions.
And have a RAM layout simple enough not to require bitwise 
masking to locate fields (the drawback of a pure "bit 
fields" solution)

> - I'll believe that, and it is
> a perfectly good reason for using it.  But if you try to tell me you are
> doing it for run-time speed reasons, then you are wrong.  Measure first,
> find out if it really is too slow, find your /real/ bottlenecks, /then/
> start looking for ways to improve the throughput.

I don't have any running stuff by now, as I'm still in the 
design time.
But I could not redesign the BASIC disk and data structure 
later, so i MUST make some premature optimization now as it 
affects the very base of the pillar.

> 
>>
>>
>> (*) well actually I do not make strictly a single load/save as stated.
>> First I save/load a very small descriptor with WIDTH/HEIGHT stored as 16
>> bit integers, then, aware of the array size of the buffer, alloc and
>> fread or fwrite down.
>>
>> It's a very "stupid" non standard format, but I don't need to export
>> this directly. When in gambas I save using its native functions. The C++
>> engine is just supposed to transform pictures and save in raw format.
>>
> 
> So don't make stupid formats.  Make /simple/ formats, 

I meant the same actually.
I store 16 bit width, 16 bit height, and then a compact 
rectangular array o 32 bit pixels (8+8+8+8 for 3 color 
channels + transparency) per each pixel.

it is is not completely "naturally aligned", but the header 
and the "body" are placed apart, in memory, and read/written 
in distinct file operations.

So I ask you, when I call NEW operator for an array of 
"pixel" structs, is it naturally aligned or not ?


> and non-standard
> formats, if that is what is best for the job.  But don't make a stupid
> format. 

I called it stupid in place of the simplest as possible.
There are no "tags" included as this is only an "internal" 
(and personal) format for intermediate raw data.
It does not seem to be more sophisticated than this.

> If the fields in the struct are not properly ("naturally")
> aligned, change the format so that they are.  That will help the gambas
> side, help the C++ side, and mean you don't have to faff around with
> "pack" extensions /or/ with extra packing and unpacking stages.
> 
> 


-- 
1) Resistere, resistere, resistere.
2) Se tutti pagano le tasse, le tasse le pagano tutti
Soviet_Mario - (aka Gatto_Vizzato)

[toc] | [prev] | [next] | [standalone]

#69982

On 22/11/2019 13:14, Soviet_Mario wrote:
> Il 22/11/19 12:23, David Brown ha scritto:
>> On 22/11/2019 11:31, Soviet_Mario wrote:
>>> Il 22/11/19 09:37, David Brown ha scritto:
>>>> On 22/11/2019 07:26, Christian Gollwitzer wrote:
>>>>> Am 21.11.19 um 17:20 schrieb Paavo Helde:
>>>>>> On 21.11.2019 13:53, Soviet_Mario wrote:
>>>>>>> My only requirement is that the plain native type BYTE used in
>>>>>>> gambas
>>>>>>> code that calls the C++ static library and that expects a 8-bit wide
>>>>>>> unsigned integral, find its matching type in the C++ code, as they
>>>>>>> need
>>>>>>> to share bitmaps and transparency maps.
>>>>>>
>>>>>> This is easy, use unsigned char or std::uint8_t.
>>>>>>
>>>>>>> The bitmaps use 32-bit packed colors+alpha channel, so I was
>>>>>>> assuming no
>>>>>>> difficulty in exchange.
>>>>>>
>>>>>> This is more tricky because the size of int or long varies much more
>>>>>> than char. For example, long is 32 bits in 64-bit MSVC and 64 bits
>>>>>> with 64-bit gcc on Linux.
>>>>>>
>>>>>> However, for your purposes std::uint32_t should work fine.
>>>>>
>>>>> Why not
>>>>>
>>>>> #pragma pack(1)
>>>>> struct pixel {
>>>>>       uint8_t red, green, blue, alpha;
>>>>> }
>>>>>
>>>>> ? That would make more sense to me than uint32_t and bit-twiddling to
>>>>> get the channel values.
>>>>>
>>>>> Christian
>>>>
>>>> If you are using "pragma pack" (or gcc "packed" attribute), you are
>>>> probably doing something wrong.  It can have some uses when dealing
>>>> with
>>>> file formats
>>>
>>> it is exactly this second case : two programs have to communicate and
>>> store picture files in a compact way (and if I'll manage to, even
>>> exchange such pictures loaded in memory) upon a single big binary
>>> "fread"  from disk or save with a single raw "fwrite" to disk.
>>> I can't find it acceptable that the DISK layout and RAM layout may
>>> differ as to packing aspects.
>>> Non only for the waste of space (a large waste as pictures could be some
>>> tens megapixels), but as in such case I would have to read/write only
>>> single elements at a time. Unacceptable (*).
>>> I must be allowed to load and save fast and in a single shot, so 1:1
>>> mapping RAM<=>DISK.
>>> This requirement is underogatable, even if I agree with all other
>>> remarks you make afterwards ...
>>
>> You have a struct with byte data - making it packed is /useless/.  There
>> is /never/ any advantage in that.
>>
>> If the structure contains data of different sizes, then first check if
>> there really are padding or alignment issues.  Most structures used for
>> file formats are "naturally aligned" - the people who designed the
>> structures are smart enough to make sure that every 2-byte data item is
>> at an even address, every 4-byte data item is at a multiple of 4
>> address, and so on.  There are exceptions, especially for some older
>> formats from the days of 16-bit cpus, but it is often the case.
>>
>> And if it is the case here, then making it "packed" is useless at best,
>> and can easily be a significant disadvantage.
>>
>> All of what you have written here is "premature optimisation" - it is
>> knee-jerk thinking "I've got a file structure, I must use packed" rather
>> than looking at what is actually relevant or helpful.  At the moment,
>> you haven't the faintest clue as to whether using a packed struct makes
>> a real and measurable difference that is worth the cost -
> 
> sure I have no idea, I put the pragma there just for "precaution"
> what cost is that supposed to have ?
> 

I've told you in several posts what the costs are.  "Packed" is /not/ a
"precaution", nor is using a struct with a bitfield instead of uint8_t.
 If you don't know what you are doing, fine - if I understand you
correctly, you are experienced with gambas programming but new to C++.
So /ask/ (and please read the answers).  Don't guess, or grab the first
thing that you find from a google search.  And when the more experienced
people in a group like this all say "don't use packed" and "use
uint8_t", then that is what you should do.  Ask /why/, and try to learn
- and once you do understand "pack", when it can be useful, when it is
useless, when it is worse than useless, what it does, and what it costs
- /then/ you can decide that because you know the full situation better
than we do, you still want to use it against general advice.

"Packed" is not a precaution - it is a last choice solution when you
have nothing better and know you can't write the code as well without
it.  (That might mean you can't write the code as quickly or clearly
without it, which is fair enough - /if/ you are sure it is necessary.)

> 
>> even if we
>> assume the file format details are not aligned.  You haven't done any
>> timings of real-world usage of your system to see if this is a
>> bottleneck.  You haven't tested alternatives like "mmap", or existing
>> libraries. 
> 
> I'm writing from scratch almost, but It will be a very simple library
> 
>> You haven't compared the time it takes to load or save the
>> file with the time it takes to do your processing. 
> 
> sure, but I'll have to do both in any case :)
> It's a non-sense comparison though
> I just thought of
> read/write all-at-once vs pixel by pixel (and choose I)
> 
> no use in knowing if I spend more time applying transformations on the
> loaded picture vs saving or loading it. They are sequential mandatory
> operations

Of course there is use in knowing this, if speed is an issue.  If the
processing takes 1000 times as long as the loading and saving, then you
know that putting the slightest effort into speed optimisation for file
access is a waste of effort - your time would be better spent trying to
shave 0.1% off the processing time than increasing the speed of the file
access by orders of magnitude.

And if your file format contains non-aligned members, then the simplest
and easiest way to improve and simplify the code, and improve the speed
of several parts, is to fix the file format.  Make sure the members are
aligned (add explicit dummy members if necessary) - then forget you ever
heard of "pragma pack".

> 
> 
>> You haven't compared
>> the time it takes to pack or unpack the format to the time it takes to
> 
> chosen not to pack/unpack anything actually, as I'm hoping the RAM /
> disk layout are the very same raw binary.
> I was asking help in making sure this was (not if it was useful or not)

No one can tell you if it is the same without knowing the file format.

There is nothing wrong with having the ram and disk layout the same -
often that is a very convenient and efficient choice.  But if the disk
layout is not directly usable in ram, then it often is /not/ a
convenient and efficient choice - and if it /is/ usable directly in ram,
you don't need or want "pack".

However you look at it, "pack" is usually (but not always) the wrong
choice.  You should only consider it once you have eliminated better
alternatives.

As for asking for help here, you are getting the help you need even if
it is not exactly what you asked for.  It's as though you said "I need a
new toothbrush.  Can you tell me how to get to the toy shop?"  It is
conceivable that a toy shop might have toothbrushes, but it is very
unlikely to be a good choice.  Would you not rather have people say "You
don't want to go to a toy shop.  You want to go to a chemist - this is
how you get there" ?

> 
>> read or write it from the disk.  You haven't compared the time the
>> processing takes for a "packed" struct to how it would run with an
>> unpacked struct.
> 
> forgive me : how to read an all packed file in an array of UNpacked data
> of different layout, in a single read operation ? I can't figure out this
> 

First, you take a step back.  You look to see if the packed file really
is "packed", in the sense of having unaligned data that would have
padding in a normal struct.  I have seen no answer to this (unless it is
later in this post), despite asking this /crucial/ question.

If the format is already good, you are done.  One memcpy, or read from
the file directly to a suitable buffer, or memmap to the file directly
(no reading or writing needed).

If the format is bad, the next step is to consider changing the format.
 This would be good for all parts of the system.  When the format is
changed, you are done.  Again, I have had no answer to whether this is
possible.

If the format is bad, and cannot be fixed, then you have to write reader
and writer functions that unpack and pack the data into the right
formats.  Reader and writer functions are common for all sorts of
reasons - you already have something to handle starting with the file
header and then allocating memory depending on the size.  It is very
normal to have a function that reads in the data format and uses it to
populate an internal format.  Perhaps your file format uses 8-bit
unsigned data for colour channels, but your internal format uses
floating point because that suits your processing needs.  Perhaps the
file format uses run-length encoding for smaller files, and you want to
expand it to a raw array.  There are all sorts of things that go on in
typical reader functions.  Making them a "single operation", whatever
that might mean, is rarely relevant - code that uses these functions has
a "single operation" of calling the reader function.


>>
>> (Yes, I realise I have made a lot of claims here about what you have and
>> haven't done.  From your posts, I think they are almost certainly true.
>>   If I am wrong there, let me know.)
> 
> no problem, most are right.
> Some are irrelevant imho

Perhaps - but not necessarily the ones you think are irrelevant.  IMHO,
of course.

(And to be clear - I did not right this to be rude, or insulting, or
demoralising, and I hope it did not come across that way.  I am writing
to push you to think about things that are important here, but which I
feel you are skipping.  The aim is that you learn more, and that you can
write better code that is clear, correct and efficient without wasting
effort on poorer techniques.)

> 
>>
>>
>> If you want to tell me that using "pack" makes your development process
>> simpler and the code easier, that's fine
> 
> I don't know. Just searching a way to do binary read/write in one single
> shot directly to/from RAM without any conversions.

Doing this all "in one single shot" should not be an aim.  Doing it in a
way that makes the code simple and clear, easy to develop, easy to read
and easily correct should be your aim.  Quite possibly your real goal
would be easiest achieved by using a standard format (like PNG) and
existing libraries for gambas and C++.  Call the one function - simple
and clear.

> And have a RAM layout simple enough not to require bitwise masking to
> locate fields (the drawback of a pure "bit fields" solution)
> 

Why would you want to bring bitfields into this?

>> - I'll believe that, and it is
>> a perfectly good reason for using it.  But if you try to tell me you are
>> doing it for run-time speed reasons, then you are wrong.  Measure first,
>> find out if it really is too slow, find your /real/ bottlenecks, /then/
>> start looking for ways to improve the throughput.
> 
> I don't have any running stuff by now, as I'm still in the design time.
> But I could not redesign the BASIC disk and data structure later, so i
> MUST make some premature optimization now as it affects the very base of
> the pillar.
> 

Fair enough.  So get it /right/ by designing the right data structure
/now/, rather than picking a bad structure from the start.

>>
>>>
>>>
>>> (*) well actually I do not make strictly a single load/save as stated.
>>> First I save/load a very small descriptor with WIDTH/HEIGHT stored as 16
>>> bit integers, then, aware of the array size of the buffer, alloc and
>>> fread or fwrite down.
>>>
>>> It's a very "stupid" non standard format, but I don't need to export
>>> this directly. When in gambas I save using its native functions. The C++
>>> engine is just supposed to transform pictures and save in raw format.
>>>
>>
>> So don't make stupid formats.  Make /simple/ formats, 
> 
> I meant the same actually.
> I store 16 bit width, 16 bit height, and then a compact rectangular
> array o 32 bit pixels (8+8+8+8 for 3 color channels + transparency) per
> each pixel.
> 
> it is is not completely "naturally aligned", but the header and the
> "body" are placed apart, in memory, and read/written in distinct file
> operations.

So make it "naturally aligned".  What will be the cost?  A couple of
extra unsused padding bytes, added manually?

struct pixel {
	uint8_t red, green, blue, alpha;
};

struct picture_format {
	uint64_t key;		// Always 0x0064727574636950ull
	uint8_t file_type_version;
	uint8_t padding1;
	uint16_t height;
	uint16_t width;
	uint16_t padding2;

	pixel pixeldata[];
};

Pick an alignment for the structure - 32-bit or 64-bit are good choices.
 Start with a member of that size for convenience - I like a constant
marker to make it easy to tell that you are using the right file type.
(The value here is the string "Picture".)  Make sure everything is
aligned sensibly.  Add padding members manually and explicitly to keep
it clear - then there is never any doubt.  (These may later be re-used
for extra information in later file format versions.)  Make sure
separate blocks, such as the header and the data section, each are
aligned to the main large alignment for efficiency.

> 
> So I ask you, when I call NEW operator for an array of "pixel" structs,
> is it naturally aligned or not ?

Yes, as long as you don't faff about with "packed" stuff.  Then you
don't know.

> 
> 
>> and non-standard
>> formats, if that is what is best for the job.  But don't make a stupid
>> format. 
> 
> I called it stupid in place of the simplest as possible.
> There are no "tags" included as this is only an "internal" (and
> personal) format for intermediate raw data.
> It does not seem to be more sophisticated than this.

Then it is inexcusable to have a format that needs "packed".  That just
means totally pointless complications, inefficiencies, and non-portable
code.

If you have learned nothing else from this conversation, hopefully you
have learned that much.

> 
>> If the fields in the struct are not properly ("naturally")
>> aligned, change the format so that they are.  That will help the gambas
>> side, help the C++ side, and mean you don't have to faff around with
>> "pack" extensions /or/ with extra packing and unpacking stages.
>>
>>
> 
> 

[toc] | [prev] | [next] | [standalone]

#70000

Il 22/11/19 15:04, David Brown ha scritto:
> On 22/11/2019 13:14, Soviet_Mario wrote:
>> Il 22/11/19 12:23, David Brown ha scritto:
>>> On 22/11/2019 11:31, Soviet_Mario wrote:
>>>> Il 22/11/19 09:37, David Brown ha scritto:
>>>>> On 22/11/2019 07:26, Christian Gollwitzer wrote:
>>>>>> Am 21.11.19 um 17:20 schrieb Paavo Helde:
>>>>>>> On 21.11.2019 13:53, Soviet_Mario wrote:
>>>>>>>> My only requirement is that the plain native type BYTE used in
>>>>>>>> gambas
>>>>>>>> code that calls the C++ static library and that expects a 8-bit wide
>>>>>>>> unsigned integral, find its matching type in the C++ code, as they
>>>>>>>> need
>>>>>>>> to share bitmaps and transparency maps.
>>>>>>>
>>>>>>> This is easy, use unsigned char or std::uint8_t.
>>>>>>>
>>>>>>>> The bitmaps use 32-bit packed colors+alpha channel, so I was
>>>>>>>> assuming no
>>>>>>>> difficulty in exchange.
>>>>>>>
>>>>>>> This is more tricky because the size of int or long varies much more
>>>>>>> than char. For example, long is 32 bits in 64-bit MSVC and 64 bits
>>>>>>> with 64-bit gcc on Linux.
>>>>>>>
>>>>>>> However, for your purposes std::uint32_t should work fine.
>>>>>>
>>>>>> Why not
>>>>>>
>>>>>> #pragma pack(1)
>>>>>> struct pixel {
>>>>>>        uint8_t red, green, blue, alpha;
>>>>>> }
>>>>>>
>>>>>> ? That would make more sense to me than uint32_t and bit-twiddling to
>>>>>> get the channel values.
>>>>>>
>>>>>> Christian
>>>>>
>>>>> If you are using "pragma pack" (or gcc "packed" attribute), you are
>>>>> probably doing something wrong.  It can have some uses when dealing
>>>>> with
>>>>> file formats
>>>>
>>>> it is exactly this second case : two programs have to communicate and
>>>> store picture files in a compact way (and if I'll manage to, even
>>>> exchange such pictures loaded in memory) upon a single big binary
>>>> "fread"  from disk or save with a single raw "fwrite" to disk.
>>>> I can't find it acceptable that the DISK layout and RAM layout may
>>>> differ as to packing aspects.
>>>> Non only for the waste of space (a large waste as pictures could be some
>>>> tens megapixels), but as in such case I would have to read/write only
>>>> single elements at a time. Unacceptable (*).
>>>> I must be allowed to load and save fast and in a single shot, so 1:1
>>>> mapping RAM<=>DISK.
>>>> This requirement is underogatable, even if I agree with all other
>>>> remarks you make afterwards ...
>>>
>>> You have a struct with byte data - making it packed is /useless/.  There
>>> is /never/ any advantage in that.
>>>
>>> If the structure contains data of different sizes, then first check if
>>> there really are padding or alignment issues.  Most structures used for
>>> file formats are "naturally aligned" - the people who designed the
>>> structures are smart enough to make sure that every 2-byte data item is
>>> at an even address, every 4-byte data item is at a multiple of 4
>>> address, and so on.  There are exceptions, especially for some older
>>> formats from the days of 16-bit cpus, but it is often the case.
>>>
>>> And if it is the case here, then making it "packed" is useless at best,
>>> and can easily be a significant disadvantage.
>>>
>>> All of what you have written here is "premature optimisation" - it is
>>> knee-jerk thinking "I've got a file structure, I must use packed" rather
>>> than looking at what is actually relevant or helpful.  At the moment,
>>> you haven't the faintest clue as to whether using a packed struct makes
>>> a real and measurable difference that is worth the cost -
>>
>> sure I have no idea, I put the pragma there just for "precaution"
>> what cost is that supposed to have ?
>>
> 
> I've told you in several posts what the costs are. 

put is alas irregular verb : I meant its PAST form actually.
I have abandoned the unsigned Bit_Field : 8 solution as soon 
as you (a lot of people) stated that u_int_8 MUST have 
exactly 8 bits.

In that case either pragma is USELESS (and hence does not 
produce any reasonable effect, neither slowness), either is 
NECESSARY (and so the slowing effect is a unavoidable price 
that I MUST pay either)

> "Packed" is /not/ a
> "precaution", nor is using a struct with a bitfield instead of uint8_t.

the second part has been abandoned early

>   If you don't know what you are doing

I'm rather weak in higher level C++ constructs, and I know 
that, but not particularly so in low level (some couple of 
tens of years ago I used to stuff C code with inline "asm" 
statements or even __emit() calls).

>, fine - if I understand you
> correctly, you are experienced with gambas

no, I am rather new to gambas actually.
I was used to VB (Visual Studio, but not even recently). 
Going to Gambas has been actually a pain (not saying it is 
bad, absolutely not : it's me that knows much less than .net 
framework). On QT I am even more newby.

>  programming but new to C++.

well, I am more of outdated than new, I had used C++ (and C) 
a lot in 1995-2000 roughly, then less and less afterwards.
But strangely enough, I find it more easy to think in C++ 
terms when I met problems in basic.

> So /ask/ (and please read the answers).  Don't guess, or grab the first
> thing that you find from a google search. 

but I DID follow the answers :)

> And when the more experienced
> people in a group like this all say "don't use packed" and "use
> uint8_t",

I didn't read or desume the two things should be mutually 
exclusive.

Anyway, there are three scenarios

1) added pragma (to the four u_int_8 members in a struct) is 
USELESS, as the struct was just packed natively (and hence 
does not produce any reasonable effect, neither slowness),
either
2) is NECESSARY, as the struct would not have been natively 
packed otherwise (and so the slowing effect is a unavoidable 
price that I MUST pay either)
either
3) it is NOT SUFFICIENT (in this case, luckily unlikely, 
nothing would have worked fine at all)

as for ALIGNMENT problem, I should not meet any I guess.
The packed (natively or forcefully) struct is 32 bit wide, 
so allocating an array should allow nice addressing of any 
item even on a 64 bit OS
And no item would cross critical addressing boundary


> then that is what you should do.  Ask /why/, and try to learn
> - and once you do understand "pack", when it can be useful, when it is
> useless, when it is worse than useless,

If you have some further patience, tell me if the 3 
scenarios fit the present situation (not a more general one, 
but just this)

> what it does, and what it costs
> - /then/ you can decide that because you know the full situation better
> than we do, you still want to use it against general advice.

I had given all details sufficient : the data layout is what 
I said, so no obscure aspects

> 
> "Packed" is not a precaution -

I call it a precaution as it reduces the chances of a non 
packed data layout. I was not saying it would be successful 
necessarily in every scenario (even sizes sequence is 
relevant, and I know this, but my members were all 1 byte long)

> it is a last choice solution when you
> have nothing better and know you can't write the code as well without
> it.  (That might mean you can't write the code as quickly or clearly
> without it, which is fair enough - /if/ you are sure it is necessary.)

let's say it explicitely

if I have
union
{
	struct
	{
		u_int_8 Red, Green, Blue, Alpha;
	};
	u_int_32  AllChannels;
};

without PRAGMA, do I have any explicit or implicit warranty 
that my union be 32 bit long ? If so, PRAGMA is useless. If 
not, it seems it might be useful.
what damages could PRAGMA PACK produce ?
If the union would be just packed natively, I guess none.
If it wasn't, at best it would pack it (and maybe slow it, 
but this is irrelevant, I need precise disk layout). At 
worsk it would be insufficient to actually pack it, so my 
problem is unsolved.
In this last worst case, are there any other means of 
getting the thing to work ?
What are them ?


> 
>>
>>> even if we
>>> assume the file format details are not aligned.  You haven't done any
>>> timings of real-world usage of your system to see if this is a
>>> bottleneck.  You haven't tested alternatives like "mmap", or existing
>>> libraries.
>>
>> I'm writing from scratch almost, but It will be a very simple library
>>
>>> You haven't compared the time it takes to load or save the
>>> file with the time it takes to do your processing.
>>
>> sure, but I'll have to do both in any case :)
>> It's a non-sense comparison though
>> I just thought of
>> read/write all-at-once vs pixel by pixel (and choose I)
>>
>> no use in knowing if I spend more time applying transformations on the
>> loaded picture vs saving or loading it. They are sequential mandatory
>> operations
> 
> Of course there is use in knowing this, if speed is an issue.  If the
> processing takes 1000 times as long as the loading and saving, 

this could be true only if load/store is "optimized" like I 
stated. If it was done an item at a time it would become 
very very slow (and depending on smartness of OS caching 
anyway, but not even so much even then, as I read/write the 
whole file, which can be 4-12 MB long and one time only, so 
caching cannot be as useful as when reusing a small region 
many times).

> then you
> know that putting the slightest effort into speed optimisation for file
> access is a waste of effort

when I made some elaboration all in gambas the times were 
"mixed" : some basic transformations with few or no floating 
point calculations went actually FASTER than disk 
read/write, with some more f.p. intensive calculations disk 
load/save were faster.
But in any case, the same "order of magnitude", not 
100-1000-etc times slower or faster.

> - your time would be better spent trying to
> shave 0.1% off the processing time than increasing the speed of the file
> access by orders of magnitude.
> 
> And if your file format contains non-aligned members, 

why making general guess when I reported precisely the layout ?
I said :
16 bit (width) - 16 bit (height) - N x 32 bit (R,G,B,alpha)
where N = width x height = number of "pixels"

every pixel is multiple of 32 address on file (but this 
seems irrelevant to me).
In the ram layout,
the BMP descriptor has width and height as "local" members, 
and a pointer to an array of struct aligned in memory by the 
NEW [] operator who is aware of alignment requirements.

Where are the possible problems ?
What problems are affected by a pragma pack directive ?


> then the simplest
> and easiest way to improve and simplify the code, and improve the speed
> of several parts, is to fix the file format.

that point is unmodificable. I strictly need full color and 
no compression. The alpha channel is maybe a bit overkill 
(255 level of transparency is more than needed ... but 
taking less is useless or impossible, and 1 byte is natural 
for the otherwise 3,sth bytes long struct)

>  Make sure the members are
> aligned (add explicit dummy members if necessary)

since when reading/writing files suffer from this ? Files 
are BYTE oriented internally (and the OS does caching 
independently and transparently, basing on disk cluster 
sizes but this is not sth I should care of)

as for memory : I instance the BMP array invoking
ptrBMP = new clsMyStructColor [TotSize];
and I expect the structs are aligned correctly as NEW knows 
its task.

> - then forget you ever
> heard of "pragma pack".
> 
>>
>>
>>> You haven't compared
>>> the time it takes to pack or unpack the format to the time it takes to
>>
>> chosen not to pack/unpack anything actually, as I'm hoping the RAM /
>> disk layout are the very same raw binary.
>> I was asking help in making sure this was (not if it was useful or not)
> 
> No one can tell you if it is the same without knowing the file format.


I just said it : the stupidest format of all

16 bit (width) - 16 bit (height) - N x 32 bit (R,G,B,alpha)
where N = width x height = number of "pixels"


> 
> There is nothing wrong with having the ram and disk layout the same -

oh happy to know ! :)

> often that is a very convenient and efficient choice.  But if the disk
> layout is not directly usable in ram, then it often is /not/ a
> convenient and efficient choice - and if it /is/ usable directly in ram,
> you don't need or want "pack".

I was asking about the specific case

> 
> However you look at it, "pack" is usually (but not always) the wrong
> choice.  You should only consider it once you have eliminated better
> alternatives.

for a starting, I tried to use a 32 bit total size, hoping 
it would have been packed regardless of any "hints" (pragma).

> 
> As for asking for help here, you are getting the help you need even if
> it is not exactly what you asked for. 

not a problem, every advice is appreciated

> It's as though you said "I need a
> new toothbrush.  Can you tell me how to get to the toy shop?"  It is
> conceivable that a toy shop might have toothbrushes, but it is very
> unlikely to be a good choice.  Would you not rather have people say "You
> don't want to go to a toy shop.  You want to go to a chemist - this is
> how you get there" ?
> 
>>
>>> read or write it from the disk.  You haven't compared the time the
>>> processing takes for a "packed" struct to how it would run with an
>>> unpacked struct.
>>
>> forgive me : how to read an all packed file in an array of UNpacked data
>> of different layout, in a single read operation ? I can't figure out this
>>
> 
> First, you take a step back.  You look to see if the packed file really
> is "packed",

it is, as both Gambas supports BYTE format (And sizeof()) 
and BYTE-oriented streams. No overspace is generated. Then I 
have to worry about the C++ back end, to comply with this fact.

I think the designer of Gambas relies very strongly upon C 
libraries and mind-shape, apart from strings (where he had 
to welcome BASIC users who want strings the basic way).

> in the sense of having unaligned data that would have
> padding in a normal struct.  I have seen no answer to this (unless it is
> later in this post), despite asking this /crucial/ question.

the files are packed.

> 
> If the format is already good, you are done.  One memcpy, or read from
> the file directly to a suitable buffer, or memmap to the file directly
> (no reading or writing needed).

memmap is a problem in that I don't know how exactly works 
and whether or not I'll be able to interoperate with memory 
files opened in gambas

I am also considering to use some memory files supported 
directly by the OS (here on linux files created on /shm 
virtual folder are designed for such FAST temporaries to be 
shared, but with portable stream syntax).
Using /shm would imply no specialized load/store code, just 
the normal file access => delegating the ram usage to the OS.

> 
> If the format is bad, the next step is to consider changing the format.

just observation.
So, the format is wrong ?

>   This would be good for all parts of the system.  When the format is
> changed, you are done.  Again, I have had no answer to whether this is
> possible.
> 
> If the format is bad, and cannot be fixed, then you have to write reader
> and writer functions that unpack and pack the data into the right
> formats.  Reader and writer functions are common for all sorts of

well, I'd be considering these for small size, complex 
structured input, not for a 4-12 MB size of simple pixels. 
It would be a blood bathe

> reasons - you already have something to handle starting with the file
> header and then allocating memory depending on the size.  It is very
> normal to have a function that reads in the data format and uses it to
> populate an internal format. 

in fact, minimally, I do this. First create the container, 
with just a couple of members, then depending on their 
value, size the buffer and allocate it, then load the file 
in it.

> Perhaps your file format uses 8-bit
> unsigned data for colour channels, but your internal format uses
> floating point because that suits your processing needs. 

On certain routines I use floating point, but only 
temporarily, as I never STORE floating point pixels. I don't 
need a huge precision, as operations are not supposed to be 
applied multiple times on the same picture, creating 
rounding additive errors.

> Perhaps the
> file format uses run-length encoding for smaller files, and you want to
> expand it to a raw array.  There are all sorts of things that go on in
> typical reader functions.  Making them a "single operation", whatever
> that might mean, 

sth encoded in a single call to fread / fwrite, at least for 
the pixel data block. The header is distinct

> is rarely relevant - code that uses these functions has
> a "single operation" of calling the reader function.

well, fread/fwrite may maybe internally operate differently 
than expected, but in this case I would not be able to do 
better I guess :)


> 
> 
>>>
>>> (Yes, I realise I have made a lot of claims here about what you have and
>>> haven't done.  From your posts, I think they are almost certainly true.
>>>    If I am wrong there, let me know.)
>>
>> no problem, most are right.
>> Some are irrelevant imho
> 
> Perhaps - but not necessarily the ones you think are irrelevant.  IMHO,
> of course.
> 
> (And to be clear - I did not right this to be rude, or insulting, or
> demoralising, and I hope it did not come across that way. 

no problem, really

> I am writing
> to push you to think about things that are important here, but which I
> feel you are skipping.  The aim is that you learn more, and that you can
> write better code that is clear, correct and efficient without wasting
> effort on poorer techniques.)
> 
>>
>>>
>>>
>>> If you want to tell me that using "pack" makes your development process
>>> simpler and the code easier, that's fine
>>
>> I don't know. Just searching a way to do binary read/write in one single
>> shot directly to/from RAM without any conversions.
> 
> Doing this all "in one single shot" should not be an aim.  Doing it in a

I wish to both get fast disk I/O but also limit the 
try/catch trapping. I have the habit to catch or check errno 
on every disk access. A single huge read/write makes the 
code thinner than splitting the accesses in many place and 
having to multiplicate the error condition testing.

> way that makes the code simple and clear, easy to develop, easy to read
> and easily correct should be your aim.  Quite possibly your real goal
> would be easiest achieved by using a standard format (like PNG) and
> existing libraries for gambas and C++.  Call the one function - simple
> and clear.
> 
>> And have a RAM layout simple enough not to require bitwise masking to
>> locate fields (the drawback of a pure "bit fields" solution)
>>
> 
> Why would you want to bring bitfields into this?
> 
>>> - I'll believe that, and it is
>>> a perfectly good reason for using it.  But if you try to tell me you are
>>> doing it for run-time speed reasons, then you are wrong.  Measure first,
>>> find out if it really is too slow, find your /real/ bottlenecks, /then/
>>> start looking for ways to improve the throughput.
>>
>> I don't have any running stuff by now, as I'm still in the design time.
>> But I could not redesign the BASIC disk and data structure later, so i
>> MUST make some premature optimization now as it affects the very base of
>> the pillar.
>>
> 
> Fair enough.  So get it /right/ by designing the right data structure
> /now/, rather than picking a bad structure from the start.

please give your judgement about the struct I have described 
before, which drawbacks seems to have or don't have.

> 
>>>
>>>>
>>>>
>>>> (*) well actually I do not make strictly a single load/save as stated.
>>>> First I save/load a very small descriptor with WIDTH/HEIGHT stored as 16
>>>> bit integers, then, aware of the array size of the buffer, alloc and
>>>> fread or fwrite down.
>>>>
>>>> It's a very "stupid" non standard format, but I don't need to export
>>>> this directly. When in gambas I save using its native functions. The C++
>>>> engine is just supposed to transform pictures and save in raw format.
>>>>
>>>
>>> So don't make stupid formats.  Make /simple/ formats,
>>
>> I meant the same actually.
>> I store 16 bit width, 16 bit height, and then a compact rectangular
>> array o 32 bit pixels (8+8+8+8 for 3 color channels + transparency) per
>> each pixel.
>>
>> it is is not completely "naturally aligned", but the header and the
>> "body" are placed apart, in memory, and read/written in distinct file
>> operations.
> 
> So make it "naturally aligned".  What will be the cost?  A couple of
> extra unsused padding bytes, added manually?
> 
> struct pixel {
> 	uint8_t red, green, blue, alpha;
> };
> 
> struct picture_format {
> 	uint64_t key;		// Always 0x0064727574636950ull
> 	uint8_t file_type_version;
> 	uint8_t padding1;
> 	uint16_t height;
> 	uint16_t width;
> 	uint16_t padding2;
> 
> 	pixel pixeldata[];
> };
> 
> Pick an alignment for the structure - 32-bit or 64-bit are good choices.
>   Start with a member of that size for convenience - I like a constant
> marker to make it easy to tell that you are using the right file type.
> (The value here is the string "Picture".) 

I have stripped everything non necessary as this raw format 
is only a temporary. When I actually store files, I export 
them as PNG of JPG or GIF by Gambas code.
So my internal format was really minimal.

> Make sure everything is
> aligned sensibly.  Add padding members manually and explicitly to keep
> it clear - then there is never any doubt.  (These may later be re-used
> for extra information in later file format versions.)  Make sure
> separate blocks, such as the header and the data section, each are
> aligned to the main large alignment for efficiency.
> 
>>
>> So I ask you, when I call NEW operator for an array of "pixel" structs,
>> is it naturally aligned or not ?
> 
> Yes, as long as you don't faff about with "packed" stuff.  Then you
> don't know.
> 
>>
>>
>>> and non-standard
>>> formats, if that is what is best for the job.  But don't make a stupid
>>> format.
>>
>> I called it stupid in place of the simplest as possible.
>> There are no "tags" included as this is only an "internal" (and
>> personal) format for intermediate raw data.
>> It does not seem to be more sophisticated than this.
> 
> Then it is inexcusable to have a format that needs "packed".  That just

oh I completely disagree with this point ! The format has to 
be packed as the C++ library receives the file from an other 
source of data that offers them in that form, so it's mandatory.
And also the caller expects to get the transformed data back 
in the same packed form.

> means totally pointless complications, inefficiencies, and non-portable
> code.

nope, I have constraints to fullfil. Maybe inefficient, 
maybe non portable (who cares ?), but surely not pointless, 
as otherwise data exchange would not work at all

> 
> If you have learned nothing else from this conversation, hopefully you
> have learned that much.
> 
>>
>>> If the fields in the struct are not properly ("naturally")
>>> aligned, change the format so that they are.  That will help the gambas
>>> side, help the C++ side, and mean you don't have to faff around with
>>> "pack" extensions /or/ with extra packing and unpacking stages.
>>>
>>>
>>
>>
> 


-- 
1) Resistere, resistere, resistere.
2) Se tutti pagano le tasse, le tasse le pagano tutti
Soviet_Mario - (aka Gatto_Vizzato)

[toc] | [prev] | [next] | [standalone]

#70001

On 11/22/19 1:35 PM, Soviet_Mario wrote:
> Il 22/11/19 15:04, David Brown ha scritto:
...
> I have abandoned the unsigned Bit_Field : 8 solution as soon 
> as you (a lot of people) stated that u_int_8 MUST have

uint8_t

> exactly 8 bits.

It must have exactly 8 bits, if it exists. uint8_t is optional. And on
any C99+ implementation that fails to support uint8_t, and on any C
platform where unsigned char doesn't have 8 bits, you can't do what you
want to do anyway.

> In that case either pragma is USELESS (and hence does not 
> produce any reasonable effect, neither slowness), either is 
> NECESSARY (and so the slowing effect is a unavoidable price 
> that I MUST pay either)

Correct - it is both useless and unnecessary - in this context. Don't
extrapolate from this case to the general case. It is very specifically
the fact that you are trying to pack something that cannot possibly be
bigger than a byte, and therefore cannot have an alignment requirement
bigger than 1, that renders the pragma useless and unnecessary.

>>   If you don't know what you are doing
> 
> I'm rather weak in higher level C++ constructs, and I know 
> that, but not particularly so in low level (some couple of 
> tens of years ago I used to stuff C code with inline "asm" 
> statements or even __emit() calls).

You don't appear to understand why the fact that you're trying to use an
8-bit type guarantees that #pragma pack will be useless and unnecessary
- so there's some low-level stuff you don't understand yet.

>> And when the more experienced
>> people in a group like this all say "don't use packed" and "use
>> uint8_t",
> 
> I didn't read or desume the two things should be mutually 
> exclusive.
> 
> Anyway, there are three scenarios
> 
> 1) added pragma (to the four u_int_8 members in a struct) is 
> USELESS, as the struct was just packed natively (and hence 
> does not produce any reasonable effect, neither slowness),
> either
> 2) is NECESSARY, as the struct would not have been natively 
> packed otherwise (and so the slowing effect is a unavoidable 
> price that I MUST pay either)
> either
> 3) it is NOT SUFFICIENT (in this case, luckily unlikely, 
> nothing would have worked fine at all)

On implementations where CHAR_BIT==8, scenario #1 applies. On
implementations where CHAR_BIT>8, scenarios #1 and #3 both apply.

...
> 	u_int_32  AllChannels;

uint_32


>> in the sense of having unaligned data that would have
>> padding in a normal struct.  I have seen no answer to this (unless it is
>> later in this post), despite asking this /crucial/ question.
> 
> the files are packed.

OK - which fields are positioned differently from the way they would be
positioned if not packed?

[toc] | [prev] | [next] | [standalone]

#70003

Il 22/11/19 20:11, James Kuyper ha scritto:
> On 11/22/19 1:35 PM, Soviet_Mario wrote:
>> Il 22/11/19 15:04, David Brown ha scritto:
> ...
>> I have abandoned the unsigned Bit_Field : 8 solution as soon
>> as you (a lot of people) stated that u_int_8 MUST have
> 
> uint8_t

here the macro or typedef is

u_int8_t actually

> 
>> exactly 8 bits.
> 
> It must have exactly 8 bits, if it exists. uint8_t is optional. And on
> any C99+ implementation that fails to support uint8_t, and on any C
> platform where unsigned char doesn't have 8 bits, you can't do what you
> want to do anyway.

ok, that's a fact

> 
>> In that case either pragma is USELESS (and hence does not
>> produce any reasonable effect, neither slowness), either is
>> NECESSARY (and so the slowing effect is a unavoidable price
>> that I MUST pay either)
> 
> Correct - it is both useless and unnecessary - in this context. Don't
> extrapolate from this case to the general case. It is very specifically
> the fact that you are trying to pack something that cannot possibly be
> bigger than a byte, and therefore cannot have an alignment requirement
> bigger than 1, that renders the pragma useless and unnecessary.

when I say "in that case" I am not extrapolating, I am 
contextualizing to the specific case

> 
>>>    If you don't know what you are doing
>>
>> I'm rather weak in higher level C++ constructs, and I know
>> that, but not particularly so in low level (some couple of
>> tens of years ago I used to stuff C code with inline "asm"
>> statements or even __emit() calls).
> 
> You don't appear to understand why the fact that you're trying to use an
> 8-bit type guarantees that #pragma pack will be useless and unnecessary
> - so there's some low-level stuff you don't understand yet.
> 
>>> And when the more experienced
>>> people in a group like this all say "don't use packed" and "use
>>> uint8_t",
>>
>> I didn't read or desume the two things should be mutually
>> exclusive.
>>
>> Anyway, there are three scenarios
>>
>> 1) added pragma (to the four u_int_8 members in a struct) is
>> USELESS, as the struct was just packed natively (and hence
>> does not produce any reasonable effect, neither slowness),
>> either
>> 2) is NECESSARY, as the struct would not have been natively
>> packed otherwise (and so the slowing effect is a unavoidable
>> price that I MUST pay either)
>> either
>> 3) it is NOT SUFFICIENT (in this case, luckily unlikely,
>> nothing would have worked fine at all)
> 
> On implementations where CHAR_BIT==8, scenario #1 applies. On
> implementations where CHAR_BIT>8, scenarios #1 and #3 both apply.
> 
> ...
>> 	u_int_32  AllChannels;
> 
> uint_32

maybe QT headers decorate differently

here it is u_int32_t

> 
> 
>>> in the sense of having unaligned data that would have
>>> padding in a normal struct.  I have seen no answer to this (unless it is
>>> later in this post), despite asking this /crucial/ question.
>>
>> the files are packed.
> 
> OK - which fields are positioned differently from the way they would be
> positioned if not packed?

as I said, I do not have a working complete program yet, and 
I hope to get to a scenario where this wrong situation is 
avoided at all

> 


-- 
1) Resistere, resistere, resistere.
2) Se tutti pagano le tasse, le tasse le pagano tutti
Soviet_Mario - (aka Gatto_Vizzato)

[toc] | [prev] | [next] | [standalone]

#70005

On 11/22/19 6:25 PM, Soviet_Mario wrote:
> Il 22/11/19 20:11, James Kuyper ha scritto:
>> On 11/22/19 1:35 PM, Soviet_Mario wrote:
>>> Il 22/11/19 15:04, David Brown ha scritto:
>> ...
>>> I have abandoned the unsigned Bit_Field : 8 solution as soon
>>> as you (a lot of people) stated that u_int_8 MUST have
>>
>> uint8_t
> 
> here the macro or typedef is
> 
> u_int8_t actually

Everyone who's been talking with you about this subject has been talking
about std::uint8_t from <cstdint>. If you're talking about something
else, you need to identify what it is and where is it is declared.

In gcc, and in many other compilers as well, there's a command line
option (-E in gcc) that produces output after pre-processing, but before
any other of the translation phases, but with #line directives inserted
to make sure that if you compile the output from -E, any resulting error
messages will refer to the correct file.
Compile one of the modules that uses u_int8_t using -E. Search the
output for the first occurrence of u_int8_t, which should be a typedef
declaration. What file is referred to by the last #line directive
preceding that declaration?

...
>>> 	u_int_32  AllChannels;
>>
>> uint_32
> 
> maybe QT headers decorate differently

A search of the QT online reference manual for u_int8_t turns up nothing:
<https://doc.qt.io/qt-5/search-results.html?q=%22u_int8_t%22>

Nor do I get anything from u_int_32:
<https://doc.qt.io/qt-5/search-results.html?q=%22u_int_32%22>

Whatever is the source of this typedef, I don't think it's QT.

>>>> in the sense of having unaligned data that would have
>>>> padding in a normal struct.  I have seen no answer to this (unless it is
>>>> later in this post), despite asking this /crucial/ question.
>>>
>>> the files are packed.
>>
>> OK - which fields are positioned differently from the way they would be
>> positioned if not packed?
> 
> as I said, I do not have a working complete program yet, and 
> I hope to get to a scenario where this wrong situation is 
> avoided at all

I think you can avoid it perfectly without doing anything at all about
it, because it can't occur. I recommend not wasting any more time
worrying about it until you can come up with a plausible reason for
doing so.

[toc] | [prev] | [next] | [standalone]

#70009

Il 23/11/19 05:56, James Kuyper ha scritto:
> On 11/22/19 6:25 PM, Soviet_Mario wrote:
>> Il 22/11/19 20:11, James Kuyper ha scritto:
>>> On 11/22/19 1:35 PM, Soviet_Mario wrote:
>>>> Il 22/11/19 15:04, David Brown ha scritto:
>>> ...
>>>> I have abandoned the unsigned Bit_Field : 8 solution as soon
>>>> as you (a lot of people) stated that u_int_8 MUST have
>>>
>>> uint8_t
>>
>> here the macro or typedef is
>>
>> u_int8_t actually
> 
> Everyone who's been talking with you about this subject has been talking
> about std::uint8_t from <cstdint>. If you're talking about something
> else, you need to identify what it is and where is it is declared.

I'm using QT who seems to like to replicate anything its own 
way even when avoiding specific Q-prefixes.

so, HERE, including
#include <sys/types.h>

I find this type (everyone might guess it is equivalent to 
the other), u_int8_t, which finally stems from a typedef to 
UNSIGNED CHAR.


> 
> In gcc, and in many other compilers as well, there's a command line
> option (-E in gcc) that produces output after pre-processing, but before
> any other of the translation phases, but with #line directives inserted
> to make sure that if you compile the output from -E, any resulting error
> messages will refer to the correct file.

I'm not so confident with QT environment to manually set 
options to the compiler as yet, I just use the ide-forms to 
set or unset some warning levels (I keep enabled almost all 
even if the outcome is very verbose)

> Compile one of the modules that uses u_int8_t using -E. Search the
> output for the first occurrence of u_int8_t, which should be a typedef
> declaration.

it is an unsigned char, I saw it DIRECTLY opening the 
included header file (sys/types.h)

this type is also replicated in another header
<stdint.h> actually, ending up always in unsigned char


> What file is referred to by the last #line directive
> preceding that declaration?
> 
> ...
>>>> 	u_int_32  AllChannels;
>>>
>>> uint_32
>>
>> maybe QT headers decorate differently
> 
> A search of the QT online reference manual for u_int8_t turns up nothing:
> <https://doc.qt.io/qt-5/search-results.html?q=%22u_int8_t%22>
> 
> Nor do I get anything from u_int_32:
> <https://doc.qt.io/qt-5/search-results.html?q=%22u_int_32%22>

sorry, but not an issue.

> 
> Whatever is the source of this typedef, I don't think it's QT.

omg ... are you saying I am just lieing for the fun of it ???
what the heck am I supposed to know about which kind of 
headers has included this install ?
I am using the FREE version from a .DEB package (which in 
turn launched an updated and installed tons of "kits" by 
itself").
I just opened the headers and never dreamed about editing 
them. Just did a search.

the header file reports
/* Copyright (C) 1991-2016 Free Software Foundation, Inc.
    This file is part of the GNU C Library.

    The GNU C Library is free software; you can redistribute 
it and/or
    modify it under the terms of the GNU Lesser General Public
    License as published by the Free Software Foundation; either
    version 2.1 of the License, or (at your option) any 
later version.

    The GNU C Library is distributed in the hope that it 
will be useful,
    but WITHOUT ANY WARRANTY; without even the implied 
warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. 
See the GNU
    Lesser General Public License for more details.

    You should have received a copy of the GNU Lesser 
General Public
    License along with the GNU C Library; if not, see
    <http://www.gnu.org/licenses/>.  */

/*
  *	POSIX Standard: 2.6 Primitive System Data Types 
<sys/types.h>
  */


and the macros and typedefs for such sized ints seem to 
apply for GCC >= 2.7  (but I might misunderstand)

#if !__GNUC_PREREQ (2, 7)



> 
>>>>> in the sense of having unaligned data that would have
>>>>> padding in a normal struct.  I have seen no answer to this (unless it is
>>>>> later in this post), despite asking this /crucial/ question.
>>>>
>>>> the files are packed.
>>>
>>> OK - which fields are positioned differently from the way they would be
>>> positioned if not packed?
>>
>> as I said, I do not have a working complete program yet, and
>> I hope to get to a scenario where this wrong situation is
>> avoided at all
> 
> I think you can avoid it perfectly without doing anything at all about
> it, because it can't occur. I recommend not wasting any more time
> worrying about it until you can come up with a plausible reason for
> doing so.
> 


-- 
1) Resistere, resistere, resistere.
2) Se tutti pagano le tasse, le tasse le pagano tutti
Soviet_Mario - (aka Gatto_Vizzato)

[toc] | [prev] | [next] | [standalone]

#70014

On 23.11.2019 12:12, Soviet_Mario wrote:
> Il 23/11/19 05:56, James Kuyper ha scritto:
>> On 11/22/19 6:25 PM, Soviet_Mario wrote:
>>> Il 22/11/19 20:11, James Kuyper ha scritto:
>>>> On 11/22/19 1:35 PM, Soviet_Mario wrote:
>>>>> Il 22/11/19 15:04, David Brown ha scritto:
>>>> ...
>>>>> I have abandoned the unsigned Bit_Field : 8 solution as soon
>>>>> as you (a lot of people) stated that u_int_8 MUST have
>>>>
>>>> uint8_t
>>>
>>> here the macro or typedef is
>>>
>>> u_int8_t actually
>>
>> Everyone who's been talking with you about this subject has been talking
>> about std::uint8_t from <cstdint>. If you're talking about something
>> else, you need to identify what it is and where is it is declared.
>
> I'm using QT who seems to like to replicate anything its own way even
> when avoiding specific Q-prefixes.
>
> so, HERE, including
> #include <sys/types.h>

<sys/types.h> belongs to The Open Group which is "the certifying body 
for the UNIX trademark". It has nothing to do with QT and neither with 
standard C++. Also, it appears the u_int8_t typedef is not documented, 
meaning that it is an implementation detail which you can't rely on. For 
example, in Windows there is a <sys/types.h> header, but it does not 
define u_int8_t.

So why cannot you listen to everybody here who are trying to give you 
good advice, and use <cstdint> and std::uint8_t?

[toc] | [prev] | [next] | [standalone]

#70060

On 11/23/19 5:12 AM, Soviet_Mario wrote:
> Il 23/11/19 05:56, James Kuyper ha scritto:
>> On 11/22/19 6:25 PM, Soviet_Mario wrote:
>>> Il 22/11/19 20:11, James Kuyper ha scritto:
...
>>>> uint8_t
>>>
>>> here the macro or typedef is
>>>
>>> u_int8_t actually
>>
>> Everyone who's been talking with you about this subject has been talking
>> about std::uint8_t from <cstdint>. If you're talking about something
>> else, you need to identify what it is and where is it is declared.
> 
> I'm using QT who seems to like to replicate anything its own 
> way even when avoiding specific Q-prefixes.
> 
> so, HERE, including
> #include <sys/types.h>
> 
> I find this type (everyone might guess it is equivalent to 
> the other), u_int8_t, which finally stems from a typedef to 
> UNSIGNED CHAR.

Please do not change the capitalization of the characters in C++ code.
As I keep telling you, C++ is a case-sensitive language. If UNSIGNED
CHAR even exists, in principle, it's not necessarily related in any way
to unsigned char. In practice, there's likely to be a connection, but if
anyone did bother creating, for instance, macros with those names, the
connections is unlikely to be a simple one. In such as case, they
probably defined those macros precisely because there are circumstances
where they do NOT expand to "unsigned char".

Such names as u_int8_t are, as far as C++ itself is concerned, reserved
for use by user code. However, POSIX reserves identifiers ending in _t
to itself to represent types, and sys/types.h is not a C++ standard
header, so I that makes it legal for that header to declare such
identifiers. However, I can find no mention of them in the Single UNIX
standard, so I've no idea why they actually are declared. There's
certainly no good reason for using them rather than uint8_t for this
purpose.


>> Whatever is the source of this typedef, I don't think it's QT.
> 
> omg ... are you saying I am just lieing for the fun of it ???

No, I'm just assuming that you're confused. Using u_int8_t rather than
uint8_t, just because such an identifier exists, tends to justify such
assumptions.

[toc] | [prev] | [next] | [standalone]

Page 3 of 4 — ← Prev page 1 2 [3] 4  Next page →

Back to top | Article view | comp.lang.c++

csiph-web