Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #69727

Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding

Show all headers | View raw

On 24/11/2019 20:53, David Brown wrote:
> <snip>
> 
>>> And if your file format contains non-aligned members, 
>>
>> why making general guess when I reported precisely the 
>> layout ?
>> I said :
>> 16 bit (width) - 16 bit (height) - N x 32 bit (R,G,B,alpha)
>> where N = width x height = number of "pixels"
> 
> That does not tell us anything about alignment.  Are your 
> fields properly aligned or not?

these fields start at the offset 0 of the file.
And they are loaded (fwrite) not in a generic void * buffer 
but in an array of typed structs of the kind we are saying.
So I think they are ...

> 
>>
>> every pixel is multiple of 32 address on file (but this 
>> seems irrelevant to me).
> 
> I presume you mean multiple of 32 bits, i.e., 4 bytes.  And 
> yes, this is /highly/ relevant.  It means your data is well 
> aligned, and you can use a union with a uint32_t for your 
> pixels (if you want).  Are you also sure that your 16-bit 
> values (height and width) are 16-bit aligned? 

they are in the FILE only, but are freaded separately.
The struct itself does not contain these fields.

> As long as 
> all your elements are naturally aligned, then the struct is 
> properly aligned and you can use a plain, simple struct with 
> no "packed" complications.

okay.

> 
> This is /critical/ to being able to support your structure 
> simply, cleanly, efficiently and portably.  That is why I 
> have repeatedly asked you if your format is properly aligned.
> 
>> In the ram layout,
>> the BMP descriptor has width and height as "local" 
>> members, and a pointer to an array of struct aligned in 
>> memory by the NEW [] operator who is aware of alignment 
>> requirements.
>>
>> Where are the possible problems ?
>> What problems are affected by a pragma pack directive ?
>>
>>
>>> then the simplest
>>> and easiest way to improve and simplify the code, and 
>>> improve the speed
>>> of several parts, is to fix the file format.
>>
>> that point is unmodificable. I strictly need full color 
>> and no compression. The alpha channel is maybe a bit 
>> overkill (255 level of transparency is more than needed 
>> ... but taking less is useless or impossible, and 1 byte 
>> is natural for the otherwise 3,sth bytes long struct)
> 
> I haven't suggested using smaller types for your data - I 
> assume that these are appropriate types.
> 
>>
>>>  Make sure the members are
>>> aligned (add explicit dummy members if necessary)
>>
>> since when reading/writing files suffer from this ? Files 
>> are BYTE oriented internally (and the OS does caching 
>> independently and transparently, basing on disk cluster 
>> sizes but this is not sth I should care of)
> 
> It does not affect the file reading and writing - it affects 
> the struct in memory that is accessed by the code.  C and 
> C++ only define access to correctly aligned objects and 
> subobjects.  Access to unaligned data is implementation 
> dependent, may require extra extensions (like pragma pack), 
> may be subject to various restrictions, may be inefficient 
> (either extra code, or inefficiencies in the cpu 
> processing), and may cause problems if the data is used with 
> third-party libraries or code.

we agree that this will not happen here
CUT

>>>
>>> However you look at it, "pack" is usually (but not 
>>> always) the wrong
>>> choice.  You should only consider it once you have 
>>> eliminated better
>>> alternatives.
>>
>> for a starting, I tried to use a 32 bit total size, hoping 
>> it would have been packed regardless of any "hints" (pragma).
> 
> That is a good plan.  When you do that, you avoid any need 
> for "pack" extensions.
> 
> It appears that you have designed a /sensible/ file format.  
> It is /simple/, but certainly not "stupid".  It has taken a 
> while to get you to give enough details to establish this, 
> but it seems to have been a sensible format all along.
> 
> What you have got wrong is your determination to distrust 
> your own sensible decisions in the file layout and add worse 
> than useless extra "pack" junk.  Forget "pack" - forget you 
> ever heard of it.  It is not part of the C++ language, it is 
> rarely useful, and it is certainly not useful here.  Just 
> use your file format in the clear, sensible layout you have 
> designed.

LOL ... okay ! Surrendered :)


>>> First, you take a step back.  You look to see if the 
>>> packed file really
>>> is "packed",
>>
>> it is, as both Gambas supports BYTE format (And sizeof()) 
>> and BYTE-oriented streams. No overspace is generated. Then 
>> I have to worry about the C++ back end, to comply with 
>> this fact.
> 
> My knowledge of Gambas is very limited, but according to the 
> documentation I found its structs are /not/ packed:

no, they arent, but the internal format it uses for 2D 
graphics is. Also my very simple structure layout (4 bytes) 
was easy enough for even gambas to produce a file with the 
expected size (and consistent content, as I realoaded it).

> 
> <http://gambaswiki.org/wiki/lang/structdecl>
> 
> Padding /is/ generated - if needed to get the natural 
> alignment of the fields.  You can be entirely confident that 
> on any platform supported by Gambas, the padding, alignment 
> and layout of a Gambas struct and a C++ struct will be the 
> same.

I had the sensation gambas is little more than a wrapper 
around C, with some syntactic sugar.

> 
>>
>> I think the designer of Gambas relies very strongly upon C 
>> libraries and mind-shape, apart from strings (where he had 
>> to welcome BASIC users who want strings the basic way).
> 
> Of course.  And while the details of C struct layouts is 
> implementation dependent, it is invariably the simplest and 
> most obvious but efficient layout for the target.

in fact it was working OK in gambas.
I would have stayed there, the performance were very good, 
but I was starting not to be able to manage the code.
C++ helps so much to organize the structure and to think 
correctly.
I am also changing to huge switch () case: to arrays of 
pointers to functions for single bit manipulation variants.

Uh ... btw, I go Off Topic.

I was declaring some 3 lines long function as inlined
But, as I started to get their address and populate an array 
of pointers, I removed the inline suggestion.

just for, lol :) precaution.
What's the effect of inline linkage when one takes the 
address of a function (and stores it) ? I assumed, without 
reading documentation supporting this, It would just ignore 
inline suggestion.
Or is it undefinite behaviour ?


>>>
>>> If the format is already good, you are done.  One memcpy, 
>>> or read from
>>> the file directly to a suitable buffer, or memmap to the 
>>> file directly
>>> (no reading or writing needed).
>>
>> memmap is a problem in that I don't know how exactly works 
>> and whether or not I'll be able to interoperate with 
>> memory files opened in gambas
>>
>> I am also considering to use some memory files supported 
>> directly by the OS (here on linux files created on /shm 
>> virtual folder are designed for such FAST temporaries to 
>> be shared, but with portable stream syntax).
>> Using /shm would imply no specialized load/store code, 
>> just the normal file access => delegating the ram usage to 
>> the OS.
> 
> If your files are in a ram file system (/shm, tmpfs, etc.) 
> then memmap would not be faster anyway.

I am reading now (The GNU C library reference manual)
that there are functions seeming to directly and 
specifically support the /shm shared virtual folder

shm_open, shm_unlink =>
both support NAMED memory streams
and they seem very fit for sharing big pictures, as
exchanging ascii names will be simpler and more readable;
at need even "hardwired" in constants : there won't be many 
temporaries at a time, and recycled over and over again in 
case, so not a true limitations. Once the results are 
passed, they will be saved in regular files (.PNG or .JPG).

also memfd_create seem to have this same advantage (use 
names instead of file descriptors).


But this will be the homework for when I am finished the 
other parts.


> 
>>
>> well, I'd be considering these for small size, complex 
>> structured input, not for a 4-12 MB size of simple pixels. 
>> It would be a blood bathe
> 
> You are basing that on speculation with no evidence.
> 
> I am not at all saying that it is a bad idea to use a direct 
> mapping from file format to struct - indeed, it is a 
> sensible idea whenever you can define the format.
> 
> As far as I understand your project, you are using Gambas 
> for a gui but for some kind of image processing, you are 
> saving the pictures from Gambas as files, 

originally I was doing in gambas, saving / loading included 
(as it exports in standard formats, allows for desktop 
grabbing and so), but also then I converted colors to my 
modified format for greater control and effects that I did 
not find the way I wanted them.

Then it grow and I started to no longer be able to "read the 
code", find things, know the impact of modifications.

the fact is that in C++ layers are fare more isolated and 
when code is written well, modifications often are just 
local and do not alter the way other layers use the underlying.
In fact Im slowing regaining control of what happens :)

but the GUI with QT is out of my skills. I just roughly 
understand its tipical sockets/signal lazy binding.

> then starting an 
> external C++ program that reads these files, processes them 
> in some way, and saves them again, and then the Gambas 
> program will read them in again.

possibly using /shm files, not to slow down too much

> 
> I have no idea if this is a good method - 

no, in general is not. But I'm not a "pro" neither in gambas 
nor in C++, so I am trying to mix things to use the easier 
part of each in its own natural context.
The GUI in gambas is really simple to manage, compared to 
QT. Almost as easy as Visual Studio.

But I was losing control over gambas code, it does not 
enforce nor allows for tidyness and sealed objects, its 
project management is a nightmare (one file => one class) to 
me, so I simply spent a lot of time re-reading my own code.
I was obliget to return to C++ to sweep the dirt :)


> either from the 
> viewpoint of development ease, or run-time efficiency.  I 
> can think of many alternative ways to structure this task, 
> which may or may not be better - without knowing details of 
> the task, or why you might prefer a particular setup.

the bottleneck is by far my own skills with this two tools.
With the old Visual Basic 2012 or C# I would have finished 
it all in days.
But I left windows, so I have to use other tools I am not 
much familiar with
I cannot stand windows, but I must admit Visual Studio was 
really powerful, complete, cozy and all, and above all 
worked always out of the bag, preconfigured consistently. I 
really miss it ! QT creator is hard to configure :\

> 
> But it does seem to me that you are making your decisions 
> based on gut feeling rather than research or measurement.  

absolutely true. But I AM the bottleneck :) of the all chain.

> That does not mean your decisions are wrong, but it might 
> mean you should not be so confident in them.

oh I am not confident in the solution : I simply crashed in 
using gambas alone for both the gui and more complex 
elaborations (struggling with the computations), and also 
using QT alone for both (struggling with GUI designer).

Then I decided to try to split the interface from the 
engine, at the price of data exchange (possibly with memory 
mapped files). I did not have much else left to do.

Maybe I'll crash again, and resign to reinstall a virtual 
machine with Visual Studio inside ! LOL.
No ... I'm not so willing to resign actually.



> enough.
> 
> I want you to understand what is important when making such 
> structures - that everything is aligned properly (the best 
> being "natural" alignment), and I strongly recommend adding 
> any padding manually as unused "padding" fields.  I want you 
> to understand /why/ this is important.  And once you 
> understand that, you will realise that "pragma pack" and 
> similar extensions are useless here and not only are you 
> able to avoid them, but you /should/ avoid them.


all true, but as I said : the file is packed. And I don't 
want accessors like iomanip and that stuff.
I need to binary write in one block


CUT


>>
>> oh I completely disagree with this point ! The format has 
>> to be packed as the C++ library receives the file from an 
>> other source of data that offers them in that form, so 
>> it's mandatory.
> 
> No, it is not.  Not remotely.  It would be a sad state for C 
> and C++ if they required compiler-specific extensions to 
> deal with data from external sources.
> 
> I think perhaps you are mixing up the concepts of "a 
> structure where the fields are adjacent with no padding" and 
> "uses pragma packed".  I am recommending using structs fit 
> the former, without ever using the later.

okay. Some day I'll test if it loads well or not.

> 
> Or perhaps you think compilers add random padding and 
> re-arrange structs in odd and unpredictable ways. 

:) well, not randomly. I think they just try to keep int_32, 
float at addresses multiple of 4, and int_64, double 
multiple of 8 and so.

I'm not sure of the degree of freedom they have in 
"reordering" members though.
Sometimes in constructors I get some warning about the 
members will be initializated in a different order than 
declared.
But I did it not too much care, as they did not depend on 
each other and so actual order was not important.

> The 
> details here are usually specified by the platform's ABI, 
> precisely so that programs can exchange data safely (by 
> files, pipes, streams, shared memory, shared libraries, etc.).
> 
>> And also the caller expects to get the transformed data 
>> back in the same packed form.
>>
>>> means totally pointless complications, inefficiencies, 
>>> and non-portable
>>> code.
>>
>> nope, I have constraints to fullfil. Maybe inefficient, 
>> maybe non portable (who cares ?), but surely not 
>> pointless, as otherwise data exchange would not work at all
> 
> "pragma pack" is pointless here.
> 
>>
>>>
>>> If you have learned nothing else from this conversation, 
>>> hopefully you
>>> have learned that much.
>>>
>>>>

ok ... discussion closed, apart from some OT issues


-- 
1) Resistere, resistere, resistere.
2) Se tutti pagano le tasse, le tasse le pagano tutti
Soviet_Mario - (aka Gatto_Vizzato)

Back to comp.lang.c++ | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-24 21:53 +0100
  Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Öö Tiib <ootiib@hot.ee> - 2019-11-24 13:46 -0800
    Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Soviet_Mario <SovietMario@CCCP.MIR> - 2019-11-25 12:44 +0100
      Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Öö Tiib <ootiib@hot.ee> - 2019-11-25 06:55 -0800
  Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding James Kuyper <jameskuyper@alumni.caltech.edu> - 2019-11-25 13:18 -0500
    Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding David Brown <david.brown@hesbynett.no> - 2019-11-26 08:22 +0100
      Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding Paavo Helde <myfirstname@osa.pri.ee> - 2019-11-26 10:19 +0200
        Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding David Brown <david.brown@hesbynett.no> - 2019-11-26 10:15 +0100
      Re: [QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding James Kuyper <jameskuyper@alumni.caltech.edu> - 2019-11-26 11:59 -0500

csiph-web