Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #10864 > unrolled thread

reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?)

Back to article view | Back to comp.lang.c++

Contents

  reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-15 12:00 -0700
    Re: reading binary file into memory. Converting from char to uint32,   float, double, ASCII strings etc (static_cast< > ?) Ian Collins <ian-news@hotmail.com> - 2011-10-16 09:29 +1300
      Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-15 14:17 -0700
        Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Ebenezer <woodbrian77@gmail.com> - 2011-10-15 14:41 -0700
        Re: reading binary file into memory. Converting from char to uint32,   float, double, ASCII strings etc (static_cast< > ?) Ian Collins <ian-news@hotmail.com> - 2011-10-16 10:45 +1300
          Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Martin Jørgensen <mjoerg@gmail.com> - 2011-10-15 15:43 -0700
            Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-15 21:37 -0400
              Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-15 23:23 -0700
                Re: reading binary file into memory. Converting from char to uint32,   float, double, ASCII strings etc (static_cast< > ?) Ian Collins <ian-news@hotmail.com> - 2011-10-16 20:53 +1300
                  Re: reading binary file into memory. Converting from char to uint32,   float, double, ASCII strings etc (static_cast< > ?) Paavo Helde <myfirstname@osa.pri.ee> - 2011-10-16 03:04 -0500
                    Re: reading binary file into memory. Converting from char to uint32,   float, double, ASCII strings etc (static_cast< > ?) Ian Collins <ian-news@hotmail.com> - 2011-10-16 21:52 +1300
              Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Jorgen Grahn <grahn+nntp@snipabacken.se> - 2011-10-16 18:21 +0000
                Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-16 16:40 -0400
                  Re: reading binary file into memory. Converting from char to uint32,   float, double, ASCII strings etc (static_cast< > ?) Ian Collins <ian-news@hotmail.com> - 2011-10-17 10:00 +1300
                    Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-16 15:19 -0700
                      Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-16 20:03 -0400
                        Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-16 17:49 -0700
                          Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-16 21:29 -0400
                  Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Jorgen Grahn <grahn+nntp@snipabacken.se> - 2011-10-17 09:10 +0000
    Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-15 17:35 -0400
      Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Martin Jørgensen <mjoerg@gmail.com> - 2011-10-15 15:39 -0700
        Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-15 16:12 -0700
          Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-15 16:32 -0700
            Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-15 19:41 -0400
              Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-16 11:33 -0700
                Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Paavo Helde <myfirstname@osa.pri.ee> - 2011-10-16 13:54 -0500
                  Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Paavo Helde <myfirstname@osa.pri.ee> - 2011-10-16 14:01 -0500
                  Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-16 15:12 -0700
                    Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-16 20:14 -0400
                      Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-16 17:56 -0700
        Re: reading binary file into memory. Converting from char to uint32,   float, double, ASCII strings etc (static_cast< > ?) Ian Collins <ian-news@hotmail.com> - 2011-10-16 12:33 +1300
          Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-15 16:37 -0700
          Re: reading binary file into memory. Converting from char to uint32,   float, double, ASCII strings etc (static_cast< > ?) Leigh Johnston <leigh@i42.co.uk> - 2011-10-16 19:59 +0100
            Re: reading binary file into memory. Converting from char to uint32,   float, double, ASCII strings etc (static_cast< > ?) Leigh Johnston <leigh@i42.co.uk> - 2011-10-16 20:13 +0100
              Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Joshua Maurice <joshuamaurice@gmail.com> - 2011-10-17 18:17 -0700
                Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Joshua Maurice <joshuamaurice@gmail.com> - 2011-10-17 18:32 -0700
    Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Pavel <pauldontspamtolk@removeyourself.dontspam.yahoo> - 2011-10-15 19:56 -0400
      Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-15 23:27 -0700
        Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Paavo Helde <myfirstname@osa.pri.ee> - 2011-10-16 02:48 -0500
          Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-16 10:57 -0700
            Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Paavo Helde <myfirstname@osa.pri.ee> - 2011-10-16 13:18 -0500
              Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-16 12:52 -0700
    Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-16 10:58 -0700
      Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Paavo Helde <myfirstname@osa.pri.ee> - 2011-10-16 13:37 -0500
        Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-16 14:37 -0700
          Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Paavo Helde <myfirstname@osa.pri.ee> - 2011-10-17 01:21 -0500

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#10872

On Oct 15, 11:35 pm, Richard Damon <news.x.richardda...@xoxy.net>
wrote:
> On 10/15/11 3:00 PM, someone wrote:
==
> > I'm stuck...
>
> > And what about big endian / little endian? I'm confused. Looking
> > forward to hear from you!
>
> First, what is the first byte in the file? you are saying that you are
> expecting it to be 72 but you are getting 49. Have you looked at the file.

Good point. The first four bytes of the input file has the values:
[048h 00h 00h 00h] (hexadecimal, I used a hex editor to see this). And
0x48H = 72 decimal... I don't even know if this is little endian or
not, but what I find very strange is the number 49 then... That number
makes no sense to me (why is it adding 1, if it's printing the number
in hexadecimal? What's it thinking?)..?

> Next you talk about getting R1begin which should be 4 bytes, but you are
> setting it to the value of the first byte of the file (memblock[0] is
> the first byte of the file) the static_cast to int is unneeded, as char
> will convert to int implicitly.

Oops, sorry. You're right. I should take the first 4 bytes and convert
them into the correct integer format (which I think is unsigned int
32, but I'll have to test it). However, I still don't know exactly how
to do the conversion, sorry, I'm a bit unexperienced and it took quite
some hours to produce the first code in the original post :-(

> Lastly, you are saying that you are expecting to read 72 bytes at the
> beginning of the file, and also that you are expecting to get the value
> 72 from the beginning of the file, implying that R1begin is a byte count
> for the record, but then don't describe the record format depending on

Exactly.

> that value at all, and then only look at 1 byte for that info, when you
> describe it as a 4 byte number.

You're right. I gave up on the last and concluded that I needed some
help. I was trying to get inspired by some template<T>-code I found.
However that code only worked for reading directly from the file and
now I copied the whole file into memory, in a buffer... The correct
thing to do, is to take out the first  4 bytes and then convert that
into the relevant type. I'm currently struggling with that.

> If R1_begin ins't a data length field, why are you expecting it to have
> the value 72?

You're completely right. It is the data length field and I expect to
read in the value of 72 because: Total bytes read = 4+24+40+4 = 72
bytes. You don't happen to have some genius template that works for
this kind of problem, do you?

I'm sorry, but I still don't get the right value of 72. Here are 2 new
attempts, and I don't quite understand the output:


  int R1_begin;
  R1_begin = *(reinterpret_cast<char*> (&memblock));
  cout << "Value = " << R1_begin << endl; // writes out: Value = 96 ??
why ?


  int R1_begin; // makes no difference if I write unsigned int
R1_begin here!
  memcpy(&R1_begin, &memblock[0], sizeof(R1_begin));
  cout << "Value = " << R1_begin << endl; // writes out: Value =
875770417 ?? why ?


Once I get understand this and get it right, I'll look into the endian-
thing (I'm still not sure whether I have an endian problem or not)...

[toc] | [prev] | [next] | [standalone]

#10874

On 16 Okt., 00:39, Martin Jørgensen <mjo...@gmail.com> wrote:
Sorry if it caused any confusion, but I'm the original poster (I had
two browser windows open with 2 different gmail accounts).
I now found out that my system is little endian. I also think the data
in the file is stored in little endian format (hex editor shows the
first four bytes: [048h 00h 00h 00h]) and that should correspond to an
integer value of 72, even though I cannot get it right yet... Guess
I'm casting incorrectly from char* to int.

[toc] | [prev] | [next] | [standalone]

#10875

On 16 Okt., 01:12, someone <newsbo...@gmail.com> wrote:
> integer value of 72, even though I cannot get it right yet... Guess
> I'm casting incorrectly from char* to int.

Oooops... Stupid me! I had outcommented the correct file name and used
a much smaller working file in the line: infile.open("DELETEME.res",
ios::in|ios::binary)

--- sorry guys, maybe it's too late to work for me now! :-)


Last question for today and I shall soon close the thread: Can I ask
you which of the following you think is the best to use (or suggest
alternatives)?


  if (0) // my way of switching between alternate "solutions" when I
want both in the code
     R1_begin = *(reinterpret_cast<unsigned char*> (&memblock[0]));//,
1, 'uint32',endianNess);
  else
     memcpy(&R1_begin, &memblock[0], sizeof(R1_begin));


?

[toc] | [prev] | [next] | [standalone]

#10878

On 10/15/11 7:32 PM, someone wrote:
> On 16 Okt., 01:12, someone<newsbo...@gmail.com>  wrote:
>> integer value of 72, even though I cannot get it right yet... Guess
>> I'm casting incorrectly from char* to int.
>
> Oooops... Stupid me! I had outcommented the correct file name and used
> a much smaller working file in the line: infile.open("DELETEME.res",
> ios::in|ios::binary)
>
> --- sorry guys, maybe it's too late to work for me now! :-)
>
>
> Last question for today and I shall soon close the thread: Can I ask
> you which of the following you think is the best to use (or suggest
> alternatives)?
>
>
>    if (0) // my way of switching between alternate "solutions" when I
> want both in the code
>       R1_begin = *(reinterpret_cast<unsigned char*>  (&memblock[0]));//,
> 1, 'uint32',endianNess);
>    else
>       memcpy(&R1_begin,&memblock[0], sizeof(R1_begin));
>
>
> ?

The first doesn't do what you want. &memblock[0] is already a char* so 
casting it to unsigned char* and dereferencing isn't going to make a big 
change. If you did a cast to unsigned int*, that would be different.


You also shouldn't use reinterpret_cast here, static_cast is good 
enough, and is actually what you want.

The memcpy is better, as the cast has the danger that some fields might 
not line up on the right word boundaries in your data buffer. It also 
says you can later change them to the network order calls later to make 
your program more portable.

[toc] | [prev] | [next] | [standalone]

#10895

On 16 Okt., 01:41, Richard Damon <news.x.richardda...@xoxy.net> wrote:
> On 10/15/11 7:32 PM, someone wrote:
...
> > Last question for today and I shall soon close the thread: Can I ask
> > you which of the following you think is the best to use (or suggest
> > alternatives)?
>
> >    if (0) // my way of switching between alternate "solutions" when I
> > want both in the code
> >       R1_begin = *(reinterpret_cast<unsigned char*>  (&memblock[0]));//,
> > 1, 'uint32',endianNess);
> >    else
> >       memcpy(&R1_begin,&memblock[0], sizeof(R1_begin));
>
> > ?
>
> The first doesn't do what you want. &memblock[0] is already a char* so
> casting it to unsigned char* and dereferencing isn't going to make a big
> change. If you did a cast to unsigned int*, that would be different.

Ok, it was probably a coincidence that it worked... It sounds right,
what you write. Thanks.

> You also shouldn't use reinterpret_cast here, static_cast is good
> enough, and is actually what you want.

I ended up doing this (which I find is quite good, I stole some part
of the idea from the internet):

//=================================
template <class T>
void moveBytes(T *destination, const char *source, const unsigned int
multiplier=1)
{
  //  cout << "sizeof(*destination) = " << dec << sizeof(*destination)
<< endl;
  memcpy(destination, &source[memLocation],
		 (multiplier*sizeof(*destination)));
  memLocation += (multiplier*sizeof(*destination)); // memLocation =
global variable/counter!
}
//=================================

This allows me to extract different data types from memory and do,
e.g.:

  unsigned int R1_begin;
  moveBytes(&R1_begin, memblock); // memLocation automatically
increases for next time

  unsigned int * R2_SensNum;
  R2_SensNum = new unsigned int[R2_N];
  moveBytes(R2_SensNum, memblock, R2_N);

etc... And I think it works for all data types: floats, doubles,
uint32, int32, whatever...


> The memcpy is better, as the cast has the danger that some fields might
> not line up on the right word boundaries in your data buffer. It also
> says you can later change them to the network order calls later to make
> your program more portable.

Ok, memcpy it is :-)

Now, I have one last serious issue with my program before it works...
I sucessfully loaded all the "header"-information by using the
moveBytes-template (see above). After the header, follows a long loop
of consecutive data records so I do something like this (short
version, some irrelevant code parts is omitted):


//=================================
 RTSensor_array = new float[R2_N]; // should be array of size R2_N!
 vector<float> RTSensorReading[R2_N]; // R2_N = 10 (this is the sensor
array length)

 while ( memLocation < fsize ) // fsize was the filesize, memLocation
is also seen above (global)
	{
	  curLoopNumber++; // keep track of number of records (for debugging)

	  moveBytes(&myuint_begin, memblock); // this works, read 4 bytes!
	  moveBytes(&float_time, memblock); // this works, read 4 bytes!
	  moveBytes(&myuint_end, memblock); // this works, read 4 bytes!
	  moveBytes(&RTSensor_array, memblock); // issue: much longer than 4
bytes, i.e. 40++ bytes

	  RTbegin.push_back( myuint_begin ); // this works, save 4 bytes!
	  RTtime.push_back( float_time ); // this works, save 4 bytes!
	  RTend.push_back( myuint_end ); // this works, save 4 bytes!
	  RTSensorReading.push_back( RTSensor_array ); // <==== ISSUE, not
working, trying to save 40 bytes!
	}
//=================================

The last line gives this compiler error:
error: request for member ‘push_back’ in ‘RTSensorReading’, which is
of non-class type ‘std::vector<float> [(((long unsigned int)(((long
int)R2_N) + -0x00000000000000001)) + 1)]’


So... In order to make it more clear: For each time-step (e.g.
time={0,1,2,3,4...} seconds), maybe 10 or 20 sensors values is stored.
Because float is 4 bytes, let's assume that I want to read 10 sensors,
e.g. RTSensor_array = 10x4 = 40 bytes. For each consecutive data
record, I have some temporary variables, i.e. the first 4 moveBytes-
lines: unsigned int myuint_begin, float float_time, unsigned int
myuint_end, vector<float> RTSensor_array.

My question: What is the most efficient way of storing this float
array using C++ ?

Maybe later I want to access sensor number 4 (index 4 in the
RTSensor_array) as a function of the time which is saved in
vector<float> RTtime;

I hope the question is understandable and else I'll try to explain it
better... The vector issue is quite essential to C++ so I hope I get
some helpful answers/hints which I can learn a great deal from. Thanks
so far!

[toc] | [prev] | [next] | [standalone]

#10897

someone <newsboost@gmail.com> wrote in
news:d3681042-ab0a-4fa2-98d4-11ab3cc806c8@f16g2000yqk.googlegroups.com: 

> I ended up doing this (which I find is quite good, I stole some part
> of the idea from the internet):
> 
> //================================template <class T>
> void moveBytes(T *destination, const char *source, const unsigned int
> multiplier=1)
> {
>   //  cout << "sizeof(*destination) = " << dec << sizeof(*destination)
> << endl;
>   memcpy(destination, &source[memLocation],
>            (multiplier*sizeof(*destination)));

ok, this should work, assuming that the endiannesses match.

>   memLocation += (multiplier*sizeof(*destination)); // memLocation
>   global variable/counter! 
> }
> //================================
> This allows me to extract different data types from memory and do,
> e.g.:
> 
>   unsigned int R1_begin;
>   moveBytes(&R1_begin, memblock); // memLocation automatically
> increases for next time

And memLocation is a global or static variable? In the long term this 
will hurt you. In C++ if one wants to keep a state one creates a class 
for that.

> 
> Now, I have one last serious issue with my program before it works...
> //================================ RTSensor_array = new float[R2_N];
> // should be array of size R2_N! 
>  vector<float> RTSensorReading[R2_N]; // R2_N = 10 (this is the sensor
> array length)

This should be:

vector<float> RTSensorReading(R2_N);

However, notice that push_back() would then append the 11-th etc. 
elements here, probably you want to start with an empty vector instead:

vector<float> RTSensorReading;

hth
Paavo

[toc] | [prev] | [next] | [standalone]

#10899

Paavo Helde <myfirstname@osa.pri.ee> wrote in 
news:Xns9F80DED1C590Cmyfirstnameosapriee@216.196.109.131:

> someone <newsboost@gmail.com> wrote in
>>   memcpy(destination, &source[memLocation],
>>            (multiplier*sizeof(*destination)));
> 
> ok, this should work, assuming that the endiannesses match.

Oops, I should have stressed that also the data type sizes and 
representations must match. 

>> etc... And I think it works for all data types: floats, doubles,
>> uint32, int32, whatever...

Yes, uint32_t, int32_t are good with binary transfer; avoid types like long 
or int whose size depends on the implementation.

Cheers
Paavo

[toc] | [prev] | [next] | [standalone]

#10904

On 16 Okt., 20:54, Paavo Helde <myfirstn...@osa.pri.ee> wrote:
> someone <newsbo...@gmail.com> wrote innews:d3681042-ab0a-4fa2-98d4-11ab3cc806c8@f16g2000yqk.googlegroups.com:
>
> > I ended up doing this (which I find is quite good, I stole some part
> > of the idea from the internet):
>
> > //================================template <class T>
> > void moveBytes(T *destination, const char *source, const unsigned int
> > multiplier=1)
> > {
> >   //  cout << "sizeof(*destination) = " << dec << sizeof(*destination)
> > << endl;
> >   memcpy(destination, &source[memLocation],
> >            (multiplier*sizeof(*destination)));
>
> ok, this should work, assuming that the endiannesses match.

My attempts to verify the code has so far worked with the above
template :-)

> >   memLocation += (multiplier*sizeof(*destination)); // memLocation
> >   global variable/counter!
> > }
> > //================================
> > This allows me to extract different data types from memory and do,
> > e.g.:
>
> >   unsigned int R1_begin;
> >   moveBytes(&R1_begin, memblock); // memLocation automatically
> > increases for next time
>
> And memLocation is a global or static variable? In the long term this
> will hurt you. In C++ if one wants to keep a state one creates a class
> for that.

It is currently a global variable, declared in the top of my *.cpp
file (right below the #include's and just before any real code
begins).
I just finished the program and I think everything works (although it
is pretty ugly code). I'm currently transforming the code into a
class, because I totally agree with you (that this is a bit ugly)...


> > Now, I have one last serious issue with my program before it works...
> > //================================ RTSensor_array = new float[R2_N];
> > // should be array of size R2_N!
> >  vector<float> RTSensorReading[R2_N]; // R2_N = 10 (this is the sensor
> > array length)
>
> This should be:
>
> vector<float> RTSensorReading(R2_N);
>
> However, notice that push_back() would then append the 11-th etc.
> elements here, probably you want to start with an empty vector instead:
>
> vector<float> RTSensorReading;

Ah, thanks... Because I had to use my memcpy-template I ended up with
the below code (nothing else worked, by googling it seemed to me like
I could not push an array of floats into the vector so now I push a
vector into another vector instead and it seems to work):

//------------------
	  vector<float> RTSensorReading; // create empty vector of floats
	  for (int a=0; a<R2_N; a++)
		{
		  moveBytes(&float_read, memblock); // read
		  RTSensorReading.push_back( float_read ); // save
		}
	  if (RTSensorReading.size() != R2_N)
		cout << "... warning message" << endl;

      ....

	  RTSensor.push_back( RTSensorReading ); // save vector of floats
//------------------

Where the last line/variable is declared:

   vector< vector<float> > RTSensor;


Everything seems to work now! The binary reader program went through
15000 records without complaining about inconsistent record length and
that indicates that everything works... I've already began to rewrite
the code into a class and in a few days, I'll take the read input data
and plot it (I successfully managed to use mathgl to plot some data a
few days ago, so that's the end of the story for now) :-)

Thank you very much to everybody who helped!

If there are more suggestions/remarks, I'll still read them and maybe
reply :-)

[toc] | [prev] | [next] | [standalone]

#10908

On 10/16/11 6:12 PM, someone wrote:
> Thank you very much to everybody who helped!
>
> If there are more suggestions/remarks, I'll still read them and maybe
> reply :-)
>

One comment, looking at the processing you are doing, there appears to 
be no reason you needed to read the whole file into memory to begin 
with. For each step where you are getting n bytes from your memory 
buffer, you could just as easily read those n bytes from the file.

You could then encapsulate the file in a class, with members to extract 
the various fundamental types, localizing the details of any format 
conversions that might be needed, things like endianness and floating 
point formats.

[toc] | [prev] | [next] | [standalone]

#10910

On Oct 17, 2:14 am, Richard Damon <news.x.richardda...@xoxy.net>
wrote:
> On 10/16/11 6:12 PM, someone wrote:
>
> > Thank you very much to everybody who helped!
>
> > If there are more suggestions/remarks, I'll still read them and maybe
> > reply :-)
>
> One comment, looking at the processing you are doing, there appears to
> be no reason you needed to read the whole file into memory to begin
> with. For each step where you are getting n bytes from your memory
> buffer, you could just as easily read those n bytes from the file.

Yes, I found out that too late. There was an excellent piece of code I
could have stolen from google if I didn't read everything into memory
before processing the data :-)

> You could then encapsulate the file in a class, with members to extract
> the various fundamental types, localizing the details of any format
> conversions that might be needed, things like endianness and floating
> point formats.

Sounds like a good idea... I've made a structure instead of a class
(without endianness stuff, currently). I haven't made a specific class
for the various fundamental types yet, but maybe I should try that
whenever I get some time... I'm currently moving the class (structure)
out to separate *.cpp and *.h files so I can include it in other
source codes... I did it some years ago, but then I stopped
programming for quite some time and now I have to refresh my memory...
But I think I'll succeed with this - google helps me a lot :-)

Thanks. Within a few days, I'll plot the data and make sure it is like
I expect. If I get into new problems, maybe I should create a new
thread/post.

My main issue in this thread has been resolved now :-)

[toc] | [prev] | [next] | [standalone]

#10876 — Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?)

On 10/16/11 11:39 AM, Martin Jørgensen wrote:
> On Oct 15, 11:35 pm, Richard Damon<news.x.richardda...@xoxy.net>
> wrote:
>> On 10/15/11 3:00 PM, someone wrote:
> ==
>>> I'm stuck...
>>
>>> And what about big endian / little endian? I'm confused. Looking
>>> forward to hear from you!
>>
>> First, what is the first byte in the file? you are saying that you are
>> expecting it to be 72 but you are getting 49. Have you looked at the file..
>
> Good point. The first four bytes of the input file has the values:
> [048h 00h 00h 00h] (hexadecimal, I used a hex editor to see this). And
> 0x48H = 72 decimal... I don't even know if this is little endian or
> not, but what I find very strange is the number 49 then... That number
> makes no sense to me (why is it adding 1, if it's printing the number
> in hexadecimal? What's it thinking?)..?

If you are in a 32 bit little endian (x86) machine, this will work:

#include <iostream>

int main()
{
   char tmp[4] = {0x48,0,0,0};

   std::cout << *reinterpret_cast<int*>(tmp) << std::endl;

   return 0;
}

If not, you will have to build the integer value from the individual 
bytes.  For example if the data were written on a big endian machine:

   char be[4] = {0,0,0,0x48};

   int n = be[3] + (be[2]<<8) + (be[1]<<16) + (be[0]<<24);

   std::cout << n << std::endl;

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#10880

On 16 Okt., 01:33, Ian Collins <ian-n...@hotmail.com> wrote:
> On 10/16/11 11:39 AM, Martin Jørgensen wrote:
> If you are in a 32 bit little endian (x86) machine, this will work:
>
> #include <iostream>
>
> int main()
> {
>    char tmp[4] = {0x48,0,0,0};
>
>    std::cout << *reinterpret_cast<int*>(tmp) << std::endl;
>
>    return 0;
>
> }
>
> If not, you will have to build the integer value from the individual
> bytes.  For example if the data were written on a big endian machine:
>
>    char be[4] = {0,0,0,0x48};
>
>    int n = be[3] + (be[2]<<8) + (be[1]<<16) + (be[0]<<24);
>
>    std::cout << n << std::endl;


THANKS A LOT TO ALL WHO HELPED! :-)

I'll get back if I get problems with reading the rest of the file,
however I think (I hope) everything will work out now :-)

[toc] | [prev] | [next] | [standalone]

#10898 — Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?)

On 16/10/2011 00:33, Ian Collins wrote:
> On 10/16/11 11:39 AM, Martin Jørgensen wrote:
>> On Oct 15, 11:35 pm, Richard Damon<news.x.richardda...@xoxy.net>
>> wrote:
>>> On 10/15/11 3:00 PM, someone wrote:
>> ==
>>>> I'm stuck...
>>>
>>>> And what about big endian / little endian? I'm confused. Looking
>>>> forward to hear from you!
>>>
>>> First, what is the first byte in the file? you are saying that you are
>>> expecting it to be 72 but you are getting 49. Have you looked at the
>>> file..
>>
>> Good point. The first four bytes of the input file has the values:
>> [048h 00h 00h 00h] (hexadecimal, I used a hex editor to see this). And
>> 0x48H = 72 decimal... I don't even know if this is little endian or
>> not, but what I find very strange is the number 49 then... That number
>> makes no sense to me (why is it adding 1, if it's printing the number
>> in hexadecimal? What's it thinking?)..?
>
> If you are in a 32 bit little endian (x86) machine, this will work:
>
> #include <iostream>
>
> int main()
> {
> char tmp[4] = {0x48,0,0,0};
>
> std::cout << *reinterpret_cast<int*>(tmp) << std::endl;
>
> return 0;
> }

Formally isn't that invoking Undefined Behaviour?

/Leigh

[toc] | [prev] | [next] | [standalone]

#10900 — Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?)

On 16/10/2011 19:59, Leigh Johnston wrote:
> On 16/10/2011 00:33, Ian Collins wrote:
>> On 10/16/11 11:39 AM, Martin Jørgensen wrote:
>>> On Oct 15, 11:35 pm, Richard Damon<news.x.richardda...@xoxy.net>
>>> wrote:
>>>> On 10/15/11 3:00 PM, someone wrote:
>>> ==
>>>>> I'm stuck...
>>>>
>>>>> And what about big endian / little endian? I'm confused. Looking
>>>>> forward to hear from you!
>>>>
>>>> First, what is the first byte in the file? you are saying that you are
>>>> expecting it to be 72 but you are getting 49. Have you looked at the
>>>> file..
>>>
>>> Good point. The first four bytes of the input file has the values:
>>> [048h 00h 00h 00h] (hexadecimal, I used a hex editor to see this). And
>>> 0x48H = 72 decimal... I don't even know if this is little endian or
>>> not, but what I find very strange is the number 49 then... That number
>>> makes no sense to me (why is it adding 1, if it's printing the number
>>> in hexadecimal? What's it thinking?)..?
>>
>> If you are in a 32 bit little endian (x86) machine, this will work:
>>
>> #include <iostream>
>>
>> int main()
>> {
>> char tmp[4] = {0x48,0,0,0};
>>
>> std::cout << *reinterpret_cast<int*>(tmp) << std::endl;
>>
>> return 0;
>> }
>
> Formally isn't that invoking Undefined Behaviour?

3.10/10

"If a program attempts to access the stored value of an object through a 
glvalue of other than one of the
following types the behavior is undefined:52
— the dynamic type of the object,
— a cv-qualified version of the dynamic type of the object,
— a type similar (as defined in 4.4) to the dynamic type of the object,
— a type that is the signed or unsigned type corresponding to the 
dynamic type of the object,
— a type that is the signed or unsigned type corresponding to a 
cv-qualified version of the dynamic type
of the object,
— an aggregate or union type that includes one of the aforementioned 
types among its elements or nonstatic
data members (including, recursively, an element or non-static data 
member of a subaggregate
or contained union),
— a type that is a (possibly cv-qualified) base class type of the 
dynamic type of the object,
— a char or unsigned char type."

/Leigh

[toc] | [prev] | [next] | [standalone]

#10950

On Oct 16, 12:13 pm, Leigh Johnston <le...@i42.co.uk> wrote:
> On 16/10/2011 19:59, Leigh Johnston wrote:
>
>
>
>
>
>
>
>
>
> > On 16/10/2011 00:33, Ian Collins wrote:
> >> On 10/16/11 11:39 AM, Martin Jørgensen wrote:
> >>> On Oct 15, 11:35 pm, Richard Damon<news.x.richardda...@xoxy.net>
> >>> wrote:
> >>>> On 10/15/11 3:00 PM, someone wrote:
> >>> ==
> >>>>> I'm stuck...
>
> >>>>> And what about big endian / little endian? I'm confused. Looking
> >>>>> forward to hear from you!
>
> >>>> First, what is the first byte in the file? you are saying that you are
> >>>> expecting it to be 72 but you are getting 49. Have you looked at the
> >>>> file..
>
> >>> Good point. The first four bytes of the input file has the values:
> >>> [048h 00h 00h 00h] (hexadecimal, I used a hex editor to see this). And
> >>> 0x48H = 72 decimal... I don't even know if this is little endian or
> >>> not, but what I find very strange is the number 49 then... That number
> >>> makes no sense to me (why is it adding 1, if it's printing the number
> >>> in hexadecimal? What's it thinking?)..?
>
> >> If you are in a 32 bit little endian (x86) machine, this will work:
>
> >> #include <iostream>
>
> >> int main()
> >> {
> >> char tmp[4] = {0x48,0,0,0};
>
> >> std::cout << *reinterpret_cast<int*>(tmp) << std::endl;
>
> >> return 0;
> >> }
>
> > Formally isn't that invoking Undefined Behaviour?
>
> 3.10/10
>
> "If a program attempts to access the stored value of an object through a
> glvalue of other than one of the
> following types the behavior is undefined:52
> — the dynamic type of the object,
> — a cv-qualified version of the dynamic type of the object,
> — a type similar (as defined in 4.4) to the dynamic type of the object,
> — a type that is the signed or unsigned type corresponding to the
> dynamic type of the object,
> — a type that is the signed or unsigned type corresponding to a
> cv-qualified version of the dynamic type
> of the object,
> — an aggregate or union type that includes one of the aforementioned
> types among its elements or nonstatic
> data members (including, recursively, an element or non-static data
> member of a subaggregate
> or contained union),
> — a type that is a (possibly cv-qualified) base class type of the
> dynamic type of the object,
> — a char or unsigned char type."

Yep, I was about to mention this too. Let me reproduce the specific
code fragment:

> char tmp[4] = {0x48,0,0,0};
>
> std::cout << *reinterpret_cast<int*>(tmp) << std::endl;

You are reading a char object (or char array, whatever) through an int
lvalue. This is undefined behavior.

For example, under the default compiler options for gcc, this can blow
up in your face. In practice, gcc sometimes reorders or omits writes
and reads between improperly aliased objects, resulting in surprising
values.

In this case, I'd suggest simple bit shifting. Otherwise, if you /
really/ need to do this, then do your reads and writes through char*
or unsigned char*, or use memcpy, which all have a "get out of jail
free card" from the aliasing rules. Ex:

> char tmp[4] = {0x48,0,0,0};
> int x;
> static_assert(sizeof(x) == sizeof(tmp));
> memcpy(&x, tmp, sizeof(tmp);
> std::cout << x << std::endl;

Alternatively:

> char tmp[4] = {0x48,0,0,0};
> int x;
> static_assert(sizeof(x) == sizeof(tmp));
> for (int i=0; i<sizeof(tmp); ++i)
>   reinterpret_cast<char*>(&x)[i] = tmp[i];
> std::cout << x << std::endl;

[toc] | [prev] | [next] | [standalone]

#10951

On Oct 17, 6:17 pm, Joshua Maurice <joshuamaur...@gmail.com> wrote:
> On Oct 16, 12:13 pm, Leigh Johnston <le...@i42.co.uk> wrote:
>
> > On 16/10/2011 19:59, Leigh Johnston wrote:
>
> > > On 16/10/2011 00:33, Ian Collins wrote:
> > >> On 10/16/11 11:39 AM, Martin Jørgensen wrote:
> > >>> On Oct 15, 11:35 pm, Richard Damon<news.x.richardda...@xoxy.net>
> > >>> wrote:
> > >>>> On 10/15/11 3:00 PM, someone wrote:
> > >>> ==
> > >>>>> I'm stuck...
>
> > >>>>> And what about big endian / little endian? I'm confused. Looking
> > >>>>> forward to hear from you!
>
> > >>>> First, what is the first byte in the file? you are saying that you are
> > >>>> expecting it to be 72 but you are getting 49. Have you looked at the
> > >>>> file..
>
> > >>> Good point. The first four bytes of the input file has the values:
> > >>> [048h 00h 00h 00h] (hexadecimal, I used a hex editor to see this). And
> > >>> 0x48H = 72 decimal... I don't even know if this is little endian or
> > >>> not, but what I find very strange is the number 49 then... That number
> > >>> makes no sense to me (why is it adding 1, if it's printing the number
> > >>> in hexadecimal? What's it thinking?)..?
>
> > >> If you are in a 32 bit little endian (x86) machine, this will work:
>
> > >> #include <iostream>
>
> > >> int main()
> > >> {
> > >> char tmp[4] = {0x48,0,0,0};
>
> > >> std::cout << *reinterpret_cast<int*>(tmp) << std::endl;
>
> > >> return 0;
> > >> }
>
> > > Formally isn't that invoking Undefined Behaviour?
>
> > 3.10/10
>
> > "If a program attempts to access the stored value of an object through a
> > glvalue of other than one of the
> > following types the behavior is undefined:52
> > — the dynamic type of the object,
> > — a cv-qualified version of the dynamic type of the object,
> > — a type similar (as defined in 4.4) to the dynamic type of the object,
> > — a type that is the signed or unsigned type corresponding to the
> > dynamic type of the object,
> > — a type that is the signed or unsigned type corresponding to a
> > cv-qualified version of the dynamic type
> > of the object,
> > — an aggregate or union type that includes one of the aforementioned
> > types among its elements or nonstatic
> > data members (including, recursively, an element or non-static data
> > member of a subaggregate
> > or contained union),
> > — a type that is a (possibly cv-qualified) base class type of the
> > dynamic type of the object,
> > — a char or unsigned char type."
>
> Yep, I was about to mention this too. Let me reproduce the specific
> code fragment:
>
> > char tmp[4] = {0x48,0,0,0};
>
> > std::cout << *reinterpret_cast<int*>(tmp) << std::endl;
>
> You are reading a char object (or char array, whatever) through an int
> lvalue. This is undefined behavior.
>
> For example, under the default compiler options for gcc, this can blow
> up in your face. In practice, gcc sometimes reorders or omits writes
> and reads between improperly aliased objects, resulting in surprising
> values.
>
> In this case, I'd suggest simple bit shifting. Otherwise, if you /
> really/ need to do this, then do your reads and writes through char*
> or unsigned char*, or use memcpy, which all have a "get out of jail
> free card" from the aliasing rules. Ex:
>
> > char tmp[4] = {0x48,0,0,0};
> > int x;
> > static_assert(sizeof(x) == sizeof(tmp));
> > memcpy(&x, tmp, sizeof(tmp);
> > std::cout << x << std::endl;
>
> Alternatively:
>
> > char tmp[4] = {0x48,0,0,0};
> > int x;
> > static_assert(sizeof(x) == sizeof(tmp));
> > for (int i=0; i<sizeof(tmp); ++i)
> >   reinterpret_cast<char*>(&x)[i] = tmp[i];
> > std::cout << x << std::endl;

Oh, of course this can still crash and burn if you write a trap
representation. char and unsigned char are guaranteed to not have trap
representations, but int is not.

[toc] | [prev] | [next] | [standalone]

#10881

someone wrote:
> Hi,
>
> Here's my program:
>
> ==========================
> #include<fstream>
> #include<iostream>
> #include<string>
> #include<sstream>
> #include<stdio.h>
> //#include<boost/cstdint.hpp>  //<-- I know this is not the group to
> ask in, maybe unnecessary
>
> using namespace std;
>
>
> int main( int argc, const char* argv[] )
> {
>    ifstream::pos_type fsize;
>    char * memblock;
>    ifstream infile;
>
>    cout<<  "Program to read SENSOR file (for FLEX5). Build on: "<<
> 	__DATE__<<  " / "<<  __TIME__<<  endl;
>
>    infile.open("DELETEME.res", ios::in|ios::binary);
>
>    if (infile.is_open())
> 	{
> 	  infile.seekg (0, ios::end); // go to end of file
> 	  fsize = infile.tellg();
> 	  infile.seekg (0, ios::beg); // go to beginning
>
> 	  cout<<  "Size of file is: "<<  fsize<<  " bytes."<<  endl;
> 	  memblock = new char [fsize];
>
> 	  infile.read (memblock, fsize);
> 	  infile.close();
> 	  //cout<<  "the complete file content is in memory"<<  endl;
> 	}
>    else
> 	cout<<  "Unable to open file"<<  endl;
>
>    cout<<  endl; // delimiter before program outputs read data
>
>    int R1_begin;
>    R1_begin = static_cast<typeof R1_begin>  (memblock[0]);//, 1,
> 'uint32',endianNess);
>    cout<<  "Value = "<<  R1_begin<<  endl;
>
>   delete[] memblock;
>    return 0;
> }
> ==========================
>
> Problem:
>
> R1_begin is the first value read from a binary file. But it's wrong!
> It should be 72 but the program prints 49 out to the screen. So I
> cannot continue before I get this right. The first few bytes I'm going
> to read is:
>
> R1begin = 1 byte times sizeof 'uint32' = 4 bytes
> I1_I6 = 6 bytes times sizeof 'uint32' = 6x4 = 24 bytes
> TEXT = 40 ASCII characters of type 'char' = 40 bytes
> R1end = 1 byte times sizeof 'uint32' = 4 bytes
>
> Total bytes read = 4+24+40+4 = 72 bytes. That's why I must read 72
> bytes from the very first byte in the binary file... And I cannot
> solve this! :-(
>
> Please help (and please provide hints for reading the next few bytes
> and how I should convert from data in memory into different data
> types). Later I also need to convert some bytes from memblock[ ] into
> floats and doubles...
>
> I'm stuck...
>
> And what about big endian / little endian? I'm confused. Looking
> forward to hear from you!

I think previous responders answered most of your questions. I just noticed few 
other things that may go wrong with your code sooner or later:

1. No check the return value of infile.read().
2. memblock was not initiazlied so delete[] can coredump (I would just use 
std::vector or unique_ptr with array deleter if implemented and I cared about 
performance).

-Pavel

[toc] | [prev] | [next] | [standalone]

#10885

On 16 Okt., 01:56, Pavel
<pauldontspamt...@removeyourself.dontspam.yahoo> wrote:
> someone wrote:
> I think previous responders answered most of your questions. I just noticed few
> other things that may go wrong with your code sooner or later:
>
> 1. No check the return value of infile.read().

Ah, thanks a lot. I changed it into this:

 if (infile.is_open())
	{
	  infile.seekg (0, ios::end); // go to end of file
	  fsize = infile.tellg();
	  infile.seekg (0, ios::beg); // go to beginning

	  cout << "Size of file is: " << fsize << " bytes." << endl;
	  memblock = new char[fsize];

	  infile.read (memblock, fsize);
	  if (!infile)
		{
		  cout << "Some error occured while reading the file, after: " <<
			infile.gcount() << " bytes..." << endl;
		  return(1);
		}
	  infile.close();
	  //cout << "the complete file content is in memory" << endl;
	}
  else
	cout << "Unable to open file" << endl;


> 2. memblock was not initiazlied so delete[] can coredump (I would just use
> std::vector or unique_ptr with array deleter if implemented and I cared about
> performance).

Huh? I don't understand this. Even though it was not initialized, it
has been allocated and therefore I try to clean up in the end. With
the command:

	  infile.read (memblock, fsize);

I guess that I'm initializing it... Or what do you suggest?
I can't see why it should coredump, so please advice and thanks :-)

[toc] | [prev] | [next] | [standalone]

#10886

someone <newsboost@gmail.com> wrote in
news:76015696-f1b7-4ad3-9875-70a8c8ba570e@v7g2000yqf.googlegroups.com: 

> On 16 Okt., 01:56, Pavel
> <pauldontspamt...@removeyourself.dontspam.yahoo> wrote:
[...]
>> 2. memblock was not initiazlied so delete[] can coredump (I would
>> just use std::vector or unique_ptr with array deleter if implemented
>> and I cared about performance).
> 
> Huh? I don't understand this. Even though it was not initialized, it
> has been allocated and therefore I try to clean up in the end. With
> the command:
> 
>        infile.read (memblock, fsize);
> 
> I guess that I'm initializing it... Or what do you suggest?
> I can't see why it should coredump, so please advice and thanks :-)

What Pavel meant was thet you have:

  char * memblock;
  // ...
  if (infile.is_open())
  {
    // ...
    memblock = new char [fsize];
    // ...
  }
  // ...
  delete[] memblock;
  return 0;
}

infile is a pointer, which is left uninitialized in the point where it is 
defined. It is initialized only inside the if branch, but it is not 
guaranteed that you enter the if branch always. If you don't, delete[] 
gets a random value which is UB which might or might not crash your 
application, or crash only when installed on the customer machine, for 
example.

Suggesting to use std::vector<char> instead of char* pointer and array 
new, this avoids a potential for such kind of bugs. There are very few 
situations which warrant the array new construct, most probably you will 
never encounter one.

hth
Paavo

[toc] | [prev] | [next] | [standalone]

#10892

On 16 Okt., 09:48, Paavo Helde <myfirstn...@osa.pri.ee> wrote:
> someone <newsbo...@gmail.com> wrote innews:76015696-f1b7-4ad3-9875-70a8c8ba570e@v7g2000yqf.googlegroups.com:
> What Pavel meant was thet you have:
>
>   char * memblock;
>   // ...
>   if (infile.is_open())
>   {
>     // ...
>     memblock = new char [fsize];
>     // ...
>   }
>   // ...
>   delete[] memblock;
>   return 0;
>
> }
>
> infile is a pointer, which is left uninitialized in the point where it is
> defined. It is initialized only inside the if branch, but it is not
> guaranteed that you enter the if branch always. If you don't, delete[]
> gets a random value which is UB which might or might not crash your
> application, or crash only when installed on the customer machine, for
> example.

Aah, thanks... You're both right... For now I just made a flag so in
the end I write:

 if (delete_memblock==true) // true if allocated
	delete[] memblock;

> Suggesting to use std::vector<char> instead of char* pointer and array
> new, this avoids a potential for such kind of bugs. There are very few
> situations which warrant the array new construct, most probably you will
> never encounter one.

I'm actually a bit more confident in ordinary C than in C++ but I want
to improve my object-oriented skills. I think I understand the idea of
using std::vector<char> but I don't understand how to use "array
new" ?

In any case, for now I think the delete_memblock==true flag should
work. I have a more important issue about vectors and will write a new
post explaining it within a few minutes... Once this issue is done,
the program version 1.0 would work, I think :-)

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | comp.lang.c++

csiph-web