Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #10864 > unrolled thread

reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?)

Back to article view | Back to comp.lang.c++

Contents

  reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-15 12:00 -0700
    Re: reading binary file into memory. Converting from char to uint32,   float, double, ASCII strings etc (static_cast< > ?) Ian Collins <ian-news@hotmail.com> - 2011-10-16 09:29 +1300
      Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-15 14:17 -0700
        Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Ebenezer <woodbrian77@gmail.com> - 2011-10-15 14:41 -0700
        Re: reading binary file into memory. Converting from char to uint32,   float, double, ASCII strings etc (static_cast< > ?) Ian Collins <ian-news@hotmail.com> - 2011-10-16 10:45 +1300
          Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Martin Jørgensen <mjoerg@gmail.com> - 2011-10-15 15:43 -0700
            Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-15 21:37 -0400
              Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-15 23:23 -0700
                Re: reading binary file into memory. Converting from char to uint32,   float, double, ASCII strings etc (static_cast< > ?) Ian Collins <ian-news@hotmail.com> - 2011-10-16 20:53 +1300
                  Re: reading binary file into memory. Converting from char to uint32,   float, double, ASCII strings etc (static_cast< > ?) Paavo Helde <myfirstname@osa.pri.ee> - 2011-10-16 03:04 -0500
                    Re: reading binary file into memory. Converting from char to uint32,   float, double, ASCII strings etc (static_cast< > ?) Ian Collins <ian-news@hotmail.com> - 2011-10-16 21:52 +1300
              Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Jorgen Grahn <grahn+nntp@snipabacken.se> - 2011-10-16 18:21 +0000
                Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-16 16:40 -0400
                  Re: reading binary file into memory. Converting from char to uint32,   float, double, ASCII strings etc (static_cast< > ?) Ian Collins <ian-news@hotmail.com> - 2011-10-17 10:00 +1300
                    Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-16 15:19 -0700
                      Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-16 20:03 -0400
                        Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-16 17:49 -0700
                          Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-16 21:29 -0400
                  Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Jorgen Grahn <grahn+nntp@snipabacken.se> - 2011-10-17 09:10 +0000
    Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-15 17:35 -0400
      Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Martin Jørgensen <mjoerg@gmail.com> - 2011-10-15 15:39 -0700
        Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-15 16:12 -0700
          Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-15 16:32 -0700
            Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-15 19:41 -0400
              Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-16 11:33 -0700
                Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Paavo Helde <myfirstname@osa.pri.ee> - 2011-10-16 13:54 -0500
                  Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Paavo Helde <myfirstname@osa.pri.ee> - 2011-10-16 14:01 -0500
                  Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-16 15:12 -0700
                    Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Richard Damon <news.x.richarddamon@xoxy.net> - 2011-10-16 20:14 -0400
                      Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-16 17:56 -0700
        Re: reading binary file into memory. Converting from char to uint32,   float, double, ASCII strings etc (static_cast< > ?) Ian Collins <ian-news@hotmail.com> - 2011-10-16 12:33 +1300
          Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-15 16:37 -0700
          Re: reading binary file into memory. Converting from char to uint32,   float, double, ASCII strings etc (static_cast< > ?) Leigh Johnston <leigh@i42.co.uk> - 2011-10-16 19:59 +0100
            Re: reading binary file into memory. Converting from char to uint32,   float, double, ASCII strings etc (static_cast< > ?) Leigh Johnston <leigh@i42.co.uk> - 2011-10-16 20:13 +0100
              Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Joshua Maurice <joshuamaurice@gmail.com> - 2011-10-17 18:17 -0700
                Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Joshua Maurice <joshuamaurice@gmail.com> - 2011-10-17 18:32 -0700
    Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Pavel <pauldontspamtolk@removeyourself.dontspam.yahoo> - 2011-10-15 19:56 -0400
      Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-15 23:27 -0700
        Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Paavo Helde <myfirstname@osa.pri.ee> - 2011-10-16 02:48 -0500
          Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-16 10:57 -0700
            Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Paavo Helde <myfirstname@osa.pri.ee> - 2011-10-16 13:18 -0500
              Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-16 12:52 -0700
    Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-16 10:58 -0700
      Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Paavo Helde <myfirstname@osa.pri.ee> - 2011-10-16 13:37 -0500
        Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) someone <newsboost@gmail.com> - 2011-10-16 14:37 -0700
          Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?) Paavo Helde <myfirstname@osa.pri.ee> - 2011-10-17 01:21 -0500

Page 1 of 3  [1] 2 3  Next page →

#10864 — reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?)

Hi,

Here's my program:

==========================
#include <fstream>
#include <iostream>
#include <string>
#include <sstream>
#include <stdio.h>
//#include <boost/cstdint.hpp> // <-- I know this is not the group to
ask in, maybe unnecessary

using namespace std;


int main( int argc, const char* argv[] )
{
  ifstream::pos_type fsize;
  char * memblock;
  ifstream infile;

  cout << "Program to read SENSOR file (for FLEX5). Build on: " <<
	__DATE__ << " / " << __TIME__ << endl;

  infile.open("DELETEME.res", ios::in|ios::binary);

  if (infile.is_open())
	{
	  infile.seekg (0, ios::end); // go to end of file
	  fsize = infile.tellg();
	  infile.seekg (0, ios::beg); // go to beginning

	  cout << "Size of file is: " << fsize << " bytes." << endl;
	  memblock = new char [fsize];

	  infile.read (memblock, fsize);
	  infile.close();
	  //cout << "the complete file content is in memory" << endl;
	}
  else
	cout << "Unable to open file" << endl;

  cout << endl; // delimiter before program outputs read data

  int R1_begin;
  R1_begin = static_cast<typeof R1_begin> (memblock[0]);//, 1,
'uint32',endianNess);
  cout << "Value = " << R1_begin << endl;

 delete[] memblock;
  return 0;
}
==========================

Problem:

R1_begin is the first value read from a binary file. But it's wrong!
It should be 72 but the program prints 49 out to the screen. So I
cannot continue before I get this right. The first few bytes I'm going
to read is:

R1begin = 1 byte times sizeof 'uint32' = 4 bytes
I1_I6 = 6 bytes times sizeof 'uint32' = 6x4 = 24 bytes
TEXT = 40 ASCII characters of type 'char' = 40 bytes
R1end = 1 byte times sizeof 'uint32' = 4 bytes

Total bytes read = 4+24+40+4 = 72 bytes. That's why I must read 72
bytes from the very first byte in the binary file... And I cannot
solve this! :-(

Please help (and please provide hints for reading the next few bytes
and how I should convert from data in memory into different data
types). Later I also need to convert some bytes from memblock[ ] into
floats and doubles...

I'm stuck...

And what about big endian / little endian? I'm confused. Looking
forward to hear from you!

[toc] | [next] | [standalone]

#10865 — Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?)

On 10/16/11 08:00 AM, someone wrote:
> Hi,
>
> Here's my program:
>
> ==========================
> #include<fstream>
> #include<iostream>
> #include<string>
> #include<sstream>
> #include<stdio.h>
> //#include<boost/cstdint.hpp>  //<-- I know this is not the group to
> ask in, maybe unnecessary
>
> using namespace std;
>
>
> int main( int argc, const char* argv[] )
> {
>    ifstream::pos_type fsize;
>    char * memblock;
>    ifstream infile;
>
>    cout<<  "Program to read SENSOR file (for FLEX5). Build on: "<<
> 	__DATE__<<  " / "<<  __TIME__<<  endl;
>
>    infile.open("DELETEME.res", ios::in|ios::binary);
>
>    if (infile.is_open())
> 	{
> 	  infile.seekg (0, ios::end); // go to end of file
> 	  fsize = infile.tellg();
> 	  infile.seekg (0, ios::beg); // go to beginning
>
> 	  cout<<  "Size of file is: "<<  fsize<<  " bytes."<<  endl;
> 	  memblock = new char [fsize];
>
> 	  infile.read (memblock, fsize);
> 	  infile.close();
> 	  //cout<<  "the complete file content is in memory"<<  endl;
> 	}
>    else
> 	cout<<  "Unable to open file"<<  endl;
>
>    cout<<  endl; // delimiter before program outputs read data
>
>    int R1_begin;
>    R1_begin = static_cast<typeof R1_begin>  (memblock[0]);//, 1,
> 'uint32',endianNess);

You can't just cast an array of char to an int and expect the right 
result (unless the data is written in the same byte order as your machine).

The most convenient way to convert the byte order is to use the network 
ordering functions (if your platform has them).  See ntohl.

<snip>

> And what about big endian / little endian? I'm confused. Looking
> forward to hear from you!

Google "endian"

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#10866

On 15 Okt., 22:29, Ian Collins <ian-n...@hotmail.com> wrote:
> On 10/16/11 08:00 AM, someone wrote:

> >    int R1_begin;
> >    R1_begin = static_cast<typeof R1_begin>  (memblock[0]);//, 1,
> > 'uint32',endianNess);
>
> You can't just cast an array of char to an int and expect the right
> result (unless the data is written in the same byte order as your machine).

So how should I do it?
[to begin with, we can just assume that the data is written in the
same byte order and see if that works...]


> The most convenient way to convert the byte order is to use the network
> ordering functions (if your platform has them).  See ntohl.

Sorry, I have absolutely no idea about any network functions...
I was (pretty) sure that somebody had some code they could show here?
This "ntohl" looks a bit confusing to me - besides that, it's not
standard C++.

> <snip>
>
> > And what about big endian / little endian? I'm confused. Looking
> > forward to hear from you!
>
> Google "endian"

I did it. But what I meant is, what are your thoughts about it, when
reading a binary file like shown in my code?

[toc] | [prev] | [next] | [standalone]

#10868

On Oct 15, 4:17 pm, someone <newsbo...@gmail.com> wrote:
> On 15 Okt., 22:29, Ian Collins <ian-n...@hotmail.com> wrote:
>
> > On 10/16/11 08:00 AM, someone wrote:
> > >    int R1_begin;
> > >    R1_begin = static_cast<typeof R1_begin>  (memblock[0]);//, 1,
> > > 'uint32',endianNess);
>
> > You can't just cast an array of char to an int and expect the right
> > result (unless the data is written in the same byte order as your machine).
>
> So how should I do it?
> [to begin with, we can just assume that the data is written in the
> same byte order and see if that works...]
>
> > The most convenient way to convert the byte order is to use the network
> > ordering functions (if your platform has them).  See ntohl.
>
> Sorry, I have absolutely no idea about any network functions...
> I was (pretty) sure that somebody had some code they could show here?
> This "ntohl" looks a bit confusing to me - besides that, it's not
> standard C++.
>

I think these files may be what you're looking for:
http://webEbenezer.net/misc/Formatting.hh
http://webEbenezer.net/misc/ReceiveBuffer.hh

They are part of this archive --
http://webEbenezer.net/misc/direct.tar.bz2 .

Brian Wood
Ebenezer Enterprises
www.webEbenezer.net

[toc] | [prev] | [next] | [standalone]

#10869 — Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?)

On 10/16/11 10:17 AM, someone wrote:
> On 15 Okt., 22:29, Ian Collins<ian-n...@hotmail.com>  wrote:
>> On 10/16/11 08:00 AM, someone wrote:
>
>>>     int R1_begin;
>>>     R1_begin = static_cast<typeof R1_begin>    (memblock[0]);//, 1,
>>> 'uint32',endianNess);
>>
>> You can't just cast an array of char to an int and expect the right
>> result (unless the data is written in the same byte order as your machine).
>
> So how should I do it?
> [to begin with, we can just assume that the data is written in the
> same byte order and see if that works...]
>
>> The most convenient way to convert the byte order is to use the network
>> ordering functions (if your platform has them).  See ntohl.
>
> Sorry, I have absolutely no idea about any network functions...
> I was (pretty) sure that somebody had some code they could show here?
> This "ntohl" looks a bit confusing to me - besides that, it's not
> standard C++.

They are typically macros, just look them up if you want to see how they 
work.

>>> And what about big endian / little endian? I'm confused. Looking
>>> forward to hear from you!
>>
>> Google "endian"
>
> I did it. But what I meant is, what are your thoughts about it, when
> reading a binary file like shown in my code?

I use the network byte order functions, that's what they are there for.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#10879

On Oct 15, 11:45 pm, Ian Collins <ian-n...@hotmail.com> wrote:
> On 10/16/11 10:17 AM, someone wrote:

> >> The most convenient way to convert the byte order is to use the network
> >> ordering functions (if your platform has them).  See ntohl.
>
> > Sorry, I have absolutely no idea about any network functions...
> > I was (pretty) sure that somebody had some code they could show here?
> > This "ntohl" looks a bit confusing to me - besides that, it's not
> > standard C++.
>
> They are typically macros, just look them up if you want to see how they
> work.

Ok.

> >>> And what about big endian / little endian? I'm confused. Looking
> >>> forward to hear from you!
>
> >> Google "endian"
>
> > I did it. But what I meant is, what are your thoughts about it, when
> > reading a binary file like shown in my code?
>
> I use the network byte order functions, that's what they are there for.

Hmm... Based on my output to Richard - do I then have an endian
problem? Sorry I'm a bit too unexperienced to fully understand this
yet. A few replies and I think I'll understand it..

Thanks a lot for your (you, Richard and Ebenezer) help so far! :-)

[toc] | [prev] | [next] | [standalone]

#10882

On 10/15/11 6:43 PM, Martin Jørgensen wrote:
> On Oct 15, 11:45 pm, Ian Collins<ian-n...@hotmail.com>  wrote:
>> On 10/16/11 10:17 AM, someone wrote:
>
>>>> The most convenient way to convert the byte order is to use the network
>>>> ordering functions (if your platform has them).  See ntohl.
>>
>>> Sorry, I have absolutely no idea about any network functions...
>>> I was (pretty) sure that somebody had some code they could show here?
>>> This "ntohl" looks a bit confusing to me - besides that, it's not
>>> standard C++.
>>
>> They are typically macros, just look them up if you want to see how they
>> work.
>
> Ok.
>
>>>>> And what about big endian / little endian? I'm confused. Looking
>>>>> forward to hear from you!
>>
>>>> Google "endian"
>>
>>> I did it. But what I meant is, what are your thoughts about it, when
>>> reading a binary file like shown in my code?
>>
>> I use the network byte order functions, that's what they are there for.
>
> Hmm... Based on my output to Richard - do I then have an endian
> problem? Sorry I'm a bit too unexperienced to fully understand this
> yet. A few replies and I think I'll understand it..
>
> Thanks a lot for your (you, Richard and Ebenezer) help so far! :-)

Your data in the file is little endian. You have a problem if your 
processor is not little endian.

[toc] | [prev] | [next] | [standalone]

#10884

On 16 Okt., 03:37, Richard Damon <news.x.richardda...@xoxy.net> wrote:
> On 10/15/11 6:43 PM, Martin Jørgensen wrote:
> > Hmm... Based on my output to Richard - do I then have an endian
> > problem? Sorry I'm a bit too unexperienced to fully understand this
> > yet. A few replies and I think I'll understand it..
>
> > Thanks a lot for your (you, Richard and Ebenezer) help so far! :-)
>
> Your data in the file is little endian. You have a problem if your
> processor is not little endian.

Yep... Currently I think there's no problem (based on the first
unsigned integer, which is 4 bytes so now I don't want to learn more
than I can absorb) :-)

[toc] | [prev] | [next] | [standalone]

#10887 — Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?)

On 10/16/11 07:23 PM, someone wrote:
> On 16 Okt., 03:37, Richard Damon<news.x.richardda...@xoxy.net>  wrote:
>> On 10/15/11 6:43 PM, Martin Jørgensen wrote:
>>> Hmm... Based on my output to Richard - do I then have an endian
>>> problem? Sorry I'm a bit too unexperienced to fully understand this
>>> yet. A few replies and I think I'll understand it..
>>
>>> Thanks a lot for your (you, Richard and Ebenezer) help so far! :-)
>>
>> Your data in the file is little endian. You have a problem if your
>> processor is not little endian.
>
> Yep... Currently I think there's no problem (based on the first
> unsigned integer, which is 4 bytes so now I don't want to learn more
> than I can absorb) :-)

If you are happy that the file's number representations match your 
machine's and portability isn't an issue, why don't you just stream the 
values in directly rather than buggering about with a buffer?

int R1_begin;

infile >> R1_begin;

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#10888 — Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?)

Ian Collins <ian-news@hotmail.com> wrote in news:9fvgseFpouU1
@mid.individual.net:
> 
> If you are happy that the file's number representations match your 
> machine's and portability isn't an issue, why don't you just stream the 
> values in directly rather than buggering about with a buffer?
> 
> int R1_begin;
> 
> infile >> R1_begin;

Maybe because that the file is binary, but the stream >> operator assumes 
it is formatted ASCII text? 

Anyway, C++ streams are mostly useful for formatted input-output, such like 
composing error messages. For dealing with binary files I have not found 
any use for them, the old C fopen()/fread()/fwrite() are easier to use and 
understand as they don't have so much formatting features to ignore or work 
around.

Cheers
Paavo

[toc] | [prev] | [next] | [standalone]

#10889 — Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?)

On 10/16/11 09:04 PM, Paavo Helde wrote:
> Ian Collins<ian-news@hotmail.com>  wrote in news:9fvgseFpouU1
> @mid.individual.net:
>>
>> If you are happy that the file's number representations match your
>> machine's and portability isn't an issue, why don't you just stream the
>> values in directly rather than buggering about with a buffer?
>>
>> int R1_begin;
>>
>> infile>>  R1_begin;
>
> Maybe because that the file is binary, but the stream>>  operator assumes
> it is formatted ASCII text?

Oops...

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#10894

On Sun, 2011-10-16, Richard Damon wrote:
> On 10/15/11 6:43 PM, Martin Jørgensen wrote:
...
>> Hmm... Based on my output to Richard - do I then have an endian
>> problem? Sorry I'm a bit too unexperienced to fully understand this
>> yet. A few replies and I think I'll understand it..

> Your data in the file is little endian. You have a problem if your 
> processor is not little endian.

The usual interpretation is that this means he *does* have a problem.
Good code is supposed to have the same behavior no matter what
endianness the processor happens to have.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#10902

On 10/16/11 2:21 PM, Jorgen Grahn wrote:
> On Sun, 2011-10-16, Richard Damon wrote:
>> On 10/15/11 6:43 PM, Martin Jørgensen wrote:
> ...
>>> Hmm... Based on my output to Richard - do I then have an endian
>>> problem? Sorry I'm a bit too unexperienced to fully understand this
>>> yet. A few replies and I think I'll understand it..
>
>> Your data in the file is little endian. You have a problem if your
>> processor is not little endian.
>
> The usual interpretation is that this means he *does* have a problem.
> Good code is supposed to have the same behavior no matter what
> endianness the processor happens to have.
>
> /Jorgen
>

Only so far as he need to confirm that his machine is little endian. 
Good code runs reliably on the machines it is targeted, and minimizes 
the possible issues in moving it to other machines it might possible by 
run on.

If the program is know to only need to be run on a limited set of 
machines (like it is a windows program), than assuming properties common 
to the windows platform isn't totally bad. Later version of windows are 
very unlikely to switch endianness without other major issues coming up.

Him move to having a template function move a given type of data is an 
improvement, as it localizes the assumption. I would not at this point 
rewrite those functions to be endian neutral.

[toc] | [prev] | [next] | [standalone]

#10903 — Re: reading binary file into memory. Converting from char to uint32, float, double, ASCII strings etc (static_cast< > ?)

On 10/17/11 09:40 AM, Richard Damon wrote:
> On 10/16/11 2:21 PM, Jorgen Grahn wrote:
>> On Sun, 2011-10-16, Richard Damon wrote:
>>> On 10/15/11 6:43 PM, Martin Jørgensen wrote:
>> ...
>>>> Hmm... Based on my output to Richard - do I then have an endian
>>>> problem? Sorry I'm a bit too unexperienced to fully understand this
>>>> yet. A few replies and I think I'll understand it..
>>
>>> Your data in the file is little endian. You have a problem if your
>>> processor is not little endian.
>>
>> The usual interpretation is that this means he *does* have a problem.
>> Good code is supposed to have the same behavior no matter what
>> endianness the processor happens to have.
>
> Only so far as he need to confirm that his machine is little endian.
> Good code runs reliably on the machines it is targeted, and minimizes
> the possible issues in moving it to other machines it might possible by
> run on.
>
> If the program is know to only need to be run on a limited set of
> machines (like it is a windows program), than assuming properties common
> to the windows platform isn't totally bad. Later version of windows are
> very unlikely to switch endianness without other major issues coming up.

That assumption caught out a lot of SunOS programmers who thought their 
code would only run on 68K or Sparc.  Then Sun started producing x86 
machines...

> Him move to having a template function move a given type of data is an
> improvement, as it localizes the assumption. I would not at this point
> rewrite those functions to be endian neutral.

Fair enough.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#10905

On 16 Okt., 23:00, Ian Collins <ian-n...@hotmail.com> wrote:
> On 10/17/11 09:40 AM, Richard Damon wrote:
>
>
>
>
>
>
>
>
>
> > On 10/16/11 2:21 PM, Jorgen Grahn wrote:
> >> On Sun, 2011-10-16, Richard Damon wrote:
> >>> On 10/15/11 6:43 PM, Martin Jørgensen wrote:
> >> ...
> >>>> Hmm... Based on my output to Richard - do I then have an endian
> >>>> problem? Sorry I'm a bit too unexperienced to fully understand this
> >>>> yet. A few replies and I think I'll understand it..
>
> >>> Your data in the file is little endian. You have a problem if your
> >>> processor is not little endian.
>
> >> The usual interpretation is that this means he *does* have a problem.
> >> Good code is supposed to have the same behavior no matter what
> >> endianness the processor happens to have.
>
> > Only so far as he need to confirm that his machine is little endian.
> > Good code runs reliably on the machines it is targeted, and minimizes
> > the possible issues in moving it to other machines it might possible by
> > run on.
>
> > If the program is know to only need to be run on a limited set of
> > machines (like it is a windows program), than assuming properties common
> > to the windows platform isn't totally bad. Later version of windows are
> > very unlikely to switch endianness without other major issues coming up.
>
> That assumption caught out a lot of SunOS programmers who thought their
> code would only run on 68K or Sparc.  Then Sun started producing x86
> machines...
>
> > Him move to having a template function move a given type of data is an
> > improvement, as it localizes the assumption. I would not at this point
> > rewrite those functions to be endian neutral.
>
> Fair enough.


But ofcourse, *if* it's easy to change the template:
//------------------
template <class T>
void moveBytes(T *destination, const char *source, const unsigned int
multiplier=1)
{
  //  cout << "sizeof(*destination) = " << dec << sizeof(*destination)
<< endl;
  memcpy(destination, &source[memLocation],
		 (multiplier*sizeof(*destination)));
  memLocation += (multiplier*sizeof(*destination));
}
//------------------

to be endian neutral, then I would be interested in seeing/hearing
more about that.

However, I don't think I'll need it for this project (it would only be
good to know, for my own interests sake)... Most Windows and Linux
pc's are little-endian, I guess? (I'm on ubuntu linux and I think the
binary file I'm reading came from a windows pc)..

[toc] | [prev] | [next] | [standalone]

#10907

On 10/16/11 6:19 PM, someone wrote:
> On 16 Okt., 23:00, Ian Collins<ian-n...@hotmail.com>  wrote:
>> On 10/17/11 09:40 AM, Richard Damon wrote:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>> On 10/16/11 2:21 PM, Jorgen Grahn wrote:
>>>> On Sun, 2011-10-16, Richard Damon wrote:
>>>>> On 10/15/11 6:43 PM, Martin Jørgensen wrote:
>>>> ...
>>>>>> Hmm... Based on my output to Richard - do I then have an endian
>>>>>> problem? Sorry I'm a bit too unexperienced to fully understand this
>>>>>> yet. A few replies and I think I'll understand it..
>>
>>>>> Your data in the file is little endian. You have a problem if your
>>>>> processor is not little endian.
>>
>>>> The usual interpretation is that this means he *does* have a problem.
>>>> Good code is supposed to have the same behavior no matter what
>>>> endianness the processor happens to have.
>>
>>> Only so far as he need to confirm that his machine is little endian.
>>> Good code runs reliably on the machines it is targeted, and minimizes
>>> the possible issues in moving it to other machines it might possible by
>>> run on.
>>
>>> If the program is know to only need to be run on a limited set of
>>> machines (like it is a windows program), than assuming properties common
>>> to the windows platform isn't totally bad. Later version of windows are
>>> very unlikely to switch endianness without other major issues coming up.
>>
>> That assumption caught out a lot of SunOS programmers who thought their
>> code would only run on 68K or Sparc.  Then Sun started producing x86
>> machines...
>>
>>> Him move to having a template function move a given type of data is an
>>> improvement, as it localizes the assumption. I would not at this point
>>> rewrite those functions to be endian neutral.
>>
>> Fair enough.
>
>
> But ofcourse, *if* it's easy to change the template:
> //------------------
> template<class T>
> void moveBytes(T *destination, const char *source, const unsigned int
> multiplier=1)
> {
>    //  cout<<  "sizeof(*destination) = "<<  dec<<  sizeof(*destination)
> <<  endl;
>    memcpy(destination,&source[memLocation],
> 		 (multiplier*sizeof(*destination)));
>    memLocation += (multiplier*sizeof(*destination));
> }
> //------------------
>
> to be endian neutral, then I would be interested in seeing/hearing
> more about that.

one endian neutral way to write this for specific types would be:

void moveByte<int32_t>(int32_t *destination, const char *source, const 
unsigned int multiplier=1){
   unsigned char* sour = static_cast<unsigned char*>(source);

   for(int i=0; i<multiplier; i++){
     *destination++ = (static_cast<int32_t>(sour[3])<<24) | 
(static_cast<int32_t>(sour[2])<<16) | (static_cast<int32_t>(sour[1])<<8) 
| (static_cast<int32_t>(sour[0]));
     sour += 4;
   }
   memLocation += multiplier*sizeof(*destination);
}


>
> However, I don't think I'll need it for this project (it would only be
> good to know, for my own interests sake)... Most Windows and Linux
> pc's are little-endian, I guess? (I'm on ubuntu linux and I think the
> binary file I'm reading came from a windows pc)..

The processor used by Windows, and most Linux installations is the 
"Intel x86 compatible processor" which is little endian. Linux is 
available on both endian processors though. The fact that Unix lived on 
both endians is one reason there is the network order libraries, as they 
were helpful in writing portable libraries to handle network traffic.

[toc] | [prev] | [next] | [standalone]

#10909

On Oct 17, 2:03 am, Richard Damon <news.x.richardda...@xoxy.net>
wrote:
> On 10/16/11 6:19 PM, someone wrote:
> > But ofcourse, *if* it's easy to change the template:
> > //------------------
> > template<class T>
> > void moveBytes(T *destination, const char *source, const unsigned int
> > multiplier=1)
> > {
> >    //  cout<<  "sizeof(*destination) = "<<  dec<<  sizeof(*destination)
> > <<  endl;
> >    memcpy(destination,&source[memLocation],
> >             (multiplier*sizeof(*destination)));
> >    memLocation += (multiplier*sizeof(*destination));
> > }
> > //------------------
>
> > to be endian neutral, then I would be interested in seeing/hearing
> > more about that.
>
> one endian neutral way to write this for specific types would be:
>
> void moveByte<int32_t>(int32_t *destination, const char *source, const
> unsigned int multiplier=1){
>    unsigned char* sour = static_cast<unsigned char*>(source);
>
>    for(int i=0; i<multiplier; i++){
>      *destination++ = (static_cast<int32_t>(sour[3])<<24) |
> (static_cast<int32_t>(sour[2])<<16) | (static_cast<int32_t>(sour[1])<<8)
> | (static_cast<int32_t>(sour[0]));
>      sour += 4;
>    }
>    memLocation += multiplier*sizeof(*destination);
>
> }

Ah, ok... It's a shame that by saying that this is only valid for
int32_t then the "template-effect" of being able to work on many
different datatypes is lost. I had this sizeof(*destination) so the
template worked for 16bit, 32 and 64 bit data types. But if this is
how to do it, then I suppose that this is how to do it :-)

> > However, I don't think I'll need it for this project (it would only be
> > good to know, for my own interests sake)... Most Windows and Linux
> > pc's are little-endian, I guess? (I'm on ubuntu linux and I think the
> > binary file I'm reading came from a windows pc)..
>
> The processor used by Windows, and most Linux installations is the
> "Intel x86 compatible processor" which is little endian. Linux is
> available on both endian processors though. The fact that Unix lived on
> both endians is one reason there is the network order libraries, as they
> were helpful in writing portable libraries to handle network traffic.

Ok... Maybe I should have a look at that network stuff some day... I
have access to real unix-machines and then they're probably big-endian
and then I would probably get a problem... Most of the time, however I
use intel x86 processors...

Thanks! :-)

[toc] | [prev] | [next] | [standalone]

#10911

On 10/16/11 8:49 PM, someone wrote:
> On Oct 17, 2:03 am, Richard Damon<news.x.richardda...@xoxy.net>
> wrote:
>> On 10/16/11 6:19 PM, someone wrote:
>>> But ofcourse, *if* it's easy to change the template:
>>> //------------------
>>> template<class T>
>>> void moveBytes(T *destination, const char *source, const unsigned int
>>> multiplier=1)
>>> {
>>>     //  cout<<    "sizeof(*destination) = "<<    dec<<    sizeof(*destination)
>>> <<    endl;
>>>     memcpy(destination,&source[memLocation],
>>>              (multiplier*sizeof(*destination)));
>>>     memLocation += (multiplier*sizeof(*destination));
>>> }
>>> //------------------
>>
>>> to be endian neutral, then I would be interested in seeing/hearing
>>> more about that.
>>
>> one endian neutral way to write this for specific types would be:
>>
>> void moveByte<int32_t>(int32_t *destination, const char *source, const
>> unsigned int multiplier=1){
>>     unsigned char* sour = static_cast<unsigned char*>(source);
>>
>>     for(int i=0; i<multiplier; i++){
>>       *destination++ = (static_cast<int32_t>(sour[3])<<24) |
>> (static_cast<int32_t>(sour[2])<<16) | (static_cast<int32_t>(sour[1])<<8)
>> | (static_cast<int32_t>(sour[0]));
>>       sour += 4;
>>     }
>>     memLocation += multiplier*sizeof(*destination);
>>
>> }
>
> Ah, ok... It's a shame that by saying that this is only valid for
> int32_t then the "template-effect" of being able to work on many
> different datatypes is lost. I had this sizeof(*destination) so the
> template worked for 16bit, 32 and 64 bit data types. But if this is
> how to do it, then I suppose that this is how to do it :-)
>

This isn't the only way, but is a straight forward way. If you look 
closely at the code, you may be able to convert the fetch into a length 
independent routine. Writing it out discreetly for a single case does 
make it clear to read in my opinion.

One thing to note, is that when reading floating point numbers, there is 
more to be concerned about than endianness, as the exact placement of 
bits is not specified by the C standard. Again, the fact the the two 
machines are likely on the same family of processors lets you ignore 
some of the thornier issues.


>>> However, I don't think I'll need it for this project (it would only be
>>> good to know, for my own interests sake)... Most Windows and Linux
>>> pc's are little-endian, I guess? (I'm on ubuntu linux and I think the
>>> binary file I'm reading came from a windows pc)..
>>
>> The processor used by Windows, and most Linux installations is the
>> "Intel x86 compatible processor" which is little endian. Linux is
>> available on both endian processors though. The fact that Unix lived on
>> both endians is one reason there is the network order libraries, as they
>> were helpful in writing portable libraries to handle network traffic.
>
> Ok... Maybe I should have a look at that network stuff some day... I
> have access to real unix-machines and then they're probably big-endian
> and then I would probably get a problem... Most of the time, however I
> use intel x86 processors...
>
> Thanks! :-)
>
>
>
>

[toc] | [prev] | [next] | [standalone]

#10915

On Sun, 2011-10-16, Richard Damon wrote:
> On 10/16/11 2:21 PM, Jorgen Grahn wrote:
>> On Sun, 2011-10-16, Richard Damon wrote:
>>> On 10/15/11 6:43 PM, Martin Jørgensen wrote:
>> ...
>>>> Hmm... Based on my output to Richard - do I then have an endian
>>>> problem? Sorry I'm a bit too unexperienced to fully understand this
>>>> yet. A few replies and I think I'll understand it..
>>
>>> Your data in the file is little endian. You have a problem if your
>>> processor is not little endian.
>>
>> The usual interpretation is that this means he *does* have a problem.
>> Good code is supposed to have the same behavior no matter what
>> endianness the processor happens to have.

> Only so far as he need to confirm that his machine is little endian. 
> Good code runs reliably on the machines it is targeted, and minimizes 
> the possible issues in moving it to other machines it might possible by 
> run on.

I disagree, although I won't insist.

For two reasons:

- Doing binary I/O by casting various types (int, structs) gets you
  into trouble sooner or later. Learning to do it right is a good
  investment.

- I've spent months of my life, recently, fixing subtle endian bugs
  caused by reasoning like the above.  And I cursed the original
  developers all the way.  This is one of those things that is much,
  much easier to get right from the start compared to fixing it up
  afterwards.

> If the program is know to only need to be run on a limited set of 
> machines (like it is a windows program), than assuming properties common 
> to the windows platform isn't totally bad. Later version of windows are 
> very unlikely to switch endianness without other major issues coming up.

Yes, because people do things like this ... Unix software manages to
be portable to many different architectures without any major effort.

> Him move to having a template function move a given type of data is an 
> improvement, as it localizes the assumption. I would not at this point 
> rewrite those functions to be endian neutral.

Yes, isolating it to one place (and perhaps adding a TODO) is a good
start.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#10867

On 10/15/11 3:00 PM, someone wrote:
> Hi,
>
> Here's my program:
...
>    int R1_begin;
>    R1_begin = static_cast<typeof R1_begin>  (memblock[0]);//, 1,
> 'uint32',endianNess);
>    cout<<  "Value = "<<  R1_begin<<  endl;
>
>   delete[] memblock;
>    return 0;
> }
> ==========================
>
> Problem:
>
> R1_begin is the first value read from a binary file. But it's wrong!
> It should be 72 but the program prints 49 out to the screen. So I
> cannot continue before I get this right. The first few bytes I'm going
> to read is:
>
> R1begin = 1 byte times sizeof 'uint32' = 4 bytes
> I1_I6 = 6 bytes times sizeof 'uint32' = 6x4 = 24 bytes
> TEXT = 40 ASCII characters of type 'char' = 40 bytes
> R1end = 1 byte times sizeof 'uint32' = 4 bytes
>
> Total bytes read = 4+24+40+4 = 72 bytes. That's why I must read 72
> bytes from the very first byte in the binary file... And I cannot
> solve this! :-(
>
> Please help (and please provide hints for reading the next few bytes
> and how I should convert from data in memory into different data
> types). Later I also need to convert some bytes from memblock[ ] into
> floats and doubles...
>
> I'm stuck...
>
> And what about big endian / little endian? I'm confused. Looking
> forward to hear from you!
>
>
First, what is the first byte in the file? you are saying that you are 
expecting it to be 72 but you are getting 49. Have you looked at the file.

Next you talk about getting R1begin which should be 4 bytes, but you are 
setting it to the value of the first byte of the file (memblock[0] is 
the first byte of the file) the static_cast to int is unneeded, as char 
will convert to int implicitly.

Lastly, you are saying that you are expecting to read 72 bytes at the 
beginning of the file, and also that you are expecting to get the value 
72 from the beginning of the file, implying that R1begin is a byte count 
for the record, but then don't describe the record format depending on 
that value at all, and then only look at 1 byte for that info, when you 
describe it as a 4 byte number.

If R1_begin ins't a data length field, why are you expecting it to have 
the value 72?

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | comp.lang.c++

csiph-web