Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #142757 > unrolled thread

memcpy or assign data to uint64 on little endian(linux) system

Back to article view | Back to comp.lang.c

Contents

  memcpy or assign data to uint64 on little endian(linux) system WalkeR Park <walker.ykpark@gmail.com> - 2019-04-02 18:14 -0700
    Re: memcpy or assign data to uint64 on little endian(linux) system Geoff <geoff@invalid.invalid> - 2019-04-02 22:49 -0700
      Re: memcpy or assign data to uint64 on little endian(linux) system Ike Naar <ike@iceland.freeshell.org> - 2019-04-03 06:14 +0000
        Re: memcpy or assign data to uint64 on little endian(linux) system Geoff <geoff@invalid.invalid> - 2019-04-02 23:22 -0700
        Re: memcpy or assign data to uint64 on little endian(linux) system Ben Bacarisse <ben.usenet@bsb.me.uk> - 2019-04-03 11:07 +0100
          Re: memcpy or assign data to uint64 on little endian(linux) system Jorgen Grahn <grahn+nntp@snipabacken.se> - 2019-04-03 12:56 +0000
            Re: memcpy or assign data to uint64 on little endian(linux) system fir <profesor.fir@gmail.com> - 2019-04-03 06:52 -0700
            Re: memcpy or assign data to uint64 on little endian(linux) system Ben Bacarisse <ben.usenet@bsb.me.uk> - 2019-04-03 16:06 +0100
    Re: memcpy or assign data to uint64 on little endian(linux) system fir <profesor.fir@gmail.com> - 2019-04-03 04:03 -0700
      Re: memcpy or assign data to uint64 on little endian(linux) system fir <profesor.fir@gmail.com> - 2019-04-03 04:13 -0700
    Re: memcpy or assign data to uint64 on little endian(linux) system Jorgen Grahn <grahn+nntp@snipabacken.se> - 2019-04-03 12:37 +0000
    Re: memcpy or assign data to uint64 on little endian(linux) system Ike Naar <ike@iceland.freeshell.org> - 2019-04-03 16:10 +0000
      Re: memcpy or assign data to uint64 on little endian(linux) system Keith Thompson <kst-u@mib.org> - 2019-04-03 12:50 -0700

#142757 — memcpy or assign data to uint64 on little endian(linux) system

Hi,
I'm just porting some source that is written from HP-UX to the linux.
AKA, HP is Big-Endian and liunx is Little-Endian.

In source of HP, threa are many memcpy, and Assign Data Source...
Porting is OK, but In that case "memcpy or assign data to uint64",  
Data is stored reversely.

ex)
=================================================
        uint64 a;
        char b[8] = {0x38, 0x12, 0x12, 0x12, 0x81, 0x28, 0x28, 0x28};
        memcpy(&a, b, 8);
        printf("TEST [%lu]\n", a);
=================================================

I know that Just using and adjusting htonll or htonl whatnot...
But Source is huge scale, so I'm not able to Every code line that I'm gonna find.

Is there any solution using kernel setting and so on but changing every source code line?

regards.

[toc] | [next] | [standalone]

#142761

On Tue, 2 Apr 2019 18:14:31 -0700 (PDT), WalkeR Park
<walker.ykpark@gmail.com> wrote:

>Hi,
>I'm just porting some source that is written from HP-UX to the linux.
>AKA, HP is Big-Endian and liunx is Little-Endian.
>
>In source of HP, threa are many memcpy, and Assign Data Source...
>Porting is OK, but In that case "memcpy or assign data to uint64",  
>Data is stored reversely.
>
>ex)
>=================================================
>        uint64 a;
>        char b[8] = {0x38, 0x12, 0x12, 0x12, 0x81, 0x28, 0x28, 0x28};
>        memcpy(&a, b, 8);
>        printf("TEST [%lu]\n", a);
>=================================================
>
>I know that Just using and adjusting htonll or htonl whatnot...
>But Source is huge scale, so I'm not able to Every code line that I'm gonna find.
>
>Is there any solution using kernel setting and so on but changing every source code line?
>
>regards.

Consider writing a reversing memmove (r_memmove) then 
#define memcpy r_memmove
for the sections where this is needed.

Prefer memmove over memcpy.

[toc] | [prev] | [next] | [standalone]

#142762

On 2019-04-03, Geoff <geoff@invalid.invalid> wrote:
> On Tue, 2 Apr 2019 18:14:31 -0700 (PDT), WalkeR Park
><walker.ykpark@gmail.com> wrote:
>
>>Hi,
>>I'm just porting some source that is written from HP-UX to the linux.
>>AKA, HP is Big-Endian and liunx is Little-Endian.
>>
>>In source of HP, threa are many memcpy, and Assign Data Source...
>>Porting is OK, but In that case "memcpy or assign data to uint64",  
>>Data is stored reversely.
>>
>>ex)
>>=================================================
>>        uint64 a;
>>        char b[8] = {0x38, 0x12, 0x12, 0x12, 0x81, 0x28, 0x28, 0x28};
>>        memcpy(&a, b, 8);
>>        printf("TEST [%lu]\n", a);
>>=================================================
>>
>>I know that Just using and adjusting htonll or htonl whatnot...
>>But Source is huge scale, so I'm not able to Every code line that I'm gonna find.
>>
>>Is there any solution using kernel setting and so on but changing every source code line?
>>
>>regards.
>
> Consider writing a reversing memmove (r_memmove) then 
> #define memcpy r_memmove
> for the sections where this is needed.
>
> Prefer memmove over memcpy.

Or consider initializing a as

  uint64_t a = 0x3812121281282828;

This should behave the same on big-endian and little-endian machines.

[toc] | [prev] | [next] | [standalone]

#142763

On Wed, 3 Apr 2019 06:14:15 -0000 (UTC), Ike Naar
<ike@iceland.freeshell.org> wrote:

>On 2019-04-03, Geoff <geoff@invalid.invalid> wrote:
>> On Tue, 2 Apr 2019 18:14:31 -0700 (PDT), WalkeR Park
>><walker.ykpark@gmail.com> wrote:
>>
>>>Hi,
>>>I'm just porting some source that is written from HP-UX to the linux.
>>>AKA, HP is Big-Endian and liunx is Little-Endian.
>>>
>>>In source of HP, threa are many memcpy, and Assign Data Source...
>>>Porting is OK, but In that case "memcpy or assign data to uint64",  
>>>Data is stored reversely.
>>>
>>>ex)
>>>=================================================
>>>        uint64 a;
>>>        char b[8] = {0x38, 0x12, 0x12, 0x12, 0x81, 0x28, 0x28, 0x28};
>>>        memcpy(&a, b, 8);
>>>        printf("TEST [%lu]\n", a);
>>>=================================================
>>>
>>>I know that Just using and adjusting htonll or htonl whatnot...
>>>But Source is huge scale, so I'm not able to Every code line that I'm gonna find.
>>>
>>>Is there any solution using kernel setting and so on but changing every source code line?
>>>
>>>regards.
>>
>> Consider writing a reversing memmove (r_memmove) then 
>> #define memcpy r_memmove
>> for the sections where this is needed.
>>
>> Prefer memmove over memcpy.
>
>Or consider initializing a as
>
>  uint64_t a = 0x3812121281282828;
>
>This should behave the same on big-endian and little-endian machines.

It was my understanding that the OP didn't want to alter the source
code lines doing the initializations.

[toc] | [prev] | [next] | [standalone]

#142765

Ike Naar <ike@iceland.freeshell.org> writes:

> On 2019-04-03, Geoff <geoff@invalid.invalid> wrote:
>> On Tue, 2 Apr 2019 18:14:31 -0700 (PDT), WalkeR Park
>>>=================================================
>>>        uint64 a;
>>>        char b[8] = {0x38, 0x12, 0x12, 0x12, 0x81, 0x28, 0x28, 0x28};
>>>        memcpy(&a, b, 8);
>>>        printf("TEST [%lu]\n", a);
>>>=================================================
<cut>
>> Consider writing a reversing memmove (r_memmove) then 
>> #define memcpy r_memmove
>> for the sections where this is needed.
>>
>> Prefer memmove over memcpy.
>
> Or consider initializing a as
>
>   uint64_t a = 0x3812121281282828;
>
> This should behave the same on big-endian and little-endian machines.

That gives the same numerical value, but you can't just tell by looking
at a particular memcpy line, if the intent is numerical or not.  For
example, if you are setting up a struct representing a network packet,
it's the memcpy that will set everything correctly regardless of
endianness, whereas assigning numerical values will go wrong.  You won't
need much context to decide, but the copy line itself probably won't be
enough.

It looks like a thorough and careful isolation of all the parts the assume
byte order is needed.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#142776

On Wed, 2019-04-03, Ben Bacarisse wrote:
> Ike Naar <ike@iceland.freeshell.org> writes:
>
>> On 2019-04-03, Geoff <geoff@invalid.invalid> wrote:
>>> On Tue, 2 Apr 2019 18:14:31 -0700 (PDT), WalkeR Park
>>>>=================================================
>>>>        uint64 a;
>>>>        char b[8] = {0x38, 0x12, 0x12, 0x12, 0x81, 0x28, 0x28, 0x28};
>>>>        memcpy(&a, b, 8);
>>>>        printf("TEST [%lu]\n", a);
>>>>=================================================
...

> It looks like a thorough and careful isolation of all the parts the assume
> byte order is needed.

Yes.  It can be a long and painful procedure, and you learn to hate
the people who wrote the original, broken code.

My primary recommendation would be to normalize the data close to the
I/O, at the edge.  Don't end up in the situation where the main
program logic deals with a mix or native and foreign numbers, and you
need to keep track of which int is a real integer and which isn't.  If
you need to keep foreign data, put it in a struct and treat it as an
opaque blob.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#142778

W dniu środa, 3 kwietnia 2019 14:57:10 UTC+2 użytkownik Jorgen Grahn napisał:
> On Wed, 2019-04-03, Ben Bacarisse wrote:
> > Ike Naar <ike@iceland.freeshell.org> writes:
> >
> >> On 2019-04-03, Geoff <geoff@invalid.invalid> wrote:
> >>> On Tue, 2 Apr 2019 18:14:31 -0700 (PDT), WalkeR Park
> >>>>=================================================
> >>>>        uint64 a;
> >>>>        char b[8] = {0x38, 0x12, 0x12, 0x12, 0x81, 0x28, 0x28, 0x28};
> >>>>        memcpy(&a, b, 8);
> >>>>        printf("TEST [%lu]\n", a);
> >>>>=================================================
> ...
> 
> > It looks like a thorough and careful isolation of all the parts the assume
> > byte order is needed.
> 
> Yes.  It can be a long and painful procedure, and you learn to hate
> the people who wrote the original, broken code.
> 

lol, no, not learn to hate em

this is becouse they might be ones who simply not wanted to write endian-portable code

you would better blame those who wanted to port this and make that kind of screams (and ewentualy blame others on decisions they were right to take)

besides what options you got if to work on bytes of things like int64? mask and shifts? does niot look to much good

i once proposed an extension in C2 as to give in language

int64 x = smthing;

x.lo.hi.lo.lo = 15 //set left nibble of byte 2 to 0b1111

or  also

x.byte[6] = 123;
x.bit[53] = 1;

that would resolve some problems (probably not all though)



> My primary recommendation would be to normalize the data close to the
> I/O, at the edge.  Don't end up in the situation where the main
> program logic deals with a mix or native and foreign numbers, and you
> need to keep track of which int is a real integer and which isn't.  If
> you need to keep foreign data, put it in a struct and treat it as an
> opaque blob.
> 
> /Jorgen
> 

[toc] | [prev] | [next] | [standalone]

#142782

Jorgen Grahn <grahn+nntp@snipabacken.se> writes:

> On Wed, 2019-04-03, Ben Bacarisse wrote:
>> Ike Naar <ike@iceland.freeshell.org> writes:
>>
>>> On 2019-04-03, Geoff <geoff@invalid.invalid> wrote:
>>>> On Tue, 2 Apr 2019 18:14:31 -0700 (PDT), WalkeR Park
>>>>>=================================================
>>>>>        uint64 a;
>>>>>        char b[8] = {0x38, 0x12, 0x12, 0x12, 0x81, 0x28, 0x28, 0x28};
>>>>>        memcpy(&a, b, 8);
>>>>>        printf("TEST [%lu]\n", a);
>>>>>=================================================
> ...
>
>> It looks like a thorough and careful isolation of all the parts the assume
>> byte order is needed.
>
> Yes.  It can be a long and painful procedure, and you learn to hate
> the people who wrote the original, broken code.
>
> My primary recommendation would be to normalize the data close to the
> I/O, at the edge.  Don't end up in the situation where the main
> program logic deals with a mix or native and foreign numbers, and you
> need to keep track of which int is a real integer and which isn't.

That's a good point.  It can be tempting, in the name of efficiency, to
leave some data alone as it comes into the program because you can see
that it's going out again pretty soon, but avoiding the two conversions
can be a false economy.

The I/O boundary is usually easy to identify so it is a good idea to try
to sort the problem our there.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#142766

W dniu środa, 3 kwietnia 2019 03:14:48 UTC+2 użytkownik WalkeR Park napisał:
> Hi,
> I'm just porting some source that is written from HP-UX to the linux.
> AKA, HP is Big-Endian and liunx is Little-Endian.
> 
> In source of HP, threa are many memcpy, and Assign Data Source...
> Porting is OK, but In that case "memcpy or assign data to uint64",  
> Data is stored reversely.
> 
> ex)
> =================================================
>         uint64 a;
>         char b[8] = {0x38, 0x12, 0x12, 0x12, 0x81, 0x28, 0x28, 0x28};
>         memcpy(&a, b, 8);
>         printf("TEST [%lu]\n", a);
> =================================================
> 
> I know that Just using and adjusting htonll or htonl whatnot...
> But Source is huge scale, so I'm not able to Every code line that I'm gonna find.
> 
> Is there any solution using kernel setting and so on but changing every source code line?
> 
> regards.

in that case above dont think its a way to know if such memcopy should be reversed or not (becouse you would need to track that it is memcopy to uint64 (or uint32 uint16) Have you a way to know which memcopy you would need to reverse?

as a reserch and fun hoevver i could write a wrapper on memcopy and collect some statistics (like histogram, hov many memcopies on which sizes are called), than you could also try blind hit and reverse all that are 8 bytes length (4? 2?) or so

besides regatdles of hov many buig scale it is it is possible to do that manually, hov much you cant do? you got more then 1000 of it? people do a lot more, imagina all that types of work... at worsc count money for all that time of work chill and ok

[toc] | [prev] | [next] | [standalone]

#142767

W dniu środa, 3 kwietnia 2019 13:03:42 UTC+2 użytkownik fir napisał:
> W dniu środa, 3 kwietnia 2019 03:14:48 UTC+2 użytkownik WalkeR Park napisał:
> > Hi,
> > I'm just porting some source that is written from HP-UX to the linux.
> > AKA, HP is Big-Endian and liunx is Little-Endian.
> > 
> > In source of HP, threa are many memcpy, and Assign Data Source...
> > Porting is OK, but In that case "memcpy or assign data to uint64",  
> > Data is stored reversely.
> > 
> > ex)
> > =================================================
> >         uint64 a;
> >         char b[8] = {0x38, 0x12, 0x12, 0x12, 0x81, 0x28, 0x28, 0x28};
> >         memcpy(&a, b, 8);
> >         printf("TEST [%lu]\n", a);
> > =================================================
> > 
> > I know that Just using and adjusting htonll or htonl whatnot...
> > But Source is huge scale, so I'm not able to Every code line that I'm gonna find.
> > 
> > Is there any solution using kernel setting and so on but changing every source code line?
> > 
> > regards.
> 
> in that case above dont think its a way to know if such memcopy should be reversed or not (becouse you would need to track that it is memcopy to uint64 (or uint32 uint16) Have you a way to know which memcopy you would need to reverse?
> 
> as a reserch and fun hoevver i could write a wrapper on memcopy and collect some statistics (like histogram, hov many memcopies on which sizes are called), than you could also try blind hit and reverse all that are 8 bytes length (4? 2?) or so
> 
> besides regatdles of hov many buig scale it is it is possible to do that manually, hov much you cant do? you got more then 1000 of it? people do a lot more, imagina all that types of work... at worsc count money for all that time of work chill and ok

you could eventually write a script-like program (depend hov different cases you get) and detect tat cases in source and run it on source to make it be rewritten by your script

[toc] | [prev] | [next] | [standalone]

#142771

On Wed, 2019-04-03, WalkeR Park wrote:
> Hi,
> I'm just porting some source that is written from HP-UX to the linux.
> AKA, HP is Big-Endian and liunx is Little-Endian.

Linux runs on both big-endian and little-endian machines, but I
suspect you want to build for Linux running on x86_64, which is
little-endian.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#142784

On 2019-04-03, WalkeR Park <walker.ykpark@gmail.com> wrote:
> Hi,
> I'm just porting some source that is written from HP-UX to the linux.
> AKA, HP is Big-Endian and liunx is Little-Endian.
>
> In source of HP, threa are many memcpy, and Assign Data Source...
> Porting is OK, but In that case "memcpy or assign data to uint64",  
> Data is stored reversely.
>
> ex)
>=================================================
>         uint64 a;
>         char b[8] = {0x38, 0x12, 0x12, 0x12, 0x81, 0x28, 0x28, 0x28};

Perhaps you want to change that into

          unsigned char b[8] = {0x38, 0x12, 0x12, 0x12, 0x81, 0x28, 0x28, 0x28};

because, on a platform where char is signed, e.g. 0x81 will not fit.

[toc] | [prev] | [next] | [standalone]

#142811

Ike Naar <ike@iceland.freeshell.org> writes:
> On 2019-04-03, WalkeR Park <walker.ykpark@gmail.com> wrote:
[...]
>>         uint64 a;
>>         char b[8] = {0x38, 0x12, 0x12, 0x12, 0x81, 0x28, 0x28, 0x28};
>
> Perhaps you want to change that into
>
>           unsigned char b[8] = {0x38, 0x12, 0x12, 0x12, 0x81, 0x28, 0x28, 0x28};
>
> because, on a platform where char is signed, e.g. 0x81 will not fit.

Agreed.

It's very likely that the version with plain char will work, because the
implementation-defined conversion from unsigned char to plain char
almost certainly preserves the bit pattern, but using unsigned char
removes that very minor uncertainty.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c

csiph-web