Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #68899 > unrolled thread

On endianness, many #ifdefs, and the need thereof

Back to article view | Back to comp.lang.c++

Contents

  On endianness, many #ifdefs, and the need thereof Daniel <danielaparker@gmail.com> - 2019-10-02 04:00 -0700
    Re: On endianness, many #ifdefs, and the need thereof David Brown <david.brown@hesbynett.no> - 2019-10-02 13:26 +0200
      Re: On endianness, many #ifdefs, and the need thereof Daniel <danielaparker@gmail.com> - 2019-10-02 08:59 -0700
    Re: On endianness, many #ifdefs, and the need thereof Öö Tiib <ootiib@hot.ee> - 2019-10-02 05:42 -0700
    Re: On endianness, many #ifdefs, and the need thereof Juha Nieminen <nospam@thanks.invalid> - 2019-10-02 12:53 +0000
      Re: On endianness, many #ifdefs, and the need thereof David Brown <david.brown@hesbynett.no> - 2019-10-02 15:05 +0200
        Re: On endianness, many #ifdefs, and the need thereof scott@slp53.sl.home (Scott Lurndal) - 2019-10-02 13:54 +0000
          Re: On endianness, many #ifdefs, and the need thereof Keith Thompson <kst-u@mib.org> - 2019-10-02 12:54 -0700
            Re: On endianness, many #ifdefs, and the need thereof scott@slp53.sl.home (Scott Lurndal) - 2019-10-02 20:16 +0000
          Re: On endianness, many #ifdefs, and the need thereof David Brown <david.brown@hesbynett.no> - 2019-10-03 09:48 +0200
      Re: On endianness, many #ifdefs, and the need thereof Daniel <danielaparker@gmail.com> - 2019-10-02 07:52 -0700
        Re: On endianness, many #ifdefs, and the need thereof Öö Tiib <ootiib@hot.ee> - 2019-10-02 08:04 -0700
          Re: On endianness, many #ifdefs, and the need thereof Daniel <danielaparker@gmail.com> - 2019-10-02 08:31 -0700
            Re: On endianness, many #ifdefs, and the need thereof Juha Nieminen <nospam@thanks.invalid> - 2019-10-02 21:32 +0000
        Re: On endianness, many #ifdefs, and the need thereof Jorgen Grahn <grahn+nntp@snipabacken.se> - 2019-10-03 11:33 +0000
          Re: On endianness, many #ifdefs, and the need thereof David Brown <david.brown@hesbynett.no> - 2019-10-03 13:46 +0200
            Re: On endianness, many #ifdefs, and the need thereof Daniel <danielaparker@gmail.com> - 2019-10-03 05:56 -0700
              Re: On endianness, many #ifdefs, and the need thereof scott@slp53.sl.home (Scott Lurndal) - 2019-10-03 13:17 +0000
                Re: On endianness, many #ifdefs, and the need thereof David Brown <david.brown@hesbynett.no> - 2019-10-03 16:03 +0200
                  Re: On endianness, many #ifdefs, and the need thereof Daniel <danielaparker@gmail.com> - 2019-10-03 07:13 -0700
            Re: On endianness, many #ifdefs, and the need thereof Jorgen Grahn <grahn+nntp@snipabacken.se> - 2019-10-03 18:51 +0000
            Re: On endianness, many #ifdefs, and the need thereof Keith Thompson <kst-u@mib.org> - 2019-10-03 14:28 -0700
              Re: On endianness, many #ifdefs, and the need thereof David Brown <david.brown@hesbynett.no> - 2019-10-04 10:11 +0200
                Re: On endianness, many #ifdefs, and the need thereof Daniel <danielaparker@gmail.com> - 2019-10-04 09:00 -0700
          Re: On endianness, many #ifdefs, and the need thereof Paavo Helde <myfirstname@osa.pri.ee> - 2019-10-03 14:46 +0300
          Re: On endianness, many #ifdefs, and the need thereof Daniel <danielaparker@gmail.com> - 2019-10-03 05:40 -0700
    Re: On endianness, many #ifdefs, and the need thereof Bonita Montero <Bonita.Montero@gmail.com> - 2019-10-02 15:10 +0200
    Re: On endianness, many #ifdefs, and the need thereof Barry Schwarz <schwarzb@dqel.com> - 2019-10-02 08:33 -0700
      Re: On endianness, many #ifdefs, and the need thereof David Brown <david.brown@hesbynett.no> - 2019-10-03 09:54 +0200
    Re: On endianness, many #ifdefs, and the need thereof "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2019-10-02 19:35 +0200
      Re: On endianness, many #ifdefs, and the need thereof Daniel <danielaparker@gmail.com> - 2019-10-02 15:39 -0700
    Re: On endianness, many #ifdefs, and the need thereof "Oliver S." <Follow.Me@gmx.net> - 2019-10-02 19:59 +0200

Page 2 of 2 — ← Prev page 1 [2]

#68990

On Thu, 2019-10-03, David Brown wrote:
> On 03/10/2019 13:33, Jorgen Grahn wrote:
>> On Wed, 2019-10-02, Daniel wrote:
...
>>> The example is from the nlohmann json library, where I first saw it, 
>>> https://github.com/nlohmann/json/ [...]
>> 
>> Weird and slightly worrying that a JSON library would have to be
>> endianness-aware ...
>> 
>> /Jorgen,
>>  too lazy to check the source code
>
> A little look suggests that it is for parsing binary JSON formats (like
> UBJSON),

I can't help misreading that as "Undefined Behaviour JSON" :-)

Thanks.  I'm not into newish file formats, and didn't know things like
this existed.  (Maybe ASN.1 wasn't such a bad idea after all?)

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#69005

David Brown <david.brown@hesbynett.no> writes:
[...]
> A little look suggests that it is for parsing binary JSON formats (like
> UBJSON), rather than standard text-based JSON.  But is still not
> necessary, and I doubt if the code's algorithm is more efficient than
> the obvious endian independent method.

FYI, my browser shows me a "This site may be hacked." warning for
    http://ubjson.org/
I haven't investigated further.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */

[toc] | [prev] | [next] | [standalone]

#69025

On 03/10/2019 23:28, Keith Thompson wrote:
> David Brown <david.brown@hesbynett.no> writes:
> [...]
>> A little look suggests that it is for parsing binary JSON formats (like
>> UBJSON), rather than standard text-based JSON.  But is still not
>> necessary, and I doubt if the code's algorithm is more efficient than
>> the obvious endian independent method.
> 
> FYI, my browser shows me a "This site may be hacked." warning for
>     http://ubjson.org/
> I haven't investigated further.
> 

I think that's just the modern (and IMHO ridiculous and
counter-productive) trend for browsers to get their knickers in a twist
over "http" sites - as though using "https" automatically made a site
safe and secure.

It certainly doesn't look hacked to me.  Of course, it might have been a
temporary issue, or maybe you are the victim of a man-in-the-middle
attack, and https really would have been useful!

If you are remotely interested in UBJSON itself, Wikipedia is a safe bet:

<https://en.wikipedia.org/wiki/UBJSON>

It looks like an interesting format, but it bugs me a little that they
distinguish between int8 and uint8, but do not have unsigned versions of
other integer sizes.

[toc] | [prev] | [next] | [standalone]

#69060

On Friday, October 4, 2019 at 4:11:27 AM UTC-4, David Brown wrote:
> 
> <https://en.wikipedia.org/wiki/UBJSON>
> 
> It [UBJSON] looks like an interesting format, but it bugs me a little that 
> they distinguish between int8 and uint8, but do not have unsigned versions 
> of other integer sizes.

Compared to other JSON like binary formats, UBJSON has one interesting 
feature, strongly typed arrays and objects,
see http://ubjson.org/type-reference/container-types/#optimized-format-example-array. 
This allows the value type tags for items in an array to be omitted, when the 
value type is the same. In other respects, though, UBJSON falls short of other 
JSON like binary formats such as Message Pack and CBOR, both in terms of 
support for types and for compactness. CBOR, in particular, has all the 
momentum, being defined in an Internet Standards Document, RFC 7049, 
https://tools.ietf.org/html/rfc7049, and now widely supported in tooling 
across different programming languages. CBOR supports extensible tagging of 
byte sequences, and RFC's are emerging that take advantage of that, e.g. CBOR 
Tags for Typed Arrays, https://tools.ietf.org/html/draft-ietf-cbor-array-tags-
07.  

Daniel
https://github.com/danielaparker/jsoncons

[toc] | [prev] | [next] | [standalone]

#68952

On 3.10.2019 14:33, Jorgen Grahn wrote:
> Weird and slightly worrying that a JSON library would have to be
> endianness-aware ...

A comment from that library:

@note This function needs to respect the system's endianess, because bytes
in CBOR, MessagePack, and UBJSON are stored in network order (big
endian) and therefore need reordering on little endian systems.

[toc] | [prev] | [next] | [standalone]

#68953

On Thursday, October 3, 2019 at 7:34:06 AM UTC-4, Jorgen Grahn wrote:
> 
> Weird and slightly worrying that a JSON library would have to be
> endianness-aware ...
> 
A strictly JSON library need not be, but a number of JSON libraries include 
support for JSON-like binary formats, and nlohmann includes limited support for 
CBOR, Message Pack, UBJSON and BSON.

Daniel

[toc] | [prev] | [next] | [standalone]

#68906

I think it's the best to have wrapper-classes for endianess'ed data
-types which are externally exchanged. This might might internally
use, depending on the type of the compiler per #ifdef, manual swapping
of the bytes or an appropriate intrinsic.

[toc] | [prev] | [next] | [standalone]

#68919

On Wed, 2 Oct 2019 04:00:39 -0700 (PDT), Daniel
<danielaparker@gmail.com> wrote:

>Most C++ software that needs to know whether the host is big endian or little 
>endian comes with a header file with a very long sequence of #ifdef's, along 
>the lines of
>
>#    if (defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__) || \
>    (defined(__BYTE_ORDER) && defined(__BIG_ENDIAN) && __BYTE_ORDER == __BIG_ENDIAN) || \
>    (defined(BYTE_ORDER) && defined(BIG_ENDIAN) && BYTE_ORDER == BIG_ENDIAN) || \
>    (defined(_BIG_ENDIAN) && !defined(_LITTLE_ENDIAN)) || (defined(__BIG_ENDIAN__) && !defined(__LITTLE_ENDIAN__)) || \
>    defined(__ARMEB__) || defined(__MIPSEB__) || defined(__s390__) || defined(__sparc__)
>
>On the other hand, at least one library merely has
> 
>static constexpr bool little_endianess(int num = 1) noexcept
>{
>    return *reinterpret_cast<char*>(&num) == 1;
>}
>
>referencing a contribution on stackoverflow, http://stackoverflow.com/a/1001328/266378.
>
>So why the many #ifdef's, if this is enough? Or is it enough?

Both approaches will allow the programmer to generate code for each
case as appropriate.

The macro approach will allow the compiler to generate code only for
the case that applies to that compilation (assuming appropriate uses
of the #if DIRECTIVE).  This will reduce the size of the generated
code.  Whether that is a significant consideration obviously depends
on the code.

The function approach will be evaluated at run time.  The compiler
will generate code for both cases and the program will then execute
only the relevant code (assuming appropriate uses of the if
STATEMENT).  The if will be evaluated whenever the code needs to
distinguish between the two cases, thus increasing the run time.
Whether this is a significant consideration obviously depends on the
code.

IMO, the macro approach is a bit more difficult to maintain if
portability is an issue since different compilers (or even updates to
the same compiler) may use different macro names that need to be
tested not to mention the nesting of && and || operations.  

OTOH, the function approach will fail if sizeof (int) is 1 (32-bit
char).

-- 
Remove del for email

[toc] | [prev] | [next] | [standalone]

#68940

On 02/10/2019 17:33, Barry Schwarz wrote:
> On Wed, 2 Oct 2019 04:00:39 -0700 (PDT), Daniel
> <danielaparker@gmail.com> wrote:
> 
>> Most C++ software that needs to know whether the host is big endian or little 
>> endian comes with a header file with a very long sequence of #ifdef's, along 
>> the lines of
>>
>> #    if (defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__) || \
>>    (defined(__BYTE_ORDER) && defined(__BIG_ENDIAN) && __BYTE_ORDER == __BIG_ENDIAN) || \
>>    (defined(BYTE_ORDER) && defined(BIG_ENDIAN) && BYTE_ORDER == BIG_ENDIAN) || \
>>    (defined(_BIG_ENDIAN) && !defined(_LITTLE_ENDIAN)) || (defined(__BIG_ENDIAN__) && !defined(__LITTLE_ENDIAN__)) || \
>>    defined(__ARMEB__) || defined(__MIPSEB__) || defined(__s390__) || defined(__sparc__)
>>
>> On the other hand, at least one library merely has
>>
>> static constexpr bool little_endianess(int num = 1) noexcept
>> {
>>    return *reinterpret_cast<char*>(&num) == 1;
>> }
>>
>> referencing a contribution on stackoverflow, http://stackoverflow.com/a/1001328/266378.
>>
>> So why the many #ifdef's, if this is enough? Or is it enough?
> 
> Both approaches will allow the programmer to generate code for each
> case as appropriate.
> 
> The macro approach will allow the compiler to generate code only for
> the case that applies to that compilation (assuming appropriate uses
> of the #if DIRECTIVE).  This will reduce the size of the generated
> code.  Whether that is a significant consideration obviously depends
> on the code.
> 
> The function approach will be evaluated at run time.  The compiler
> will generate code for both cases and the program will then execute
> only the relevant code (assuming appropriate uses of the if
> STATEMENT).  The if will be evaluated whenever the code needs to
> distinguish between the two cases, thus increasing the run time.
> Whether this is a significant consideration obviously depends on the
> code.

That is the /logical/ behaviour.  In practice, a good compiler can
evaluate the function at compile time (even though it is not a constant
expression), and use this along with dead code elimination to remove the
run-time check and code that cannot be called.

(Since it logically has the the test, the unused branch of code still
has to be valid and compilable code, unlike when you use conditional
compilation with the preprocessor.)

> 
> IMO, the macro approach is a bit more difficult to maintain if
> portability is an issue since different compilers (or even updates to
> the same compiler) may use different macro names that need to be
> tested not to mention the nesting of && and || operations.  
> 
> OTOH, the function approach will fail if sizeof (int) is 1 (32-bit
> char).
> 

You can happily combine the approaches:

#    if (defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__) &&
__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__) || \
    (defined(__BYTE_ORDER) && defined(__BIG_ENDIAN) && __BYTE_ORDER ==
__BIG_ENDIAN) || \
    (defined(BYTE_ORDER) && defined(BIG_ENDIAN) && BYTE_ORDER ==
BIG_ENDIAN) || \
    (defined(_BIG_ENDIAN) && !defined(_LITTLE_ENDIAN)) ||
(defined(__BIG_ENDIAN__) && !defined(__LITTLE_ENDIAN__)) || \
    defined(__ARMEB__) || defined(__MIPSEB__) || defined(__s390__) ||
defined(__sparc__)

constexpr bool is_big_endian = true;
constexpr bool is_little_endian = false;
#define IS_BIG_ENDIAN 1
#define IS_LITTLE_ENDIAN 0

#else

constexpr bool is_big_endian = false;
constexpr bool is_little_endian = true;
#define IS_BIG_ENDIAN 0
#define IS_LITTLE_ENDIAN 1

#endif


Then in the rest of your code, you can use conditional compilation on
"#if IS_BIG_ENDIAN", or use "if (is_big_endian)", or even "if
constexpr(is_big_endian)" for newer C++.

Of course, you'll still be in trouble on a PDP, but you can't have
everything!

[toc] | [prev] | [next] | [standalone]

#68921

On 02.10.2019 13:00, Daniel wrote:
> Most C++ software that needs to know whether the host is big endian or little
> endian comes with a header file with a very long sequence of #ifdef's, along
> the lines of
> 
> #    if (defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__) || \
>      (defined(__BYTE_ORDER) && defined(__BIG_ENDIAN) && __BYTE_ORDER == __BIG_ENDIAN) || \
>      (defined(BYTE_ORDER) && defined(BIG_ENDIAN) && BYTE_ORDER == BIG_ENDIAN) || \
>      (defined(_BIG_ENDIAN) && !defined(_LITTLE_ENDIAN)) || (defined(__BIG_ENDIAN__) && !defined(__LITTLE_ENDIAN__)) || \
>      defined(__ARMEB__) || defined(__MIPSEB__) || defined(__s390__) || defined(__sparc__)

C++20 will provide compile time endianness information.

<url: https://en.cppreference.com/w/cpp/types/endian> notes that one 
possible implementation is

enum class endian
{
#ifdef _WIN32
     little = 0,
     big    = 1,
     native = little
#else
     little = __ORDER_LITTLE_ENDIAN__,
     big    = __ORDER_BIG_ENDIAN__,
     native = __BYTE_ORDER__
#endif
};

Evidently that's because the number of fully fledged C++ compilers today 
is very limited, where each is either compatible with Visual C++ (the 
first branch) or with g++ (the second branch).


> On the other hand, at least one library merely has
>   
> static constexpr bool little_endianess(int num = 1) noexcept
> {
>      return *reinterpret_cast<char*>(&num) == 1;
> }
> 
> referencing a contribution on stackoverflow, http://stackoverflow.com/a/1001328/266378.

As noted else-thread this cannot be evaluated at compile time by a 
conforming compiler.

As not noted there (as I remember it), since there's no way the body 
/can/ be evaluated at compile time, the function definition is not valid 
as declared `constexpr`, and should not compile at all.

Indeed, Visual C++ 2019 says

foo.cpp(1): error C3615: constexpr function 'little_endianess' cannot 
result in a constant expression

and g++ 8.2.0 says

foo.cpp:3:17: error: a reinterpret_cast is not a constant expression
          return *reinterpret_cast<char*>(&num) == 1;
                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

> So why the many #ifdef's, if this is enough? Or is it enough?

It's reasonable to assume that that macro is many years old.


- Alf

[toc] | [prev] | [next] | [standalone]

#68928

On Wednesday, October 2, 2019 at 1:36:12 PM UTC-4, Alf P. Steinbach wrote:
> On 02.10.2019 13:00, Daniel wrote:
> 
> > On the other hand, at least one library merely has
> >   
> > static constexpr bool little_endianess(int num = 1) noexcept
> > {
> >      return *reinterpret_cast<char*>(&num) == 1;
> > }
> > 
> 
> As noted else-thread this cannot be evaluated at compile time by a 
> conforming compiler.
> 
> As not noted there (as I remember it), since there's no way the body 
> /can/ be evaluated at compile time, the function definition is not valid 
> as declared `constexpr`, and should not compile at all.
> 
> Indeed, Visual C++ 2019 says
> 
> foo.cpp(1): error C3615: constexpr function 'little_endianess' cannot 
> result in a constant expression
> 
> and g++ 8.2.0 says
> 
> foo.cpp:3:17: error: a reinterpret_cast is not a constant expression
>           return *reinterpret_cast<char*>(&num) == 1;
>                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
A curiosity is that

const bool val = nlohmann::detail::binary_reader<nlohmann::json>::little_endianess();

compiles in vs2019 version 16.2.5, where little_endianess() is as shown above. 
I'm assuming it must also compile with clang and g++ on linux and OSX, as it's 
exercised in continuous integration builds. It fails in vs2019 version 16.2.5 
with the error message Alf reported when extracted from the class. More work 
for the compiler vendors! 

Daniel  

[toc] | [prev] | [next] | [standalone]

#68923

> #    if (defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__) || \
>      (defined(__BYTE_ORDER) && defined(__BIG_ENDIAN) && __BYTE_ORDER == __BIG_ENDIAN) || \
>      (defined(BYTE_ORDER) && defined(BIG_ENDIAN) && BYTE_ORDER == BIG_ENDIAN) || \
>      (defined(_BIG_ENDIAN) && !defined(_LITTLE_ENDIAN)) || (defined(__BIG_ENDIAN__) && !defined(__LITTLE_ENDIAN__)) || \
>      defined(__ARMEB__) || defined(__MIPSEB__) || defined(__s390__) || defined(__sparc__)

obfuscated preprocessor contest ...

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.c++

csiph-web