Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #70942 > unrolled thread

Union type punning in C++

Started byMr Flibble <flibbleREMOVETHISBIT@i42.co.uk>
First post2020-01-01 19:24 +0000
Last post2020-01-13 16:29 -0800
Articles 20 on this page of 135 — 18 participants

Back to article view | Back to comp.lang.c++

Contents

  Union type punning in C++ Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> - 2020-01-01 19:24 +0000
    Re: Union type punning in C++ boltar@nowhere.org - 2020-01-02 09:36 +0000
    Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-02 03:34 -0800
      Re: Union type punning in C++ Bo Persson <bo@bo-persson.se> - 2020-01-02 12:49 +0100
        Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-02 04:52 -0800
          Re: Union type punning in C++ boltar@nowhere.org - 2020-01-02 17:22 +0000
            Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-02 10:26 -0800
              Re: Union type punning in C++ Chris Vine <chris@cvine--nospam--.freeserve.co.uk> - 2020-01-02 21:54 +0000
                Re: Union type punning in C++ boltar@nowhere.org - 2020-01-03 14:49 +0000
                  Re: Union type punning in C++ David Brown <david.brown@hesbynett.no> - 2020-01-03 17:11 +0100
                  Re: Union type punning in C++ Chris Vine <chris@cvine--nospam--.freeserve.co.uk> - 2020-01-03 17:55 +0000
              Re: Union type punning in C++ boltar@nowhere.org - 2020-01-03 09:55 +0000
                Re: Union type punning in C++ David Brown <david.brown@hesbynett.no> - 2020-01-03 11:20 +0100
                Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-03 02:21 -0800
                  Re: Union type punning in C++ boltar@nowhere.org - 2020-01-03 14:55 +0000
                    Re: Union type punning in C++ Bonita Montero <Bonita.Montero@gmail.com> - 2020-01-03 16:04 +0100
                      Re: Union type punning in C++ "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2020-01-03 14:45 -0800
                        Re: Union type punning in C++ Bonita Montero <Bonita.Montero@gmail.com> - 2020-01-04 01:26 +0100
                          Re: Union type punning in C++ "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2020-01-03 16:55 -0800
                            Re: Union type punning in C++ Melzzzzz <Melzzzzz@zzzzz.com> - 2020-01-04 01:02 +0000
                              Re: Union type punning in C++ "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2020-01-03 17:06 -0800
                    Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-03 07:34 -0800
                      Re: Union type punning in C++ boltar@nowhere.org - 2020-01-03 16:12 +0000
                        Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-03 08:49 -0800
                        Re: Union type punning in C++ Bonita Montero <Bonita.Montero@gmail.com> - 2020-01-03 19:28 +0100
                          Re: Union type punning in C++ Melzzzzz <Melzzzzz@zzzzz.com> - 2020-01-03 18:54 +0000
                            Re: Union type punning in C++ Bonita Montero <Bonita.Montero@gmail.com> - 2020-01-04 01:25 +0100
                              Re: Union type punning in C++ Melzzzzz <Melzzzzz@zzzzz.com> - 2020-01-04 00:28 +0000
                                Re: Union type punning in C++ Bonita Montero <Bonita.Montero@gmail.com> - 2020-01-04 01:41 +0100
                                  Re: Union type punning in C++ Melzzzzz <Melzzzzz@zzzzz.com> - 2020-01-04 00:52 +0000
                                    Re: Union type punning in C++ Bonita Montero <Bonita.Montero@gmail.com> - 2020-01-04 08:08 +0100
                                      Re: Union type punning in C++ Bonita Montero <Bonita.Montero@gmail.com> - 2020-01-04 08:34 +0100
                                        Re: Union type punning in C++ Bonita Montero <Bonita.Montero@gmail.com> - 2020-01-04 08:43 +0100
                                        Re: Union type punning in C++ Melzzzzz <Melzzzzz@zzzzz.com> - 2020-01-04 08:20 +0000
                                          Re: Union type punning in C++ Bonita Montero <Bonita.Montero@gmail.com> - 2020-01-04 09:25 +0100
                                      Re: Union type punning in C++ Melzzzzz <Melzzzzz@zzzzz.com> - 2020-01-04 08:25 +0000
                                        Re: Union type punning in C++ Bonita Montero <Bonita.Montero@gmail.com> - 2020-01-04 09:25 +0100
                                          Re: Union type punning in C++ Melzzzzz <Melzzzzz@zzzzz.com> - 2020-01-04 08:27 +0000
                                            Re: Union type punning in C++ Bonita Montero <Bonita.Montero@gmail.com> - 2020-01-04 09:36 +0100
                                              Re: Union type punning in C++ Melzzzzz <Melzzzzz@zzzzz.com> - 2020-01-04 08:41 +0000
                                                Re: Union type punning in C++ Bonita Montero <Bonita.Montero@gmail.com> - 2020-01-04 09:49 +0100
                                                  Re: Union type punning in C++ Melzzzzz <Melzzzzz@zzzzz.com> - 2020-01-04 09:01 +0000
                                                    Re: Union type punning in C++ Bonita Montero <Bonita.Montero@gmail.com> - 2020-01-04 10:02 +0100
                                                      Re: Union type punning in C++ Melzzzzz <Melzzzzz@zzzzz.com> - 2020-01-04 09:19 +0000
                                                        Re: Union type punning in C++ Bonita Montero <Bonita.Montero@gmail.com> - 2020-01-04 10:38 +0100
                              Re: Union type punning in C++ David Brown <david.brown@hesbynett.no> - 2020-01-04 12:45 +0100
                                Re: Union type punning in C++ Melzzzzz <Melzzzzz@zzzzz.com> - 2020-01-04 12:00 +0000
                                  Re: Union type punning in C++ Bonita Montero <Bonita.Montero@gmail.com> - 2020-01-04 13:03 +0100
                                Re: Union type punning in C++ Bonita Montero <Bonita.Montero@gmail.com> - 2020-01-04 14:22 +0100
                        Re: Union type punning in C++ "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2020-01-03 14:46 -0800
                      Re: Union type punning in C++ Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2020-01-03 10:32 -0800
                        Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-03 10:43 -0800
                          Re: Union type punning in C++ Melzzzzz <Melzzzzz@zzzzz.com> - 2020-01-03 18:55 +0000
              Re: Union type punning in C++ Daniel <danielaparker@gmail.com> - 2020-01-08 09:40 -0800
                Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-08 13:23 -0800
                  Re: Union type punning in C++ Daniel <danielaparker@gmail.com> - 2020-01-08 22:41 -0800
                  Re: Union type punning in C++ Daniel <danielaparker@gmail.com> - 2020-01-08 22:52 -0800
                    Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-09 12:59 -0800
                      Re: Union type punning in C++ Daniel <danielaparker@gmail.com> - 2020-01-09 13:32 -0800
                      Re: Union type punning in C++ Mister2U@honorific.org - 2020-01-10 12:28 +0000
                        Re: Union type punning in C++ David Brown <david.brown@hesbynett.no> - 2020-01-10 14:02 +0100
                          Re: Union type punning in C++ Mister2U@honorific.org - 2020-01-10 16:21 +0000
                            Re: Union type punning in C++ David Brown <david.brown@hesbynett.no> - 2020-01-10 19:18 +0100
                              Re: Union type punning in C++ Daniel <danielaparker@gmail.com> - 2020-01-10 15:41 -0800
                                Re: Union type punning in C++ David Brown <david.brown@hesbynett.no> - 2020-01-11 12:01 +0100
                                  Re: Union type punning in C++ Daniel <danielaparker@gmail.com> - 2020-01-11 07:05 -0800
                                    Re: Union type punning in C++ David Brown <david.brown@hesbynett.no> - 2020-01-11 23:53 +0100
                                      Re: Union type punning in C++ Mr Flibble <flibble@i42.removethisbit.co.uk> - 2020-01-12 01:41 +0000
                                        Re: Union type punning in C++ Daniel <danielaparker@gmail.com> - 2020-01-11 19:00 -0800
                                          Re: Union type punning in C++ Mr Flibble <flibble@i42.removethisbit.co.uk> - 2020-01-13 17:32 +0000
                              Re: Union type punning in C++ Mister2U@honorific.org - 2020-01-11 11:53 +0000
                                Re: Union type punning in C++ David Brown <david.brown@hesbynett.no> - 2020-01-11 13:20 +0100
                                  Re: Union type punning in C++ Mister2U@honorific.org - 2020-01-11 12:33 +0000
                                    Re: Union type punning in C++ David Brown <david.brown@hesbynett.no> - 2020-01-11 14:28 +0100
                            Re: Union type punning in C++ Jorgen Grahn <grahn+nntp@snipabacken.se> - 2020-01-11 10:48 +0000
                        Re: Union type punning in C++ Jorgen Grahn <grahn+nntp@snipabacken.se> - 2020-01-11 11:05 +0000
                          Re: Union type punning in C++ Mister2U@honorific.org - 2020-01-11 11:55 +0000
            Re: Union type punning in C++ Bo Persson <bo@bo-persson.se> - 2020-01-02 21:12 +0100
              Re: Union type punning in C++ boltar@nowhere.org - 2020-01-03 14:46 +0000
                Re: Union type punning in C++ Bo Persson <bo@bo-persson.se> - 2020-01-03 20:01 +0100
            Re: Union type punning in C++ David Brown <david.brown@hesbynett.no> - 2020-01-02 23:23 +0100
    Re: Union type punning in C++ peter koch <peter.koch.larsen@gmail.com> - 2020-01-02 12:39 -0800
    Re: Union type punning in C++ "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2020-01-02 15:39 -0800
      Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-02 16:58 -0800
        Re: Union type punning in C++ "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2020-01-02 17:55 -0800
          Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-03 00:40 -0800
            Re: Union type punning in C++ Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> - 2020-01-03 14:04 +0000
              Re: Union type punning in C++ David Brown <david.brown@hesbynett.no> - 2020-01-03 15:24 +0100
                Re: Union type punning in C++ "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2020-01-03 14:34 -0800
                  Re: Union type punning in C++ Melzzzzz <Melzzzzz@zzzzz.com> - 2020-01-03 22:59 +0000
                    Re: Union type punning in C++ Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> - 2020-01-04 00:11 +0000
                      Re: Union type punning in C++ Melzzzzz <Melzzzzz@zzzzz.com> - 2020-01-04 00:21 +0000
                        Re: Union type punning in C++ Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> - 2020-01-04 00:26 +0000
                          Re: Union type punning in C++ "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2020-01-04 16:51 -0800
                      Re: Union type punning in C++ "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2020-01-03 18:47 -0800
                        Re: Union type punning in C++ "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2020-01-03 19:07 -0800
              Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-03 06:47 -0800
                Re: Union type punning in C++ Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> - 2020-01-03 14:57 +0000
                  Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-03 07:38 -0800
                    Re: Union type punning in C++ Mr Flibble <flibbleREMOVETHISBIT@i42.co.uk> - 2020-01-03 17:06 +0000
                      Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-03 11:05 -0800
            Re: Union type punning in C++ "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2020-01-03 14:38 -0800
              Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-03 22:37 -0800
        Re: Union type punning in C++ Tim Rentsch <tr.17687@z991.linuxsc.com> - 2020-01-11 07:10 -0800
          Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-11 10:23 -0800
            Re: Union type punning in C++ Tim Rentsch <tr.17687@z991.linuxsc.com> - 2020-01-13 09:01 -0800
              Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-13 11:03 -0800
                Re: Union type punning in C++ James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-01-13 12:48 -0800
                  Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-13 14:15 -0800
                    Re: Union type punning in C++ James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-01-13 21:47 -0500
                      Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-13 23:44 -0800
                        Re: Union type punning in C++ James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-01-14 08:26 -0500
                Re: Union type punning in C++ Tim Rentsch <tr.17687@z991.linuxsc.com> - 2020-01-14 07:30 -0800
                  Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-14 09:11 -0800
                    Re: Union type punning in C++ James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-01-14 21:52 -0500
                      Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-14 23:15 -0800
                    Re: Union type punning in C++ Tim Rentsch <tr.17687@z991.linuxsc.com> - 2020-01-21 02:03 -0800
      Re: Union type punning in C++ Tim Rentsch <tr.17687@z991.linuxsc.com> - 2020-01-11 07:28 -0800
        Re: Union type punning in C++ "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2020-01-11 20:40 -0800
          Re: Union type punning in C++ James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-01-12 20:16 -0500
            Re: Union type punning in C++ "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2020-01-12 17:28 -0800
              Re: Union type punning in C++ James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-01-12 21:53 -0500
                Re: Union type punning in C++ Manfred <noname@add.invalid> - 2020-01-13 18:50 +0100
                Re: Union type punning in C++ "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2020-01-13 16:21 -0800
              Re: Union type punning in C++ Tim Rentsch <tr.17687@z991.linuxsc.com> - 2020-01-13 08:57 -0800
                Re: Union type punning in C++ "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2020-01-13 16:27 -0800
                  Re: Union type punning in C++ Tim Rentsch <tr.17687@z991.linuxsc.com> - 2020-01-14 07:05 -0800
                    Re: Union type punning in C++ "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2020-01-14 20:58 -0800
                      Re: Union type punning in C++ Tim Rentsch <tr.17687@z991.linuxsc.com> - 2020-01-21 03:47 -0800
              Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-15 01:22 -0800
                Re: Union type punning in C++ James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-01-15 08:04 -0500
                  Re: Union type punning in C++ Öö Tiib <ootiib@hot.ee> - 2020-01-15 09:38 -0800
                    Re: Union type punning in C++ James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-01-15 11:01 -0800
          Re: Union type punning in C++ Tim Rentsch <tr.17687@z991.linuxsc.com> - 2020-01-13 08:37 -0800
            Re: Union type punning in C++ "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2020-01-13 16:29 -0800

Page 4 of 7 — ← Prev page 1 2 3 [4] 5 6 7  Next page →

#71255

FromDavid Brown <david.brown@hesbynett.no>
Date2020-01-10 14:02 +0100
Message-ID<qv9slf$t4i$1@dont-email.me>
In reply to#71254
On 10/01/2020 13:28, Mister2U@honorific.org wrote:
> On Thu, 9 Jan 2020 12:59:29 -0800 (PST)
> =?UTF-8?B?w5bDtiBUaWli?= <ootiib@hot.ee> wrote:
>> On Thursday, 9 January 2020 08:52:31 UTC+2, Daniel  wrote:
>>> On Wednesday, January 8, 2020 at 4:24:03 PM UTC-5, =C3=96=C3=B6 Tiib wrot=
>> e:
>>>> =20
>>>> However when we deserialize then we typically get those bytes from
>>>> somewhere as smaller packets. For example Ethernet packet is
>>>> 1500 bytes + 26 bytes of header and there are usually other
>>>> garbage by protocol so for floats there is less than 1500 bytes.
>>>> Then we memcpy from there into our floats.=20
>>> =20
>>> But if the last item in the packet is half a float, do you still memcpy
>>> directly into floats?=20
>>
>> What you mean by "half a float"? Half-received?=20
>> If half-received then may be yes, may be no. Hard to
>> tell without context.
>>
>> For example if program needs to process received floats even
>> without all received then I probably would not copy half-floats.
>> But if it may refuse to work until all floats are received then
>> I probably would copy half but would throw after timeout
>> destroying all the received floats and not processing anyway.
> 
> These sort of problems are why any genuine network programmer always includes

This is a matter of protocol design, not programming.  Any "genuine
network programmer" writes code to handle the protocol specified -
whether it includes a packet size or not.

Protocols where the length of the telegram is given either explicitly in
a header, or implied from the protocol type, make reception simpler.
But they can involve more work when generating the data.  There is a
trade-off here, and it is not as simple as saying one method is always
best.  (Having said that, I prefer to give the length explicitly when
designing protocols.)

> the expected packet size in a stream packet (usually at the beginning) instead 
> of doing something stupid like sending raw XML or json on its own where 
> constant parsing is required to see if the end of the data has been reached yet
> and may get it wrong if the formatting is bad and continue waiting for data 
> beyond the end. For UDP where data send in a single write is returned in a 
> single read that approach is fine, not so for TCP. Sadly a new generation of 
> programmers has yet to figure this out.
> 

Some protocols on top of TCP/IP or UDP, such as MQTT, have length fields
in the telegrams.

For JSON, a very simple solution is to use newline-delimited JSON - you
send the JSON string, then a linefeed character.  It's easy to see when
you have the whole string, because you have found a character 10, and
can then start parsing.

I think it would be nice to see more use of SCTP, which combines the
benefits of TCP/IP and UDP - letting you get the "datagram" feature of
UDP while also getting the guarantees and fragmentation control of
TCP/IP.  But it is not well supported in common routers (or Windows),
making it a lot harder to establish it for common use.

[toc] | [prev] | [next] | [standalone]

#71261

FromMister2U@honorific.org
Date2020-01-10 16:21 +0000
Message-ID<qva89q$qf6$1@gioia.aioe.org>
In reply to#71255
On Fri, 10 Jan 2020 14:02:38 +0100
David Brown <david.brown@hesbynett.no> wrote:
>On 10/01/2020 13:28, Mister2U@honorific.org wrote:
>> On Thu, 9 Jan 2020 12:59:29 -0800 (PST)
>> =?UTF-8?B?w5bDtiBUaWli?= <ootiib@hot.ee> wrote:
>>> On Thursday, 9 January 2020 08:52:31 UTC+2, Daniel  wrote:
>>>> On Wednesday, January 8, 2020 at 4:24:03 PM UTC-5, =C3=96=C3=B6 Tiib wrot=
>>> e:
>>>>> =20
>>>>> However when we deserialize then we typically get those bytes from
>>>>> somewhere as smaller packets. For example Ethernet packet is
>>>>> 1500 bytes + 26 bytes of header and there are usually other
>>>>> garbage by protocol so for floats there is less than 1500 bytes.
>>>>> Then we memcpy from there into our floats.=20
>>>> =20
>>>> But if the last item in the packet is half a float, do you still memcpy
>>>> directly into floats?=20
>>>
>>> What you mean by "half a float"? Half-received?=20
>>> If half-received then may be yes, may be no. Hard to
>>> tell without context.
>>>
>>> For example if program needs to process received floats even
>>> without all received then I probably would not copy half-floats.
>>> But if it may refuse to work until all floats are received then
>>> I probably would copy half but would throw after timeout
>>> destroying all the received floats and not processing anyway.
>> 
>> These sort of problems are why any genuine network programmer always includes
>
>
>This is a matter of protocol design, not programming.  Any "genuine
>network programmer" writes code to handle the protocol specified -
>whether it includes a packet size or not.

Well yes, but I was talking about when designing one.

>> and may get it wrong if the formatting is bad and continue waiting for data 
>> beyond the end. For UDP where data send in a single write is returned in a 
>> single read that approach is fine, not so for TCP. Sadly a new generation of 
>
>> programmers has yet to figure this out.
>> 
>
>Some protocols on top of TCP/IP or UDP, such as MQTT, have length fields
>in the telegrams.
>
>For JSON, a very simple solution is to use newline-delimited JSON - you
>send the JSON string, then a linefeed character.  It's easy to see when
>you have the whole string, because you have found a character 10, and
>can then start parsing.

That'll work fine - until you have ascii 10 inside a quoted string which is 
entirely plausible. The same problem exists if doing simple curly bracket 
counting. Its a non trivial problem and shouldn't be something the network
layer has to worry about.

>I think it would be nice to see more use of SCTP, which combines the

Unfortunately SCTP is de facto dead. It was a good attempt but didn't get the
traction.

[toc] | [prev] | [next] | [standalone]

#71265

FromDavid Brown <david.brown@hesbynett.no>
Date2020-01-10 19:18 +0100
Message-ID<qvaf5n$jnr$1@dont-email.me>
In reply to#71261
On 10/01/2020 17:21, Mister2U@honorific.org wrote:
> On Fri, 10 Jan 2020 14:02:38 +0100
> David Brown <david.brown@hesbynett.no> wrote:
>> On 10/01/2020 13:28, Mister2U@honorific.org wrote:
>>> On Thu, 9 Jan 2020 12:59:29 -0800 (PST)
>>> =?UTF-8?B?w5bDtiBUaWli?= <ootiib@hot.ee> wrote:
>>>> On Thursday, 9 January 2020 08:52:31 UTC+2, Daniel  wrote:
>>>>> On Wednesday, January 8, 2020 at 4:24:03 PM UTC-5, =C3=96=C3=B6 Tiib wrot=
>>>> e:
>>>>>> =20
>>>>>> However when we deserialize then we typically get those bytes from
>>>>>> somewhere as smaller packets. For example Ethernet packet is
>>>>>> 1500 bytes + 26 bytes of header and there are usually other
>>>>>> garbage by protocol so for floats there is less than 1500 bytes.
>>>>>> Then we memcpy from there into our floats.=20
>>>>> =20
>>>>> But if the last item in the packet is half a float, do you still memcpy
>>>>> directly into floats?=20
>>>>
>>>> What you mean by "half a float"? Half-received?=20
>>>> If half-received then may be yes, may be no. Hard to
>>>> tell without context.
>>>>
>>>> For example if program needs to process received floats even
>>>> without all received then I probably would not copy half-floats.
>>>> But if it may refuse to work until all floats are received then
>>>> I probably would copy half but would throw after timeout
>>>> destroying all the received floats and not processing anyway.
>>>
>>> These sort of problems are why any genuine network programmer always includes
>>
>>
>> This is a matter of protocol design, not programming.  Any "genuine
>> network programmer" writes code to handle the protocol specified -
>> whether it includes a packet size or not.
> 
> Well yes, but I was talking about when designing one.
> 
>>> and may get it wrong if the formatting is bad and continue waiting for data 
>>> beyond the end. For UDP where data send in a single write is returned in a 
>>> single read that approach is fine, not so for TCP. Sadly a new generation of 
>>
>>> programmers has yet to figure this out.
>>>
>>
>> Some protocols on top of TCP/IP or UDP, such as MQTT, have length fields
>> in the telegrams.
>>
>> For JSON, a very simple solution is to use newline-delimited JSON - you
>> send the JSON string, then a linefeed character.  It's easy to see when
>> you have the whole string, because you have found a character 10, and
>> can then start parsing.
> 
> That'll work fine - until you have ascii 10 inside a quoted string which is 
> entirely plausible. The same problem exists if doing simple curly bracket 
> counting. Its a non trivial problem and shouldn't be something the network
> layer has to worry about.

No, it is not plausible - you can't have ASCII 10 in the JSON string.
It must be escaped as "\n".

> 
>> I think it would be nice to see more use of SCTP, which combines the
> 
> Unfortunately SCTP is de facto dead. It was a good attempt but didn't get the
> traction.
> 

Exactly.

[toc] | [prev] | [next] | [standalone]

#71271

FromDaniel <danielaparker@gmail.com>
Date2020-01-10 15:41 -0800
Message-ID<1dbd66bc-4587-4172-a5fb-b3cb7414bda7@googlegroups.com>
In reply to#71265
On Friday, January 10, 2020 at 1:18:43 PM UTC-5, David Brown wrote:

> No, it is not plausible - you can't have ASCII 10 in the JSON string.
> It must be escaped as "\n".
> 
ASCII 10 in JSON quoted string values must be escaped as "\n", but 
the JSON text may otherwise contain unescaped white space characters, including 
ASCII 10.

Daniel

[toc] | [prev] | [next] | [standalone]

#71280

FromDavid Brown <david.brown@hesbynett.no>
Date2020-01-11 12:01 +0100
Message-ID<qvc9u5$5io$1@dont-email.me>
In reply to#71271
On 11/01/2020 00:41, Daniel wrote:
> On Friday, January 10, 2020 at 1:18:43 PM UTC-5, David Brown wrote:
> 
>> No, it is not plausible - you can't have ASCII 10 in the JSON string.
>> It must be escaped as "\n".
>>
> ASCII 10 in JSON quoted string values must be escaped as "\n", but
> the JSON text may otherwise contain unescaped white space characters, including
> ASCII 10.
> 

I checked again, and you are entirely correct - any of the four standard 
white characters are allowed (and ignored) outside of strings in JSON.

However, in newline-delimited JSON, which I was talking about, you can't 
have ASCII 10 or ASCII 13 in the encoding.  (Use \n and \r in strings.) 
This is precisely so that it is suitable for streamed packets, and it 
lets you find the end each message simply by scanning for ASCII 10. 
(Strings are in UTF-8, and byte 10 would not be a valid part of a UTF-8 
string except as a LF character.)

So I would have been correct if I had stuck to referring to 
line-delimited or newline-delimited JSON.  But I was wrong when I 
referred to general JSON.  Sorry for the mix-up, and thank you for the 
correction.

[toc] | [prev] | [next] | [standalone]

#71289

FromDaniel <danielaparker@gmail.com>
Date2020-01-11 07:05 -0800
Message-ID<65ffd879-b67d-488e-9905-ae58b3571303@googlegroups.com>
In reply to#71280
On Saturday, January 11, 2020 at 6:01:35 AM UTC-5, David Brown wrote:
> 
> So I would have been correct if I had stuck to referring to 
> line-delimited or newline-delimited JSON. 

But what would you have been referring to? This one?

https://github.com/ndjson/ndjson-spec

For interoperability it's preferred to refer to an Internet Standards 
Document, but it doesn't seem an RFC exists yet for this one. 

Daniel

[toc] | [prev] | [next] | [standalone]

#71295

FromDavid Brown <david.brown@hesbynett.no>
Date2020-01-11 23:53 +0100
Message-ID<qvdjl9$5oj$1@dont-email.me>
In reply to#71289
On 11/01/2020 16:05, Daniel wrote:
> On Saturday, January 11, 2020 at 6:01:35 AM UTC-5, David Brown wrote:
>>
>> So I would have been correct if I had stuck to referring to
>> line-delimited or newline-delimited JSON.
> 
> But what would you have been referring to? This one?
> 
> https://github.com/ndjson/ndjson-spec

That one would be fine.  But it is such a simple extension or addition 
to JSON that it can be described in a paragraph.

> 
> For interoperability it's preferred to refer to an Internet Standards
> Document, but it doesn't seem an RFC exists yet for this one.
> 

Yes, it would be nice to see it incorporated as part of the JSON 
standard.  (It is far too simple for the bureaucracy of its own 
standard, IMHO.)

> Daniel
>

[toc] | [prev] | [next] | [standalone]

#71298

FromMr Flibble <flibble@i42.removethisbit.co.uk>
Date2020-01-12 01:41 +0000
Message-ID<8PuSF.147473$ub6.88893@fx04.dcg>
In reply to#71295
On 11/01/2020 22:53, David Brown wrote:
> On 11/01/2020 16:05, Daniel wrote:
>> On Saturday, January 11, 2020 at 6:01:35 AM UTC-5, David Brown wrote:
>>>
>>> So I would have been correct if I had stuck to referring to
>>> line-delimited or newline-delimited JSON.
>>
>> But what would you have been referring to? This one?
>>
>> https://github.com/ndjson/ndjson-spec
> 
> That one would be fine.  But it is such a simple extension or addition 
> to JSON that it can be described in a paragraph.
> 
>>
>> For interoperability it's preferred to refer to an Internet Standards
>> Document, but it doesn't seem an RFC exists yet for this one.
>>
> 
> Yes, it would be nice to see it incorporated as part of the JSON 
> standard.  (It is far too simple for the bureaucracy of its own 
> standard, IMHO.)
> 
>> Daniel
>>

I prefer RJSON (Relaxed JSON).

/Flibble

-- 
“You won’t burn in hell. But be nice anyway.” – Ricky Gervais
“I see Atheists are fighting and killing each other again, over who 
doesn’t believe in any God the most. Oh, no..wait.. that never happens.” 
– Ricky Gervais
"Suppose it's all true, and you walk up to the pearly gates, and are 
confronted by God," Bryne asked on his show The Meaning of Life. "What 
will Stephen Fry say to him, her, or it?"
"I'd say, bone cancer in children? What's that about?" Fry replied.
"How dare you? How dare you create a world to which there is such misery 
that is not our fault. It's not right, it's utterly, utterly evil."
"Why should I respect a capricious, mean-minded, stupid God who creates 
a world that is so full of injustice and pain. That's what I would say."

[toc] | [prev] | [next] | [standalone]

#71299

FromDaniel <danielaparker@gmail.com>
Date2020-01-11 19:00 -0800
Message-ID<eda8ea44-4bf5-4b90-b2ef-cdf1388f5f8b@googlegroups.com>
In reply to#71298
On Saturday, January 11, 2020 at 8:41:43 PM UTC-5, Mr Flibble wrote:
> 
> I prefer RJSON (Relaxed JSON).
> 
There are a lot of variations on the theme of "relaxed JSON" or "Json for humans": json5 (https://json5.org/), hjson (https://github.com/hjson/hjson), jaxn (https://github.com/stand-art/jaxn/blob/master/Specification.md), and rjson (http://www.relaxedjson.org/). For configuration files, not for interoperability. But much as I'd like to avoid typing the quote marks for keys in configuration files, I think I'll stick to plain interoperable JSON.

Daniel

[toc] | [prev] | [next] | [standalone]

#71308

FromMr Flibble <flibble@i42.removethisbit.co.uk>
Date2020-01-13 17:32 +0000
Message-ID<tQ1TF.2$ha7.0@fx12.dcg>
In reply to#71299
On 12/01/2020 03:00, Daniel wrote:
> On Saturday, January 11, 2020 at 8:41:43 PM UTC-5, Mr Flibble wrote:
>>
>> I prefer RJSON (Relaxed JSON).
>>
> There are a lot of variations on the theme of "relaxed JSON" or "Json for humans": json5 (https://json5.org/), hjson (https://github.com/hjson/hjson), jaxn (https://github.com/stand-art/jaxn/blob/master/Specification.md), and rjson (http://www.relaxedjson.org/). For configuration files, not for interoperability. But much as I'd like to avoid typing the quote marks for keys in configuration files, I think I'll stick to plain interoperable JSON.

What do you mean by interop? Only allow the common denominator?

/Flibble

-- 
“You won’t burn in hell. But be nice anyway.” – Ricky Gervais
“I see Atheists are fighting and killing each other again, over who 
doesn’t believe in any God the most. Oh, no..wait.. that never happens.” 
– Ricky Gervais
"Suppose it's all true, and you walk up to the pearly gates, and are 
confronted by God," Bryne asked on his show The Meaning of Life. "What 
will Stephen Fry say to him, her, or it?"
"I'd say, bone cancer in children? What's that about?" Fry replied.
"How dare you? How dare you create a world to which there is such misery 
that is not our fault. It's not right, it's utterly, utterly evil."
"Why should I respect a capricious, mean-minded, stupid God who creates 
a world that is so full of injustice and pain. That's what I would say."

[toc] | [prev] | [next] | [standalone]

#71282

FromMister2U@honorific.org
Date2020-01-11 11:53 +0000
Message-ID<qvcd02$74p$1@gioia.aioe.org>
In reply to#71265
On Fri, 10 Jan 2020 19:18:31 +0100
David Brown <david.brown@hesbynett.no> wrote:
>On 10/01/2020 17:21, Mister2U@honorific.org wrote:
>> On Fri, 10 Jan 2020 14:02:38 +0100
>> That'll work fine - until you have ascii 10 inside a quoted string which is 
>> entirely plausible. The same problem exists if doing simple curly bracket 
>> counting. Its a non trivial problem and shouldn't be something the network
>> layer has to worry about.
>
>No, it is not plausible - you can't have ASCII 10 in the JSON string.
>It must be escaped as "\n".

I'll take your word for that, but I've definately seen json strings with raw
newlines in. Obviously not following the json spec.

[toc] | [prev] | [next] | [standalone]

#71285

FromDavid Brown <david.brown@hesbynett.no>
Date2020-01-11 13:20 +0100
Message-ID<qvcej9$ver$1@dont-email.me>
In reply to#71282
On 11/01/2020 12:53, Mister2U@honorific.org wrote:
> On Fri, 10 Jan 2020 19:18:31 +0100
> David Brown <david.brown@hesbynett.no> wrote:
>> On 10/01/2020 17:21, Mister2U@honorific.org wrote:
>>> On Fri, 10 Jan 2020 14:02:38 +0100
>>> That'll work fine - until you have ascii 10 inside a quoted string which is
>>> entirely plausible. The same problem exists if doing simple curly bracket
>>> counting. Its a non trivial problem and shouldn't be something the network
>>> layer has to worry about.
>>
>> No, it is not plausible - you can't have ASCII 10 in the JSON string.
>> It must be escaped as "\n".
> 
> I'll take your word for that, but I've definately seen json strings with raw
> newlines in. Obviously not following the json spec.
> 

See Daniel's post, and my reply to that.  ASCII 10 (and 13) are allowed 
in JSON as white space, which is ignore outside strings.  But 
specifically for line-delimited or newline-delimited JSON, where you 
have a newline character after the JSON string, you are /not/ allowed 
ASCII 10 or ASCII 13 inside the JSON string object.

[toc] | [prev] | [next] | [standalone]

#71286

FromMister2U@honorific.org
Date2020-01-11 12:33 +0000
Message-ID<qvcfa4$haj$1@gioia.aioe.org>
In reply to#71285
On Sat, 11 Jan 2020 13:20:56 +0100
David Brown <david.brown@hesbynett.no> wrote:
>On 11/01/2020 12:53, Mister2U@honorific.org wrote:
>> On Fri, 10 Jan 2020 19:18:31 +0100
>> David Brown <david.brown@hesbynett.no> wrote:
>>> On 10/01/2020 17:21, Mister2U@honorific.org wrote:
>>>> On Fri, 10 Jan 2020 14:02:38 +0100
>>>> That'll work fine - until you have ascii 10 inside a quoted string which is
>
>>>> entirely plausible. The same problem exists if doing simple curly bracket
>>>> counting. Its a non trivial problem and shouldn't be something the network
>>>> layer has to worry about.
>>>
>>> No, it is not plausible - you can't have ASCII 10 in the JSON string.
>>> It must be escaped as "\n".
>> 
>> I'll take your word for that, but I've definately seen json strings with raw
>> newlines in. Obviously not following the json spec.
>> 
>
>See Daniel's post, and my reply to that.  ASCII 10 (and 13) are allowed 
>in JSON as white space, which is ignore outside strings.  But 
>specifically for line-delimited or newline-delimited JSON, where you 
>have a newline character after the JSON string, you are /not/ allowed 
>ASCII 10 or ASCII 13 inside the JSON string object.

A rule which will almost certainly be forgotten about or not tested for by
someone at some point. For network messages the data should never be its own
delimiter. IMO anyway.

[toc] | [prev] | [next] | [standalone]

#71287

FromDavid Brown <david.brown@hesbynett.no>
Date2020-01-11 14:28 +0100
Message-ID<qvciie$l6g$1@dont-email.me>
In reply to#71286
On 11/01/2020 13:33, Mister2U@honorific.org wrote:
> On Sat, 11 Jan 2020 13:20:56 +0100
> David Brown <david.brown@hesbynett.no> wrote:
>> On 11/01/2020 12:53, Mister2U@honorific.org wrote:
>>> On Fri, 10 Jan 2020 19:18:31 +0100
>>> David Brown <david.brown@hesbynett.no> wrote:
>>>> On 10/01/2020 17:21, Mister2U@honorific.org wrote:
>>>>> On Fri, 10 Jan 2020 14:02:38 +0100
>>>>> That'll work fine - until you have ascii 10 inside a quoted string which is
>>
>>>>> entirely plausible. The same problem exists if doing simple curly bracket
>>>>> counting. Its a non trivial problem and shouldn't be something the network
>>>>> layer has to worry about.
>>>>
>>>> No, it is not plausible - you can't have ASCII 10 in the JSON string.
>>>> It must be escaped as "\n".
>>>
>>> I'll take your word for that, but I've definately seen json strings with raw
>>> newlines in. Obviously not following the json spec.
>>>
>>
>> See Daniel's post, and my reply to that.  ASCII 10 (and 13) are allowed
>> in JSON as white space, which is ignore outside strings.  But
>> specifically for line-delimited or newline-delimited JSON, where you
>> have a newline character after the JSON string, you are /not/ allowed
>> ASCII 10 or ASCII 13 inside the JSON string object.
> 
> A rule which will almost certainly be forgotten about or not tested for by
> someone at some point. For network messages the data should never be its own
> delimiter. IMO anyway.
> 

In line-delimited JSON, the delimiter is not part of the data - so it 
fulfils your requirement.

Most JSON libraries generate strings without any extra white space 
unless you specifically ask for it (which can be useful for human 
readability).

And for any communication with any protocol, you have to be sure you 
implement the protocol correctly and test appropriately.  I can't see 
why line-delimited JSON would be any different.

[toc] | [prev] | [next] | [standalone]

#71279

FromJorgen Grahn <grahn+nntp@snipabacken.se>
Date2020-01-11 10:48 +0000
Message-ID<slrnr1ja00.9in.grahn+nntp@frailea.sa.invalid>
In reply to#71261
On Fri, 2020-01-10, Mister2U@honorific.org wrote:
> On Fri, 10 Jan 2020 14:02:38 +0100
> David Brown <david.brown@hesbynett.no> wrote:
...
>>I think it would be nice to see more use of SCTP, which combines the
>
> Unfortunately SCTP is de facto dead. It was a good attempt but didn't get the
> traction.

I believe SCTP has its entrenched niche uses, in at least telecom,
replacing older horrors.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#71281

FromJorgen Grahn <grahn+nntp@snipabacken.se>
Date2020-01-11 11:05 +0000
Message-ID<slrnr1jb0d.9in.grahn+nntp@frailea.sa.invalid>
In reply to#71254
On Fri, 2020-01-10, Mister2U@honorific.org wrote:
> On Thu, 9 Jan 2020 12:59:29 -0800 (PST)
> =?UTF-8?B?w5bDtiBUaWli?= <ootiib@hot.ee> wrote:
...
>>For example if program needs to process received floats even
>>without all received then I probably would not copy half-floats.
>>But if it may refuse to work until all floats are received then
>>I probably would copy half but would throw after timeout
>>destroying all the received floats and not processing anyway.
>
> These sort of problems are why any genuine network programmer always includes
> the expected packet size in a stream packet (usually at the beginning) instead 
> of doing something stupid like sending raw XML or json on its own where 
> constant parsing is required to see if the end of the data has been reached yet

I kind of disagree.  Having an end marker is a perfectly good way to
split a stream into messages.  SMTP is one of many traditional
protocols which does it this way.

  I agree it's not good to have a weakly defined end-of-message, or
  one that's expensive to find.

Sending the message size before the message typically means you have
to generate the message and buffer it locally before you can begin to
transmit it -- which is inefficient for large messages, or messages
which are expensive to generate.

> and may get it wrong if the formatting is bad and continue waiting for data 
> beyond the end.

That's IMO part of normal robustness, which any "genuine" network
programmer needs to implement.  (Part of it involves giving up after a
timeout: if a message has been partly received and nothing more
happens after a few seconds, it's probably never going to happen.)

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#71283

FromMister2U@honorific.org
Date2020-01-11 11:55 +0000
Message-ID<qvcd3m$7kd$1@gioia.aioe.org>
In reply to#71281
On 11 Jan 2020 11:05:49 GMT
Jorgen Grahn <grahn+nntp@snipabacken.se> wrote:
>On Fri, 2020-01-10, Mister2U@honorific.org wrote:
>> On Thu, 9 Jan 2020 12:59:29 -0800 (PST)
>> and may get it wrong if the formatting is bad and continue waiting for data 
>> beyond the end.
>
>That's IMO part of normal robustness, which any "genuine" network
>programmer needs to implement.  (Part of it involves giving up after a
>timeout: if a message has been partly received and nothing more
>happens after a few seconds, it's probably never going to happen.)

Partial reception is the least of your worries - reading part of the next
message by accident is far more of an issue. All this goes away with a length
field which is why all sane network and application layer protocols use one.

[toc] | [prev] | [next] | [standalone]

#70980

FromBo Persson <bo@bo-persson.se>
Date2020-01-02 21:12 +0100
Message-ID<h7715nFn06iU1@mid.individual.net>
In reply to#70974
On 2020-01-02 at 18:22, boltar@nowhere.org wrote:
> On Thu, 2 Jan 2020 04:52:42 -0800 (PST)
> =?UTF-8?B?w5bDtiBUaWli?= <ootiib@hot.ee> wrote:
>> On Thursday, 2 January 2020 13:49:39 UTC+2, Bo Persson  wrote:
>>>
>>> The C++ committee seems unwilling to change the union rules and so
>>> instead adds a bit_cast function to package the memcpy usage with a
>>> slightly better name:
>>>
>>> template< class To, class From >
>>> constexpr To bit_cast(const From& from) noexcept;
>>>
>>> https://en.cppreference.com/w/cpp/numeric/bit_cast
>>
>> Thanks, I didn't somehow even notice that. So we can throw away
>> our own type punning wrappers around memcpy in C++20.
> 
> You realise using a union (or a cast) in theory uses very few CPU cycles to do
> type punning, whereas memcpy has the overhead of a function call + loop. Its
> an extremely inefficient way to do it if you don't need the new type to be in
> a seperate variable.
> 

Obviously the bit_cast function will be inlined, as will the 
__builtin_memcpy (or whatever it is called for a specific compiler).

And sizeof(To) and sizeof(From) are compile time constants, so no loop 
needed to copy, for example, 4 or 8 bytes. Instead we can expect a 
single register mov.


I have my favorite example from a std::string constructor using both 
strlen and memcpy to construct the string. And the compiler optimizes 
this to 1 and 2 mov-instructions, respectively:

    std::string whatever = "abcdefgh";
000000013FCC30C3  mov         rdx,qword ptr [string "abcdefgh"]
000000013FCC30CA  mov         qword ptr [whatever],rdx
000000013FCC30D2  mov         byte ptr [rsp+347h],0
000000013FCC30DA  mov         qword ptr [rsp+348h],8
000000013FCC30E6  mov         byte ptr [rsp+338h],0

"Here traits_type::copy contains a call to memcpy, which is optimized 
into a single register copy of the whole string (carefully selected to 
fit). The compiler also transforms a call to strlen into a compile time 8."

https://stackoverflow.com/a/11639305

Note that this example is from 8 years ago. Compilers haven't exactly 
become less optimizing since then.


    Bo Persson

[toc] | [prev] | [next] | [standalone]

#71015

Fromboltar@nowhere.org
Date2020-01-03 14:46 +0000
Message-ID<qunk3t$g1k$1@gioia.aioe.org>
In reply to#70980
On Thu, 2 Jan 2020 21:12:40 +0100
Bo Persson <bo@bo-persson.se> wrote:
>"Here traits_type::copy contains a call to memcpy, which is optimized 
>into a single register copy of the whole string (carefully selected to 
>fit). The compiler also transforms a call to strlen into a compile time 8."

Great. And if the string is say 150 bytes long?

[toc] | [prev] | [next] | [standalone]

#71045

FromBo Persson <bo@bo-persson.se>
Date2020-01-03 20:01 +0100
Message-ID<h79hbrF8einU1@mid.individual.net>
In reply to#71015
On 2020-01-03 at 15:46, boltar@nowhere.org wrote:
> On Thu, 2 Jan 2020 21:12:40 +0100
> Bo Persson <bo@bo-persson.se> wrote:
>> "Here traits_type::copy contains a call to memcpy, which is optimized
>> into a single register copy of the whole string (carefully selected to
>> fit). The compiler also transforms a call to strlen into a compile time 8."
> 
> Great. And if the string is say 150 bytes long?
> 

So you put a 150 byte string in a union? And then access it as some 
other type? And there is no loop?!


You stated that using memcpy is bad because it *always* includes a 
function call and a loop. I showed you one case where it doesn't. That 
takes away the *always* from the argument.

Wanna guess how the compiler handles other small types, like int, long, 
float, or double?


     Bo Persson

[toc] | [prev] | [next] | [standalone]

Page 4 of 7 — ← Prev page 1 2 3 [4] 5 6 7  Next page →

Back to top | Article view | comp.lang.c++

csiph-web