Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #150913 > unrolled thread

Protable way of reading str[4] as uint32_t

Back to article view | Back to comp.lang.c

Contents

  Protable way of reading str[4] as uint32_t wyniijj@gmail.com - 2020-03-30 11:42 -0700
    Re: Protable way of reading str[4] as uint32_t Ben Bacarisse <ben.usenet@bsb.me.uk> - 2020-03-30 20:14 +0100
      Re: Protable way of reading str[4] as uint32_t Jorgen Grahn <grahn+nntp@snipabacken.se> - 2020-03-30 20:11 +0000
    Re: Protable way of reading str[4] as uint32_t Richard Damon <Richard@Damon-Family.org> - 2020-03-30 20:04 -0400
      Re: Protable way of reading str[4] as uint32_t Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2020-03-30 19:04 -0700
        Re: Protable way of reading str[4] as uint32_t wyniijj@gmail.com - 2020-03-30 23:30 -0700
          Re: Protable way of reading str[4] as uint32_t Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2020-03-31 10:59 -0700
            Re: Protable way of reading str[4] as uint32_t Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2020-03-31 12:19 -0700
              Re: Protable way of reading str[4] as uint32_t wyniijj@gmail.com - 2020-04-01 06:37 -0700
      Re: Protable way of reading str[4] as uint32_t wyniijj@gmail.com - 2020-03-30 23:28 -0700
        Re: Portable way of reading str[4] as uint32_t Spiros Bousbouras <spibou@gmail.com> - 2020-03-31 11:29 +0000
          Re: Portable way of reading str[4] as uint32_t wyniijj@gmail.com - 2020-03-31 09:59 -0700
            Re: Portable way of reading str[4] as uint32_t Spiros Bousbouras <spibou@gmail.com> - 2020-03-31 18:03 +0000
          Re: Portable way of reading str[4] as uint32_t Tim Rentsch <tr.17687@z991.linuxsc.com> - 2020-04-02 19:09 -0700
            Re: Portable way of reading str[4] as uint32_t Spiros Bousbouras <spibou@gmail.com> - 2020-04-03 05:02 +0000
              Re: Portable way of reading str[4] as uint32_t James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-04-03 01:10 -0400
                Re: Portable way of reading str[4] as uint32_t gazelle@shell.xmission.com (Kenny McCormack) - 2020-04-03 08:18 +0000
                Re: Portable way of reading str[4] as uint32_t Spiros Bousbouras <spibou@gmail.com> - 2020-04-03 09:48 +0000
                  Re: Portable way of reading str[4] as uint32_t gazelle@shell.xmission.com (Kenny McCormack) - 2020-04-03 11:07 +0000
              Re: Portable way of reading str[4] as uint32_t BGB <cr88192@gmail.com> - 2020-04-03 09:41 -0500
                Re: Portable way of reading str[4] as uint32_t Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2020-04-03 07:51 -0700
                  Re: Portable way of reading str[4] as uint32_t Richard Damon <Richard@Damon-Family.org> - 2020-04-03 12:09 -0400
                    Re: Portable way of reading str[4] as uint32_t BGB <cr88192@gmail.com> - 2020-04-03 13:19 -0500
                      Re: Portable way of reading str[4] as uint32_t James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-04-03 14:52 -0400
                        Re: Portable way of reading str[4] as uint32_t Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2020-04-03 12:10 -0700
                          Re: Portable way of reading str[4] as uint32_t BGB <cr88192@gmail.com> - 2020-04-03 15:17 -0500
                          Re: Portable way of reading str[4] as uint32_t James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-04-03 23:39 -0400
                    Re: Portable way of reading str[4] as uint32_t Spiros Bousbouras <spibou@gmail.com> - 2020-04-03 21:28 +0000
              Re: Portable way of reading str[4] as uint32_t Tim Rentsch <tr.17687@z991.linuxsc.com> - 2020-04-04 19:28 -0700
        Re: Protable way of reading str[4] as uint32_t Richard Damon <Richard@Damon-Family.org> - 2020-03-31 08:05 -0400
          Re: Protable way of reading str[4] as uint32_t BGB <cr88192@gmail.com> - 2020-03-31 09:12 -0500
            Re: Protable way of reading str[4] as uint32_t Philipp Klaus Krause <pkk@spth.de> - 2020-03-31 17:27 +0200
              Re: Protable way of reading str[4] as uint32_t BGB <cr88192@gmail.com> - 2020-03-31 18:26 -0500
                Re: Protable way of reading str[4] as uint32_t Jorgen Grahn <grahn+nntp@snipabacken.se> - 2020-04-01 06:03 +0000
                  Re: Protable way of reading str[4] as uint32_t Richard Damon <Richard@Damon-Family.org> - 2020-04-01 08:17 -0400
                    Re: Protable way of reading str[4] as uint32_t James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-04-01 09:57 -0400
                  Re: Protable way of reading str[4] as uint32_t BGB <cr88192@gmail.com> - 2020-04-01 09:47 -0500
                    Re: Protable way of reading str[4] as uint32_t James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-04-01 11:48 -0400
                      Re: Protable way of reading str[4] as uint32_t BGB <cr88192@gmail.com> - 2020-04-01 12:37 -0500
        Re: Protable way of reading str[4] as uint32_t James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-03-31 12:56 -0400

Page 2 of 2 — ← Prev page 1 [2]

#150978 — Re: Portable way of reading str[4] as uint32_t

On Friday, 3 April 2020 15:41:20 UTC+1, BGB  wrote:
> On 4/3/2020 12:02 AM, Spiros Bousbouras wrote:
> > On Thu, 02 Apr 2020 19:09:07 -0700
> > Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
> >> Spiros Bousbouras <spibou@gmail.com> writes:
> >>> I note also that to have  uint32_t  you need
> >>> #include <stdint.h>
> >>
> >> Actually not, because <arpa/inet.h> was already #include'd.
> >> See for example the man page for htonl().
> > 
> > <arpa/inet.h>  is not part of standard C.
> > 
> 
> Yes, and <winsock.h> is better anyways... (satire)
> 
> 
> Thought: it is kind of an annoyance sometimes that there is no standard 
> way to efficiently do things like endian swapping and loading/storing 
> misaligned values (with a particular size and endianess). This stuff 
> generally ends up as a mess of boilerplate being copied from one project 
> to another...
> 
> 
> Then again, this is about like asking for "__packed" or 
> "__attribute__((packed))" to be promoted (support for packed structs not 
> exactly being free on many targets, ...).
> 
> But, as-is, one has to do things like FAT drivers and similar with 
> struct's filled with byte arrays, and then using wrapper 
> functions/macros to get/set the values for the various fields, ...
>
More broadly, C isn't good at serialisation and deserialisation. 
It's a fundamental feature that needs to be built into the language,
but it doesn't play well with different sizes of basic type, even
different numbers of bits in a byte, floating point values, undefined
endianness, padding bytes, and pointers.

[toc] | [prev] | [next] | [standalone]

#150981 — Re: Portable way of reading str[4] as uint32_t

On 4/3/20 10:51 AM, Malcolm McLean wrote:
> On Friday, 3 April 2020 15:41:20 UTC+1, BGB  wrote:
>> On 4/3/2020 12:02 AM, Spiros Bousbouras wrote:
>>> On Thu, 02 Apr 2020 19:09:07 -0700
>>> Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>>>> Spiros Bousbouras <spibou@gmail.com> writes:
>>>>> I note also that to have  uint32_t  you need
>>>>> #include <stdint.h>
>>>>
>>>> Actually not, because <arpa/inet.h> was already #include'd.
>>>> See for example the man page for htonl().
>>>
>>> <arpa/inet.h>  is not part of standard C.
>>>
>>
>> Yes, and <winsock.h> is better anyways... (satire)
>>
>>
>> Thought: it is kind of an annoyance sometimes that there is no standard 
>> way to efficiently do things like endian swapping and loading/storing 
>> misaligned values (with a particular size and endianess). This stuff 
>> generally ends up as a mess of boilerplate being copied from one project 
>> to another...
>>
>>
>> Then again, this is about like asking for "__packed" or 
>> "__attribute__((packed))" to be promoted (support for packed structs not 
>> exactly being free on many targets, ...).
>>
>> But, as-is, one has to do things like FAT drivers and similar with 
>> struct's filled with byte arrays, and then using wrapper 
>> functions/macros to get/set the values for the various fields, ...
>>
> More broadly, C isn't good at serialisation and deserialisation. 
> It's a fundamental feature that needs to be built into the language,
> but it doesn't play well with different sizes of basic type, even
> different numbers of bits in a byte, floating point values, undefined
> endianness, padding bytes, and pointers.
> 

Yes, the C language as currently defined admits too broad a class of
machines to attempt to define a portable serialization format (other
than as character strings). The issue is how does a machine with a 9 bit
byte transfer at all portably and efficiently with a machine with an 8
bit byte.


Even endianness gets tough to define.  While there are only two main
options for storing a two byte object (ignoring the real pathological
cases), either most or least significant byte first, when we get to four
byte objects there are 24 reasonably possible combinations, and that can
be reduced to 4 likely orders, all of which HAVE been used. The two
'mixed-endian' where the order of the bytes in the 16 bit word are in
the opposite order of the ordering of the two 16 bits words in the 32
bit word.

[toc] | [prev] | [next] | [standalone]

#150988 — Re: Portable way of reading str[4] as uint32_t

On 4/3/2020 11:09 AM, Richard Damon wrote:
> On 4/3/20 10:51 AM, Malcolm McLean wrote:
>> On Friday, 3 April 2020 15:41:20 UTC+1, BGB  wrote:
>>> On 4/3/2020 12:02 AM, Spiros Bousbouras wrote:
>>>> On Thu, 02 Apr 2020 19:09:07 -0700
>>>> Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>>>>> Spiros Bousbouras <spibou@gmail.com> writes:
>>>>>> I note also that to have  uint32_t  you need
>>>>>> #include <stdint.h>
>>>>>
>>>>> Actually not, because <arpa/inet.h> was already #include'd.
>>>>> See for example the man page for htonl().
>>>>
>>>> <arpa/inet.h>  is not part of standard C.
>>>>
>>>
>>> Yes, and <winsock.h> is better anyways... (satire)
>>>
>>>
>>> Thought: it is kind of an annoyance sometimes that there is no standard
>>> way to efficiently do things like endian swapping and loading/storing
>>> misaligned values (with a particular size and endianess). This stuff
>>> generally ends up as a mess of boilerplate being copied from one project
>>> to another...
>>>
>>>
>>> Then again, this is about like asking for "__packed" or
>>> "__attribute__((packed))" to be promoted (support for packed structs not
>>> exactly being free on many targets, ...).
>>>
>>> But, as-is, one has to do things like FAT drivers and similar with
>>> struct's filled with byte arrays, and then using wrapper
>>> functions/macros to get/set the values for the various fields, ...
>>>
>> More broadly, C isn't good at serialisation and deserialisation.
>> It's a fundamental feature that needs to be built into the language,
>> but it doesn't play well with different sizes of basic type, even
>> different numbers of bits in a byte, floating point values, undefined
>> endianness, padding bytes, and pointers.
>>
> 
> Yes, the C language as currently defined admits too broad a class of
> machines to attempt to define a portable serialization format (other
> than as character strings). The issue is how does a machine with a 9 bit
> byte transfer at all portably and efficiently with a machine with an 8
> bit byte.
> 

Can mostly be ignored as hardly anything anymore uses a byte size other 
than 8 bits.


> 
> Even endianness gets tough to define.  While there are only two main
> options for storing a two byte object (ignoring the real pathological
> cases), either most or least significant byte first, when we get to four
> byte objects there are 24 reasonably possible combinations, and that can
> be reduced to 4 likely orders, all of which HAVE been used. The two
> 'mixed-endian' where the order of the bytes in the 16 bit word are in
> the opposite order of the ordering of the two 16 bits words in the 32
> bit word.
> 

Or, more than 2 cases for endianess...


It would mostly just be nice if I could have something like:
   uint32_t getuint32le(void *ptr);
   void setuint32le(void *ptr, uint32_t value);

Rather than needing to supply a header with a whole bunch of compiler 
and target specific #ifdef's in order to get decent performance at these 
sorts of tasks.


Nevermind, eg, if FAT has the extra fun of some amount of its fields 
being broken up into multiple parts and needing to be glued back 
together (as a result of its FAT12 -> FAT16 -> FAT32 evolution).

More so, there is some extra hackery and bit-twiddly if one wants to 
support compatibility with a few FreeDOS extensions, or in my case there 
is also a system for crudely gluing some Unix-like metadata on top of 
FAT (while trying not to break compatibility with Windows); ...

[toc] | [prev] | [next] | [standalone]

#150991 — Re: Portable way of reading str[4] as uint32_t

On 4/3/20 2:19 PM, BGB wrote:
...
> Can mostly be ignored as hardly anything anymore uses a byte size other 
> than 8 bits.

In the embedded world, which is where a very large portion of modern C
development is done, CHAR_BIT==16 not uncommon, and such systems show no
signs of fading away.

[toc] | [prev] | [next] | [standalone]

#150992 — Re: Portable way of reading str[4] as uint32_t

James Kuyper <jameskuyper@alumni.caltech.edu> writes:
> On 4/3/20 2:19 PM, BGB wrote:
> ...
>> Can mostly be ignored as hardly anything anymore uses a byte size other 
>> than 8 bits.
>
> In the embedded world, which is where a very large portion of modern C
> development is done, CHAR_BIT==16 not uncommon, and such systems show no
> signs of fading away.

Most of the implementations I've heard of with CHAR_BIT>8 are for DSPs
(digital signal processors).  Are there systems with CHAR_BIT==16 that
*aren't* for DSPs?

-- 
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Philips Healthcare
void Void(void) { Void(); } /* The recursive call of the void */

[toc] | [prev] | [next] | [standalone]

#150995 — Re: Portable way of reading str[4] as uint32_t

On 4/3/2020 2:10 PM, Keith Thompson wrote:
> James Kuyper <jameskuyper@alumni.caltech.edu> writes:
>> On 4/3/20 2:19 PM, BGB wrote:
>> ...
>>> Can mostly be ignored as hardly anything anymore uses a byte size other
>>> than 8 bits.
>>
>> In the embedded world, which is where a very large portion of modern C
>> development is done, CHAR_BIT==16 not uncommon, and such systems show no
>> signs of fading away.
> 
> Most of the implementations I've heard of with CHAR_BIT>8 are for DSPs
> (digital signal processors).  Are there systems with CHAR_BIT==16 that
> *aren't* for DSPs?
> 

Pretty much all the normal microcontrollers I have dealt with use 8-bit 
bytes (Cortex-M, MSP430, AVR, SuperH, ...); not dealt as much with DSPs.


About the only thing I am aware of off-hand with a non-8-bit 'byte' is 
the TMS320 line. However, use-cases for this ISA are specialized enough 
that I wouldn't expect a whole lot of general purpose development to be 
done on it (and it is frequently used alongside an ARM-based 
microcontroller or similar which handles most of the "general purpose" 
tasks).

I am not entirely sure why they didn't emulate 8-bit bytes for things 
like character strings or byte-buffers and similar in C given there are 
ISA level features to facilitate doing this. Instead TI decided to do 
this with intrinsics for whatever reason.

[toc] | [prev] | [next] | [standalone]

#151002 — Re: Portable way of reading str[4] as uint32_t

On 4/3/20 3:10 PM, Keith Thompson wrote:
> James Kuyper <jameskuyper@alumni.caltech.edu> writes:
>> On 4/3/20 2:19 PM, BGB wrote:
>> ...
>>> Can mostly be ignored as hardly anything anymore uses a byte size other 
>>> than 8 bits.
>>
>> In the embedded world, which is where a very large portion of modern C
>> development is done, CHAR_BIT==16 not uncommon, and such systems show no
>> signs of fading away.
> 
> Most of the implementations I've heard of with CHAR_BIT>8 are for DSPs
> (digital signal processors).  Are there systems with CHAR_BIT==16 that
> *aren't* for DSPs?

I only know about such systems from other people, and the people who
mentioned them did say they were DSPs.

[toc] | [prev] | [next] | [standalone]

#150999 — Re: Portable way of reading str[4] as uint32_t

On Fri, 3 Apr 2020 12:09:42 -0400
Richard Damon <Richard@Damon-Family.org> wrote:
> Yes, the C language as currently defined admits too broad a class of
> machines to attempt to define a portable serialization format (other
> than as character strings). The issue is how does a machine with a 9 bit
> byte transfer at all portably and efficiently with a machine with an 8
> bit byte.

Write and read it using 2 hex digits which is fast and easy.

[toc] | [prev] | [next] | [standalone]

#151040 — Re: Portable way of reading str[4] as uint32_t

Spiros Bousbouras <spibou@gmail.com> writes:

> On Thu, 02 Apr 2020 19:09:07 -0700
> Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>
>> Spiros Bousbouras <spibou@gmail.com> writes:
>>
>>> I note also that to have  uint32_t  you need
>>> #include <stdint.h>
>>
>> Actually not, because <arpa/inet.h> was already #include'd.
>> See for example the man page for htonl().
>
> <arpa/inet.h>  is not part of standard C.

That statement notwithstanding, it is still true that the
type uint32_t may be used after a (successful) #include
of <arpa/inet.h>, without having to #include <stdint.h>

[toc] | [prev] | [next] | [standalone]

#150923

On 3/31/20 2:28 AM, wyniijj@gmail.com wrote:
> What is actually stored in str[4] is one utf8 character, unused bytes are 
> filled with zeros. I'd like to access it faster, as uint32_t.

If you want a.value to be the UCS-4 representation, then you need to
fill the str array in an order that is machine dependent.

If you are wanting internally to be processing in UCS-4, then I would
internally just use uint32_t's as your characters, and on the boundary
between UTF-8 and UCS-4 just do the conversion explicitly. It isn't that
hard, but depending on how strict you want to enforce the rules of
Unicode  (or if you want to accept a relaxed form of it) there are a few
cases you want to handle, so that probably works best as a called function.

Note, you can't just copy the UTF-8 bytes into str and get UCS-4 but
need to do an actual conversion. You could use a very nonstandard
encoding based on a word of multiple UTF-8 chars, but that will
eventually trip you up.

[toc] | [prev] | [next] | [standalone]

#150924

On 3/31/2020 7:05 AM, Richard Damon wrote:
> On 3/31/20 2:28 AM, wyniijj@gmail.com wrote:
>> What is actually stored in str[4] is one utf8 character, unused bytes are
>> filled with zeros. I'd like to access it faster, as uint32_t.
> 
> If you want a.value to be the UCS-4 representation, then you need to
> fill the str array in an order that is machine dependent.
> 
> If you are wanting internally to be processing in UCS-4, then I would
> internally just use uint32_t's as your characters, and on the boundary
> between UTF-8 and UCS-4 just do the conversion explicitly. It isn't that
> hard, but depending on how strict you want to enforce the rules of
> Unicode  (or if you want to accept a relaxed form of it) there are a few
> cases you want to handle, so that probably works best as a called function.
> 
> Note, you can't just copy the UTF-8 bytes into str and get UCS-4 but
> need to do an actual conversion. You could use a very nonstandard
> encoding based on a word of multiple UTF-8 chars, but that will
> eventually trip you up.
> 

Yeah. To get the codepoint from UTF-8, one would typically do something 
like:
uint32_t Foo_GetCharUtf8(uint8_t **rcs)
{
   uint8_t *cs;
   uint32_t i;
   cs=*rcs;
   i=*cs++;
   if(!(i&0x80))
   {
     *rcs=cs;
     return(i);
   }
   if((i&0xE0)==0xC0)
   {
     i=i&0x1F;
     i=(i<<6)|((*cs++)&0x3F);
     *rcs=cs;
     return(i);
   }
   if((i&0xF0)==0xE0)
   {
     i=i&0x0F;
     i=(i<<6)|((*cs++)&0x3F);
     i=(i<<6)|((*cs++)&0x3F);
     *rcs=cs;
     return(i);
   }
   if((i&0xF8)==0xF0)
   {
     i=i&0x07;
     i=(i<<6)|((*cs++)&0x3F);
     i=(i<<6)|((*cs++)&0x3F);
     i=(i<<6)|((*cs++)&0x3F);
     *rcs=cs;
     return(i);
   }
   ...
}


As for getting a 32-bit value from a buffer, typically something like:
   i=(cs[0])|(cs[1]<<8)|(cs[2]<<16)|(cs[3]<<24);

Which relies on implicit type promotion, and may not work correctly on 
16-bit targets (such as 8/16-bit microcontrollers), but then one may add 
a few casts:
   i=(((uint32_t)cs[0])    )|(((uint32_t)cs[1])<< 8)|
     (((uint32_t)cs[2])<<16)|(((uint32_t)cs[3])<<24);

This is generally portable, and is independent of the endianess of the 
target machine.


Reliance on a union "generally works", but:
Depends on the endianess of the machine;
Assumes that the compiler isn't being particularly aggressive about 
strict aliasing semantics (IIRC, allowing a union to be used this way is 
de-facto supported though).


Then there is always:
   memcpy(&i, cs, sizeof(uint32_t));
Which can work similarly to using a union, but whether or not it 
generates something efficient depends on the compiler (it either turns 
out "pretty good" or "really bad"; the mainstream compilers generally 
figure it out though).


And, of course, not to forget:
   i=*(uint32_t *)cs;
Which also typically works, but may not necessarily work correctly 
(depending on the particular target and/or the whims of the compiler).
This also may or may not be the fastest option.

Some compilers may also have a "__packed" attribute or similar to deal 
with this (allows misaligned access, potentially loosens aliasing 
semantics, ...), but this is a nonstandard extension.

...


Generally, this ends up as a mess of #ifdef's in a header somewhere, 
which matches up the target/compiler with the best way of doing this, 
and then falls back to a generic case if the compiler is unrecognized.

[toc] | [prev] | [next] | [standalone]

#150926

Am 31.03.20 um 16:12 schrieb BGB:
> On 3/31/2020 7:05 AM, Richard Damon wrote:
>>
> 
> Yeah. To get the codepoint from UTF-8, one would typically do something
> like:
> […]

Or use one of the functions from the standard library for such
conversions (assuming an UTF-8 locale).

Philipp

[toc] | [prev] | [next] | [standalone]

#150938

On 3/31/2020 10:27 AM, Philipp Klaus Krause wrote:
> Am 31.03.20 um 16:12 schrieb BGB:
>> On 3/31/2020 7:05 AM, Richard Damon wrote:
>>>
>>
>> Yeah. To get the codepoint from UTF-8, one would typically do something
>> like:
>> […]
> 
> Or use one of the functions from the standard library for such
> conversions (assuming an UTF-8 locale).
> 

If one gets outside of targets using a mainstream C library, support for 
things like locale and similar gets rather hit or miss.

One may find that they are dealing with a C library where this stuff is 
either hard-coded or stubbed over, and the C library doesn't natively 
support anything beyond ASCII strings.

IMO, This is one of those things that, if is needed, better to supply it 
oneself.

[toc] | [prev] | [next] | [standalone]

#150940

On Tue, 2020-03-31, BGB wrote:
> On 3/31/2020 10:27 AM, Philipp Klaus Krause wrote:
>> Am 31.03.20 um 16:12 schrieb BGB:
>>> On 3/31/2020 7:05 AM, Richard Damon wrote:
>>>>
>>>
>>> Yeah. To get the codepoint from UTF-8, one would typically do something
>>> like:
>>> […]
>> 
>> Or use one of the functions from the standard library for such
>> conversions (assuming an UTF-8 locale).
>> 
>
> If one gets outside of targets using a mainstream C library, support for 
> things like locale and similar gets rather hit or miss.

I think the bigger problem is the "assuming an UTF-8 locale" part.
If you're parsing some protocol or file format (XML, mail) you can't
botch the job just because the user didn't set an UTF-8 locale.  And
switching to an UTF-8 locale at program startup may not be acceptable,
either.

> IMO, This is one of those things that, if is needed, better to supply it 
> oneself.

Another option is to use one of the popular libraries.  libiconv?

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#150941

On 4/1/20 2:03 AM, Jorgen Grahn wrote:
> On Tue, 2020-03-31, BGB wrote:
>> On 3/31/2020 10:27 AM, Philipp Klaus Krause wrote:
>>> Am 31.03.20 um 16:12 schrieb BGB:
>>>> On 3/31/2020 7:05 AM, Richard Damon wrote:
>>>>>
>>>>
>>>> Yeah. To get the codepoint from UTF-8, one would typically do something
>>>> like:
>>>> […]
>>>
>>> Or use one of the functions from the standard library for such
>>> conversions (assuming an UTF-8 locale).
>>>
>>
>> If one gets outside of targets using a mainstream C library, support for 
>> things like locale and similar gets rather hit or miss.
> 
> I think the bigger problem is the "assuming an UTF-8 locale" part.
> If you're parsing some protocol or file format (XML, mail) you can't
> botch the job just because the user didn't set an UTF-8 locale.  And
> switching to an UTF-8 locale at program startup may not be acceptable,
> either.
> 

The biggest issue I see with assuming a UTF-8 local is that there is no
standard way to select one (and if there is one, there are probably
many, because a locale holds much more than character encoding), let
alone even assure that there is one.

Also, as you somewhat refer to, the global locale is really designed to
handle user facing I/O. When dealing with non-user-facing I/O, you
really want to use some locally imbued locale that only affects that
file, which I don't think C supports (C++ added this). This means that
when dealing with a stream that you know based on protocols (or headers
in the protocol) is UTF-8 encoded, then it is best to do that sort of
conversion explicitly. Either at the input border if not using UTF-8
internally, or the output border if you are processing in UTF-8 and need
to output in some other encoding.

>> IMO, This is one of those things that, if is needed, better to supply it 
>> oneself.
> 
> Another option is to use one of the popular libraries.  libiconv?
> 
> /Jorgen
> 

And one good way to do this explicitly is to use a library function.

[toc] | [prev] | [next] | [standalone]

#150944

On 4/1/20 8:17 AM, Richard Damon wrote:
...
> The biggest issue I see with assuming a UTF-8 local is that there is no
> standard way to select one (and if there is one, there are probably
> many, because a locale holds much more than character encoding), let
> alone even assure that there is one.

On my desktop machine, for instance, there's 61 different locales
supported, 52 of which have names that end in ".utf8", and one that ends
with ".UTF-8".

[toc] | [prev] | [next] | [standalone]

#150945

On 4/1/2020 1:03 AM, Jorgen Grahn wrote:
> On Tue, 2020-03-31, BGB wrote:
>> On 3/31/2020 10:27 AM, Philipp Klaus Krause wrote:
>>> Am 31.03.20 um 16:12 schrieb BGB:
>>>> On 3/31/2020 7:05 AM, Richard Damon wrote:
>>>>>
>>>>
>>>> Yeah. To get the codepoint from UTF-8, one would typically do something
>>>> like:
>>>> […]
>>>
>>> Or use one of the functions from the standard library for such
>>> conversions (assuming an UTF-8 locale).
>>>
>>
>> If one gets outside of targets using a mainstream C library, support for
>> things like locale and similar gets rather hit or miss.
> 
> I think the bigger problem is the "assuming an UTF-8 locale" part.
> If you're parsing some protocol or file format (XML, mail) you can't
> botch the job just because the user didn't set an UTF-8 locale.  And
> switching to an UTF-8 locale at program startup may not be acceptable,
> either.
> 

Yes, granted, but this still carries the assumption that things like 
'setlocale' actually do something, ...


Granted, stubbing over this stuff is more of a concern for 
microcontrollers than for programs running on a mainstream OS running on 
a PC or similar, which will generally also have a fully-functioning C 
library.



>> IMO, This is one of those things that, if is needed, better to supply it
>> oneself.
> 
> Another option is to use one of the popular libraries.  libiconv?
> 

Something like "libiconv" is probably a sensible option if targeting 
something "Unix-like".


I have generally been in the practice of avoiding using anything from 
3rd party libraries, as this makes portability more of an issue.

Adding maybe a few hundred lines to a program is generally less annoying 
IMO than needing to deal with a 3rd party library which may or may not 
exist on the range of targets I might want to run the code on.


Within the C library, one may take care as to which parts are used, as 
things can still get a bit chaotic here depending on the target.

Similarly "kind of annoying" on this front is the whole "C transpiled to 
WASM or JS and running in a web-browser" thing, ...


If the code is basically a more-or-less self-contained fortress, one can 
also port it to a bare-metal target or similar without too much issue 
(eg: the whole "can it run Doom?" thing; this would not be viable if 
Doom depended on a bunch of 3rd party libraries).

[toc] | [prev] | [next] | [standalone]

#150946

On 4/1/20 10:47 AM, BGB wrote:
...
> Yes, granted, but this still carries the assumption that things like 
> 'setlocale' actually do something, ...

A minimally conforming implementation of setlocale() must "do
something". It must return a string representing the current locale if
passed an argument of "", "C" or a null pointer, and can return a null
pointer when given any other argument. If the implementation defines
that the native environment is the "C" locale, then anytime it returns a
non-null pointer it can always return the same pointer, one that points
at "C", and there's no need for setlocale() to actually change anything,
no matter what argument it's given. Thus, a minimally conforming
implementation of setlocale() can be extremely minimal, but it can't do
nothing.

[toc] | [prev] | [next] | [standalone]

#150947

On 4/1/2020 10:48 AM, James Kuyper wrote:
> On 4/1/20 10:47 AM, BGB wrote:
> ...
>> Yes, granted, but this still carries the assumption that things like
>> 'setlocale' actually do something, ...
> 
> A minimally conforming implementation of setlocale() must "do
> something". It must return a string representing the current locale if
> passed an argument of "", "C" or a null pointer, and can return a null
> pointer when given any other argument. If the implementation defines
> that the native environment is the "C" locale, then anytime it returns a
> non-null pointer it can always return the same pointer, one that points
> at "C", and there's no need for setlocale() to actually change anything,
> no matter what argument it's given. Thus, a minimally conforming
> implementation of setlocale() can be extremely minimal, but it can't do
> nothing.
> 

Fair enough...

Looks at the code for the C library I am using in my projects:

char *setlocale(int category, const char *locale)
{
     (void)category;
     if (locale == NULL)
     {
         return ("C");
     }
     else if ((strcmp(locale, "C") == 0)
              || (strcmp(locale, "") == 0))
     {
         return ("C");
     }
     else
     {
         return (NULL);
     }
}


So, yeah, I guess it does do something at least, just not a whole lot 
that is particularly useful.

[toc] | [prev] | [next] | [standalone]

#150928

On 3/31/20 2:28 AM, wyniijj@gmail.com wrote:
> Richard Damon於 2020年3月31日星期二 UTC+8上午8時04分55秒寫道：
>> On 3/30/20 2:42 PM, wyniijj@gmail.com wrote:
>>> I'd like to read the value a.value in the following program as 0x78563412 in a 
>>> portable way, none of hton or ntoh is useful. How should I do? 
>>>
>>> #include <stdio.h>
>>> #include <arpa/inet.h>
>>>
>>> int main()
>>> {
>>>  union {
>>>    char str[4]; 
>>>    uint32_t value;
>>>  } a;
>>>
>>>  a.str[0]=0x12;
>>>  a.str[1]=0x34;
>>>  a.str[2]=0x56;
>>>  a.str[3]=0x78;
>>>
>>>  printf("host= %08x\n",a.value);
>>>  printf("hton= %08x\n",htonl(a.value));
>>>  printf("ntoh= %08x\n",ntohl(a.value));
>>>  return 0;
>>> }
>>>
>>> -------
>>> [me@localhost]$ gcc t6.c
>>> [me@localhost]$ ./a.out 
>>> host= 78563412
>>> hton= 12345678
>>> ntoh= 12345678
>>>
>>
>>
>> Network order is Big Endian, your problem statement defines decoding
>> little endian, so the network order functions won't work.
>>
>> In my mind the simplest is:
>>
>> a.ser[0] | (a.str[1] << 8) | (a.str[2] << 16) | (a.str[3] << 24)
>>
>> i.e., be explicit and access by bytes in the order you want.
>>
>> You could write some 'magic' routine that is the opposite of the network
>> order functions, which know if they can do the quick cheat because you
>> are on a little endian machine, but then if you want the code to be
>> 'portable' you need some magic to enable that short cut (some #define
>> that tells you its ok).
> 
> What is actually stored in str[4] is one utf8 character, unused bytes are 
> filled with zeros. I'd like to access it faster, as uint32_t.
> 

Are you, by any chance, trying to convert from UTF-8 to UTF-32 (aka UCS-4)?

An implementation of C is free to define the encoding used for both it's
source and execution charater sets. It is, in particular, allowed to
define an encoding which uses multiple bytes to represent an extended
character. Therefore, most of things that the C standard says about
characters are written in terms of multibyte character encodings.

On most systems where it makes sense to worry about UTF-8 encodings,
there's a pretty good chance that UTF-8 will be the encoding defined by
the implementation for the source or execution character sets; probably
both.

Character constants and string literals prefixed with "u8" produce
characters encoded in UTF-8, even if UTF-8 is not the encoding used by
the implementation for the execution character set.

The <uchar.h> header typedefs char32_t as a synonym for uint_least32_t,
which should be compatible uint32_t on any system which can support
uint32_t (such compatibility is not actually required by the standard).
If __STDC_UTF_32__ is pre#defined by the implementation, values of that
type are UTF-32 encoded.

You can create char32_t character constants and string literals
directly, by using a U prefix: U'b' is a char32_t character, and U"James
Kuyper" defines an unnamed array of type char32_t containing my name.
Such string literals may be used to initialize a char32_t array. In most
other contexts, a char32_t string literal gets implicitly converted into
a pointer to the first element of the unnamed array.

On any such system where UTF-8 is the encoding used for the execution
character set, and which pre#defined __STDC_UTF_32__, mbrtoc32() can be
used to covert from UTF-8 and UTF-32, and c32tomb() can be used to
perform the reverse conversion. These functions are declared in <uchar.h>.

The behaviors of mbrtoc32() and c32tomb() are locale-dependent. The
creation of char32_t characters from char32_t character constants and
string literals is defined in terms of calls to mbrtoc32() with an
implementation-defined locale.

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.c

csiph-web