Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #164369 > unrolled thread

Re: UTF-8 strings in C

Back to article view | Back to comp.lang.c

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: UTF-8 strings in C Bonita Montero <Bonita.Montero@gmail.com> - 2022-01-10 07:17 +0100
    Re: UTF-8 strings in C Mateusz Viste <mateusz@xyz.invalid> - 2022-01-10 09:32 +0100
      Re: UTF-8 strings in C Bonita Montero <Bonita.Montero@gmail.com> - 2022-01-10 11:26 +0100
        Re: UTF-8 strings in C Bonita Montero <Bonita.Montero@gmail.com> - 2022-01-10 11:55 +0100
          Re: UTF-8 strings in C Bonita Montero <Bonita.Montero@gmail.com> - 2022-01-10 14:38 +0100
            Re: UTF-8 strings in C om@iki.fi (Otto J. Makela) - 2022-01-14 14:26 +0200
              Re: UTF-8 strings in C om@iki.fi (Otto J. Makela) - 2022-01-14 16:07 +0200
                Re: UTF-8 strings in C Bonita Montero <Bonita.Montero@gmail.com> - 2022-01-14 15:42 +0100
                Re: UTF-8 strings in C Philipp Klaus Krause <pkk@spth.de> - 2022-01-15 09:08 +0100
                  Re: UTF-8 strings in C om@iki.fi (Otto J. Makela) - 2022-01-22 11:44 +0200
                    Re: UTF-8 strings in C Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2022-01-22 04:48 -0800
                      Re: UTF-8 strings in C Richard Damon <Richard@Damon-Family.org> - 2022-01-22 08:27 -0500
                    Re: UTF-8 strings in C James Kuyper <jameskuyper@alumni.caltech.edu> - 2022-01-22 14:25 -0500
                      Re: UTF-8 strings in C om@iki.fi (Otto J. Makela) - 2022-01-27 13:32 +0200
                    Re: UTF-8 strings in C om@iki.fi (Otto J. Makela) - 2022-01-27 13:33 +0200
                      Re: UTF-8 strings in C James Kuyper <jameskuyper@alumni.caltech.edu> - 2022-01-27 11:08 -0500
                        Re: UTF-8 strings in C Tim Rentsch <tr.17687@z991.linuxsc.com> - 2022-02-03 22:23 -0800
            Re: UTF-8 strings in C Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2022-01-14 04:41 -0800
              Re: UTF-8 strings in C Bonita Montero <Bonita.Montero@gmail.com> - 2022-01-14 14:43 +0100
                Re: UTF-8 strings in C Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2022-01-14 05:45 -0800
                  Re: UTF-8 strings in C Mateusz Viste <mateusz@xyz.invalid> - 2022-01-14 15:10 +0100
                    Re: UTF-8 strings in C Bonita Montero <Bonita.Montero@gmail.com> - 2022-01-14 15:43 +0100
                      Re: UTF-8 strings in C Bart <bc@freeuk.com> - 2022-01-14 15:50 +0000
                        Re: UTF-8 strings in C Bonita Montero <Bonita.Montero@gmail.com> - 2022-01-14 17:50 +0100
                      Re: UTF-8 strings in C Mateusz Viste <mateusz@xyz.invalid> - 2022-01-14 17:17 +0100
                        Re: UTF-8 strings in C Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2022-01-14 08:34 -0800
                          Re: UTF-8 strings in C Mateusz Viste <mateusz@xyz.invalid> - 2022-01-14 17:42 +0100
                        Re: UTF-8 strings in C Bonita Montero <Bonita.Montero@gmail.com> - 2022-01-14 17:51 +0100
                          Re: UTF-8 strings in C Mateusz Viste <mateusz@xyz.invalid> - 2022-01-14 18:07 +0100
                            Re: UTF-8 strings in C Bonita Montero <Bonita.Montero@gmail.com> - 2022-01-14 18:19 +0100
                        Re: UTF-8 strings in C Tim Rentsch <tr.17687@z991.linuxsc.com> - 2022-02-07 09:02 -0800

Page 1 of 2  [1] 2  Next page →

#164369 — Re: UTF-8 strings in C

Then write your own UTF-8 strlen().

[toc] | [next] | [standalone]

#164374

2022-01-10 at 07:17 +0100, Bonita Montero wrote:
> Then write your own UTF-8 strlen().

That wouldn't be CPU efficient. At this point it might be saner to
implement a special utf-8 string type that keeps the string's
render-width in a separate field.

Mateusz

[toc] | [prev] | [next] | [standalone]

#164377

Am 10.01.2022 um 09:32 schrieb Mateusz Viste:
> 2022-01-10 at 07:17 +0100, Bonita Montero wrote:
>> Then write your own UTF-8 strlen().
> 
> That wouldn't be CPU efficient. ...

I'll bet it woudln't be relevant slower than strlen()
because the memory-latency here is the limit.

[toc] | [prev] | [next] | [standalone]

#164378

Am 10.01.2022 um 11:26 schrieb Bonita Montero:
> Am 10.01.2022 um 09:32 schrieb Mateusz Viste:
>> 2022-01-10 at 07:17 +0100, Bonita Montero wrote:
>>> Then write your own UTF-8 strlen().
>>
>> That wouldn't be CPU efficient. ...

Ok, it would be slower, but I think this is the fastest possible
implementation (C++20):

size_t utf8Strlen( char const *str )
{
	struct encode_t
	{
		size_t lenIncr, strIncr;
	};
	static encode_t const encodes[] =
	{
		{ 1, 1 },
		{ 0, 0 },
		{ 1, 2 },
		{ 1, 3 },
		{ 1, 4 },
		{ 0, 0 },
		{ 0, 0 },
		{ 0, 0 },
		{ 0, 0 }
	};
	size_t len = 0;
	for( unsigned char c; (c = *str); )
	{
		encode_t const &enc = encodes[(size_t)countl_zero<unsigned char>( ~c )];
		if( !enc.lenIncr ) [[unlikely]]
			return -1;
		len += enc.lenIncr;
		str += enc.strIncr;
	}
	return len;
}

[toc] | [prev] | [next] | [standalone]

#164386

Am 10.01.2022 um 11:55 schrieb Bonita Montero:
> Am 10.01.2022 um 11:26 schrieb Bonita Montero:
>> Am 10.01.2022 um 09:32 schrieb Mateusz Viste:
>>> 2022-01-10 at 07:17 +0100, Bonita Montero wrote:
>>>> Then write your own UTF-8 strlen().
>>>
>>> That wouldn't be CPU efficient. ...
> 
> Ok, it would be slower, but I think this is the fastest possible
> implementation (C++20):

This is more correct since it completely detects encoding-errors:

size_t utf8Strlen( char const *str )
{
     struct encode_t { size_t lenIncr, strIncr; };
     static encode_t const encodes[] =
     {
         { 1, 1 },
         { 0, 0 },
         { 1, 2 },
         { 1, 3 },
         { 1, 4 },
         { 0, 0 },
         { 0, 0 },
         { 0, 0 },
         { 0, 0 }
     };
     size_t len = 0;
     for( unsigned char c; (c = *str); )
     {
         encode_t const &enc = encodes[(size_t)countl_zero<unsigned 
char>( ~c )];
         if( !enc.lenIncr ) [[unlikely]]
             return -1;
         len += enc.lenIncr;
         for( char const *cpEnd = str + enc.strIncr; ++str != cpEnd; )
             if( ((unsigned char)*str & 0x0C0) != 0x080 ) [[unlikely]]
                 return -1;
     }
     return len;
}

[toc] | [prev] | [next] | [standalone]

#164400

Bonita Montero <Bonita.Montero@gmail.com> wrote:

> This is more correct since it completely detects encoding-errors:
>
> size_t utf8Strlen( char const *str )
> {
>     struct encode_t { size_t lenIncr, strIncr; };
>     static encode_t const encodes[] =
>     {
>         { 1, 1 },
>         { 0, 0 },
>         { 1, 2 },
>         { 1, 3 },
>         { 1, 4 },
>         { 0, 0 },
>         { 0, 0 },
>         { 0, 0 },
>         { 0, 0 }
>     };
>     size_t len = 0;
>     for( unsigned char c; (c = *str); )
>     {
>         encode_t const &enc = encodes[(size_t)countl_zero<unsigned
>         char>( ~c )];
>         if( !enc.lenIncr ) [[unlikely]]
>             return -1;
>         len += enc.lenIncr;
>         for( char const *cpEnd = str + enc.strIncr; ++str != cpEnd; )
>             if( ((unsigned char)*str & 0x0C0) != 0x080 ) [[unlikely]]
>                 return -1;
>     }
>     return len;
> }

Took me a while to realize the countl_zero stuff was a C++ library call,
with the accompanying syntax.

However, this is comp.lang.c, so C solutions are appreciated.
-- 
   /* * * Otto J. Makela <om@iki.fi> * * * * * * * * * */
  /* Phone: +358 40 765 5772, ICBM: N 60 10' E 24 55' */
 /* Mail: Mechelininkatu 26 B 27,  FI-00100 Helsinki */
/* * * Computers Rule 01001111 01001011 * * * * * * */

[toc] | [prev] | [next] | [standalone]

#164404

om@iki.fi (Otto J. Makela) wrote:

> Took me a while to realize the countl_zero stuff was a C++ library
> call, with the accompanying syntax.
>
> However, this is comp.lang.c, so C solutions are appreciated.

How about this? It of course does not count the slightly odder
compositions (like flags) correctly.

----

int utf8Strlen( char const *str ) {
  int length = 0;
  int multibyte = 0;
  unsigned char c;

  while(c = *str++) {
    if(multibyte) {
      multibyte--;
      if((c & 0xC0) != 0x80 ) // UTF8 encoding error
	return -1;
    } else {
      length++;
      if (c < 0x80) // Normal 7-bit ASCII
	multibyte = 0;
      else if ((c & 0xE0) == 0xC0) // Single following byte
	multibyte = 1;
      else if ((c & 0xF0) == 0xE0) // Two following bytes
	multibyte = 2;
      else if ((c & 0xF8) == 0xF0) // Three following bytes
	multibyte = 3;
      else // UTF8 encoding error
	return -1;
    }
  }

  if(multibyte) // Ran out of string in the middle of multibyte character
    return -1;
  return length;
}

-- 
   /* * * Otto J. Makela <om@iki.fi> * * * * * * * * * */
  /* Phone: +358 40 765 5772, ICBM: N 60 10' E 24 55' */
 /* Mail: Mechelininkatu 26 B 27,  FI-00100 Helsinki */
/* * * Computers Rule 01001111 01001011 * * * * * * */

[toc] | [prev] | [next] | [standalone]

#164406

Am 14.01.2022 um 15:07 schrieb Otto J. Makela:
> om@iki.fi (Otto J. Makela) wrote:
> 
>> Took me a while to realize the countl_zero stuff was a C++ library
>> call, with the accompanying syntax.
>>
>> However, this is comp.lang.c, so C solutions are appreciated.
> 
> How about this? It of course does not count the slightly odder
> compositions (like flags) correctly.
> 
> ----
> 
> int utf8Strlen( char const *str ) {
>    int length = 0;
>    int multibyte = 0;
>    unsigned char c;
> 
>    while(c = *str++) {
>      if(multibyte) {
>        multibyte--;
>        if((c & 0xC0) != 0x80 ) // UTF8 encoding error
> 	return -1;
>      } else {
>        length++;
>        if (c < 0x80) // Normal 7-bit ASCII
> 	multibyte = 0;
>        else if ((c & 0xE0) == 0xC0) // Single following byte
> 	multibyte = 1;
>        else if ((c & 0xF0) == 0xE0) // Two following bytes
> 	multibyte = 2;
>        else if ((c & 0xF8) == 0xF0) // Three following bytes
> 	multibyte = 3;
>        else // UTF8 encoding error
> 	return -1;
>      }
>    }
> 
>    if(multibyte) // Ran out of string in the middle of multibyte character
>      return -1;
>    return length;
> }
> 

A lot of unpredictible branches.
That's what I want to prevent with the table.

[toc] | [prev] | [next] | [standalone]

#164417

On 14.01.22 15:07, Otto J. Makela wrote:
> 
> How about this? It of course does not count the slightly odder
> compositions (like flags) correctly.
> 

Why not just call mblen() in the loop instead?

[toc] | [prev] | [next] | [standalone]

#164534

Philipp Klaus Krause <pkk@spth.de> wrote:

> On 14.01.22 15:07, Otto J. Makela wrote:
>> How about this? It of course does not count the slightly odder
>> compositions (like flags) correctly.
>
> Why not just call mblen() in the loop instead?

Firstly, I will fully admit wasn't aware of its existance.

Secondly, on quick tests I can't seem to find the settings it needs to
be happy even with 2-byte characters. Apparently I don't understand it,
or my environment isn't correctly set up?

----

#include <stdio.h>
#include <stdlib.h>

int main() {
  unsigned char str[10]="Ä";

  printf("'%s' %02x %02x %02x %d\n",
         str,str[0],str[1],str[3],mblen(str,10));
  exit(0);
}

----

Produces output:

'Ä' c3 84 00 -1

-- 
   /* * * Otto J. Makela <om@iki.fi> * * * * * * * * * */
  /* Phone: +358 40 765 5772, ICBM: N 60 10' E 24 55' */
 /* Mail: Mechelininkatu 26 B 27,  FI-00100 Helsinki */
/* * * Computers Rule 01001111 01001011 * * * * * * */

[toc] | [prev] | [next] | [standalone]

#164536

On Saturday, 22 January 2022 at 09:44:52 UTC, Otto J. Makela wrote:
> Philipp Klaus Krause <p...@spth.de> wrote: 
> 
> > On 14.01.22 15:07, Otto J. Makela wrote: 
> >> How about this? It of course does not count the slightly odder 
> >> compositions (like flags) correctly. 
> > 
> > Why not just call mblen() in the loop instead?
> Firstly, I will fully admit wasn't aware of its existance. 
> 
> Secondly, on quick tests I can't seem to find the settings it needs to 
> be happy even with 2-byte characters. Apparently I don't understand it, 
> or my environment isn't correctly set up? 
> 
> ---- 
> 
> #include <stdio.h> 
> #include <stdlib.h> 
> 
> int main() { 
> unsigned char str[10]="Ä"; 
> 
> printf("'%s' %02x %02x %02x %d\n", 
> str,str[0],str[1],str[3],mblen(str,10)); 
> exit(0); 
> } 
> 
> ---- 
> 
> Produces output: 
> 
> 'Ä' c3 84 00 -1
> 
Something is wrong.
Try resetting the "shift state" with mblen(NULL, 0);

I doubt that's the problem, but it's worth a try.

[toc] | [prev] | [next] | [standalone]

#164538

On 1/22/22 7:48 AM, Malcolm McLean wrote:
> On Saturday, 22 January 2022 at 09:44:52 UTC, Otto J. Makela wrote:
>> Philipp Klaus Krause <p...@spth.de> wrote:
>>
>>> On 14.01.22 15:07, Otto J. Makela wrote:
>>>> How about this? It of course does not count the slightly odder
>>>> compositions (like flags) correctly.
>>>
>>> Why not just call mblen() in the loop instead?
>> Firstly, I will fully admit wasn't aware of its existance.
>>
>> Secondly, on quick tests I can't seem to find the settings it needs to
>> be happy even with 2-byte characters. Apparently I don't understand it,
>> or my environment isn't correctly set up?
>>
>> ----
>>
>> #include <stdio.h>
>> #include <stdlib.h>
>>
>> int main() {
>> unsigned char str[10]="Ä";
>>
>> printf("'%s' %02x %02x %02x %d\n",
>> str,str[0],str[1],str[3],mblen(str,10));
>> exit(0);
>> }
>>
>> ----
>>
>> Produces output:
>>
>> 'Ä' c3 84 00 -1
>>
> Something is wrong.
> Try resetting the "shift state" with mblen(NULL, 0);
> 
> I doubt that's the problem, but it's worth a try.

Could it be that the library isn't in a 'Unicode' mode but an 'ASCII' mode.

I thought that mblen was locale dependent, and if the default 'C' local 
defines the character set at 'ASCII', then the first byte of 0xC3 is in 
fact, an invalid character.

One of the limits of the C language is that it does not default 
(necessarily) to assuming Unicode, so the code may need an 
implementation specific call to set the locale to one that uses Unicode, 
if one exists (which isn't required).

[toc] | [prev] | [next] | [standalone]

#164546

On 1/22/22 04:44, Otto J. Makela wrote:
> #include <stdio.h>
> #include <stdlib.h>
> 
> int main() {
>   unsigned char str[10]="Ä";
> 
>   printf("'%s' %02x %02x %02x %d\n",
>          str,str[0],str[1],str[3],mblen(str,10));
>   exit(0);
> }

The set of valid extended characters is locale-dependent (5.2.1p1). In
the "C" locale on my system, there are none - only the basic character
set is supported. It's not clear to me that the C standard requires
this, but it certainly allows it.

You need to use setlocale() to select a locale that supports UTF-8.
Which locales are supported depend upon your implementation. On my
platform, the default locale (the one chosen if the second argument of
setlocale() is "") is determined by the LC_LANG environment, which I
have set to "en_US.UTF-8". When I insert setlocale(LC_CTYPE, "") in your
program before the mblen() call, I get the following results:

 'Ä' c3 84 00 2

[toc] | [prev] | [next] | [standalone]

#164671

James Kuyper <jameskuyper@alumni.caltech.edu> wrote:

> The set of valid extended characters is locale-dependent (5.2.1p1). In
> the "C" locale on my system, there are none - only the basic character
> set is supported. It's not clear to me that the C standard requires
> this, but it certainly allows it.
>
> You need to use setlocale() to select a locale that supports UTF-8.
> Which locales are supported depend upon your implementation. On my
> platform, the default locale (the one chosen if the second argument of
> setlocale() is "") is determined by the LC_LANG environment, which I
> have set to "en_US.UTF-8". When I insert setlocale(LC_CTYPE, "") in
> your program before the mblen() call, I get the following results:
>
>  'Ä' c3 84 00 2

Yep, that setlocale() call did it. I actually have the semi-joke value
of LANG=en_DK.UTF-8 in my environment, produces good results for me.

https://unix.stackexchange.com/questions/62316/why-is-there-no-euro-english-locale
-- 
   /* * * Otto J. Makela <om@iki.fi> * * * * * * * * * */
  /* Phone: +358 40 765 5772, ICBM: N 60 10' E 24 55' */
 /* Mail: Mechelininkatu 26 B 27,  FI-00100 Helsinki */
/* * * Computers Rule 01001111 01001011 * * * * * * */

[toc] | [prev] | [next] | [standalone]

#164672

ram@zedat.fu-berlin.de (Stefan Ram) wrote:

>   Here:

The end result is exactly the same as with my program,
mblen() returns -1.

James Kuyper <jameskuyper@alumni.caltech.edu> understood it, I need to
ask for the default locale. I'm a bit stumped as to why so.

-- 
   /* * * Otto J. Makela <om@iki.fi> * * * * * * * * * */
  /* Phone: +358 40 765 5772, ICBM: N 60 10' E 24 55' */
 /* Mail: Mechelininkatu 26 B 27,  FI-00100 Helsinki */
/* * * Computers Rule 01001111 01001011 * * * * * * */

[toc] | [prev] | [next] | [standalone]

#164674

On 1/27/22 06:33, Otto J. Makela wrote:
> ram@zedat.fu-berlin.de (Stefan Ram) wrote:
>
>> Here:
>
> The end result is exactly the same as with my program,
> mblen() returns -1.
>
> James Kuyper <jameskuyper@alumni.caltech.edu> understood it, I need to
> ask for the default locale. I'm a bit stumped as to why so.

Note: while I referred to it as the "default" locale, that's a bad term
for it, since the "C" locale is the true default:

"At program startup, the equivalent of
setlocale(LC_ALL, "C");
is executed." (7.11.1.1p4)

"... a value of "" for locale specifies the locale-specific native
environment." (7.11.1.1p3), so it would be better to call it the
"native" locale.

"In the "C" locale, isalpha returns true only for the characters for
which isupper or islower is true." (7.4.1.2p2)

"In the "C" locale, islower returns true only for the lowercase letters
(as defined in 5.2.1)." (7.4.1.7p2)

"In the "C" locale, isupper returns true only for the uppercase letters
(as defined in 5.2.1)." (7.4.1.11p2)

"... the 26 _uppercase letters_ of the Latin alphabet
A B C D E F G H I J K L M
N O P Q R S T U V W X Y Z

the 26 _lowercase letters_ of the Latin alphabet
a b c d e f g h i j k l m
n o p q r s t u v w x y z" (5.2.1p2)

Note that the phrases "uppercase letters" and "lowercase letters" are in
italics, an ISO convention indicating that the clause in which they
appear constitutes the official definition for those terms within this
document.

Thus, isalpha() can only be true in the "C" locale for the 26 letters of
the latin alphabet. That's not quite the same as requiring that those be
the only valid character encodings. Note, in particular, that it's not
possible to pass a member of the extended character set to isalpha() if
that member's encoding takes more than 1 byte, so nothing the standard
says about isalpha() can constrain how such characters are treated. Such
a character could be converted to wchar_t and passed to iswalpha(), but
that function has no special specifications for the "C" locale. However,
those facts certainly don't encourage support for additional valid
character encodings in the "C" locale.

[toc] | [prev] | [next] | [standalone]

#164768

James Kuyper <jameskuyper@alumni.caltech.edu> writes:

> On 1/27/22 06:33, Otto J. Makela wrote:
>
>> ram@zedat.fu-berlin.de (Stefan Ram) wrote:
>>
>>> Here:
>>
>> The end result is exactly the same as with my program,
>> mblen() returns -1.
>>
>> James Kuyper <jameskuyper@alumni.caltech.edu> understood it, I need to
>> ask for the default locale.  I'm a bit stumped as to why so.
>
> Note: while I referred to it as the "default" locale, that's a bad
> term for it, since the "C" locale is the true default:
>
> "At program startup, the equivalent of
> setlocale(LC_ALL, "C");
> is executed."  (7.11.1.1p4)
>
> "... a value of "" for locale specifies the locale-specific native
> environment."  (7.11.1.1p3), so it would be better to call it the
> "native" locale.
>
> "In the "C" locale, isalpha returns true only for the characters for
> which isupper or islower is true."  (7.4.1.2p2)
>
> "In the "C" locale, islower returns true only for the lowercase
> letters (as defined in 5.2.1)."  (7.4.1.7p2)
>
> "In the "C" locale, isupper returns true only for the uppercase
> letters (as defined in 5.2.1)."  (7.4.1.11p2)
>
> "... the 26 _uppercase letters_ of the Latin alphabet
> A B C D E F G H I J K L M
> N O P Q R S T U V W X Y Z
>
> the 26 _lowercase letters_ of the Latin alphabet
> a b c d e f g h i j k l m
> n o p q r s t u v w x y z" (5.2.1p2)
>
> Note that the phrases "uppercase letters" and "lowercase letters"
> are in italics, an ISO convention indicating that the clause in
> which they appear constitutes the official definition for those
> terms within this document.
>
> Thus, isalpha() can only be true in the "C" locale for the 26
> letters of the latin alphabet.  [...]

What I think you mean is that in the "C" locale isalpha() can be
true only for the 26 letters of the latin alphabet.

[toc] | [prev] | [next] | [standalone]

#164401

On Friday, 14 January 2022 at 12:18:42 UTC, Otto J. Makela wrote:
> Bonita Montero <Bonita....@gmail.com> wrote: 
> 
> > This is more correct since it completely detects encoding-errors: 
> > 
> > size_t utf8Strlen( char const *str ) 
> > { 
> > struct encode_t { size_t lenIncr, strIncr; }; 
> > static encode_t const encodes[] = 
> > { 
> > { 1, 1 }, 
> > { 0, 0 }, 
> > { 1, 2 }, 
> > { 1, 3 }, 
> > { 1, 4 }, 
> > { 0, 0 }, 
> > { 0, 0 }, 
> > { 0, 0 }, 
> > { 0, 0 } 
> > }; 
> > size_t len = 0; 
> > for( unsigned char c; (c = *str); ) 
> > { 
> > encode_t const &enc = encodes[(size_t)countl_zero<unsigned 
> > char>( ~c )]; 
> > if( !enc.lenIncr ) [[unlikely]] 
> > return -1; 
> > len += enc.lenIncr; 
> > for( char const *cpEnd = str + enc.strIncr; ++str != cpEnd; ) 
> > if( ((unsigned char)*str & 0x0C0) != 0x080 ) [[unlikely]] 
> > return -1; 
> > } 
> > return len; 
> > }
> I must be getting a bit old here, I don't seem to understand 
> what that countl_zero stuff (and the syntax thereabouts) is? 
> 
> PS. Also, shouldn't the first struct line be a typedef, 
> at least you seem to use it at bit later like it was? 
> PPS. And shouldn't const be before type, not after? 
>
Also, it would be safer to make the function return an int.
size_t is unsigned, and the return is -1 for error. So code like

size_t Nchars = utf8Strlen(utf8str);
size_t i;

for (i =0; i < Nchars; i++)
{
}

will run through quintillions of iterations, instead of being
a no-op, if the utf8 is malformed.

[toc] | [prev] | [next] | [standalone]

#164402

Am 14.01.2022 um 13:41 schrieb Malcolm McLean:
> On Friday, 14 January 2022 at 12:18:42 UTC, Otto J. Makela wrote:
>> Bonita Montero <Bonita....@gmail.com> wrote:
>>
>>> This is more correct since it completely detects encoding-errors:
>>>
>>> size_t utf8Strlen( char const *str )
>>> {
>>> struct encode_t { size_t lenIncr, strIncr; };
>>> static encode_t const encodes[] =
>>> {
>>> { 1, 1 },
>>> { 0, 0 },
>>> { 1, 2 },
>>> { 1, 3 },
>>> { 1, 4 },
>>> { 0, 0 },
>>> { 0, 0 },
>>> { 0, 0 },
>>> { 0, 0 }
>>> };
>>> size_t len = 0;
>>> for( unsigned char c; (c = *str); )
>>> {
>>> encode_t const &enc = encodes[(size_t)countl_zero<unsigned
>>> char>( ~c )];
>>> if( !enc.lenIncr ) [[unlikely]]
>>> return -1;
>>> len += enc.lenIncr;
>>> for( char const *cpEnd = str + enc.strIncr; ++str != cpEnd; )
>>> if( ((unsigned char)*str & 0x0C0) != 0x080 ) [[unlikely]]
>>> return -1;
>>> }
>>> return len;
>>> }
>> I must be getting a bit old here, I don't seem to understand
>> what that countl_zero stuff (and the syntax thereabouts) is?
>>
>> PS. Also, shouldn't the first struct line be a typedef,
>> at least you seem to use it at bit later like it was?
>> PPS. And shouldn't const be before type, not after?
>>
> Also, it would be safer to make the function return an int.
> size_t is unsigned, and the return is -1 for error. ...

All compilers support this because it is very common.

> 
> size_t Nchars = utf8Strlen(utf8str);
> size_t i;
> 
> for (i =0; i < Nchars; i++)
> {
> }
> 
> will run through quintillions of iterations, instead of being
> a no-op, if the utf8 is malformed.

[toc] | [prev] | [next] | [standalone]

#164403

On Friday, 14 January 2022 at 13:43:16 UTC, Bonita Montero wrote:
> Am 14.01.2022 um 13:41 schrieb Malcolm McLean: 
> > On Friday, 14 January 2022 at 12:18:42 UTC, Otto J. Makela wrote: 
> >> Bonita Montero <Bonita....@gmail.com> wrote: 
> >> 
> >>> This is more correct since it completely detects encoding-errors: 
> >>> 
> >>> size_t utf8Strlen( char const *str ) 
> >>> { 
> >>> struct encode_t { size_t lenIncr, strIncr; }; 
> >>> static encode_t const encodes[] = 
> >>> { 
> >>> { 1, 1 }, 
> >>> { 0, 0 }, 
> >>> { 1, 2 }, 
> >>> { 1, 3 }, 
> >>> { 1, 4 }, 
> >>> { 0, 0 }, 
> >>> { 0, 0 }, 
> >>> { 0, 0 }, 
> >>> { 0, 0 } 
> >>> }; 
> >>> size_t len = 0; 
> >>> for( unsigned char c; (c = *str); ) 
> >>> { 
> >>> encode_t const &enc = encodes[(size_t)countl_zero<unsigned 
> >>> char>( ~c )]; 
> >>> if( !enc.lenIncr ) [[unlikely]] 
> >>> return -1; 
> >>> len += enc.lenIncr; 
> >>> for( char const *cpEnd = str + enc.strIncr; ++str != cpEnd; ) 
> >>> if( ((unsigned char)*str & 0x0C0) != 0x080 ) [[unlikely]] 
> >>> return -1; 
> >>> } 
> >>> return len; 
> >>> } 
> >> I must be getting a bit old here, I don't seem to understand 
> >> what that countl_zero stuff (and the syntax thereabouts) is? 
> >> 
> >> PS. Also, shouldn't the first struct line be a typedef, 
> >> at least you seem to use it at bit later like it was? 
> >> PPS. And shouldn't const be before type, not after? 
> >> 
> > Also, it would be safer to make the function return an int.
> > size_t is unsigned, and the return is -1 for error. ... 
> 
> All compilers support this because it is very common.
>
Yes. It should compile cleanly. That's not the objection.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.c

csiph-web