Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.unix.programmer > #13660

Re: UTF-8 overlong encodings

Show all headers | View raw

Ben Bacarisse <ben.usenet@bsb.me.uk> writes:
> Rainer Weikusat <rweikusat@talktalk.net> writes:
>> Ben Bacarisse <ben.usenet@bsb.me.uk> writes:

[...]


>> u_len is the length of the sequence in bytes,
>
> How have you calculated u_len?  You can detect and overlong sequence
> without knowing it, so there is some risk in using it when it's not
> needed.

Assuming x is the start byte of a UTF-8 sequences stored as unsigned
32-bit integer, the length of the sequence is (using a gcc extension) 

__builtin_clz(x ^ 0xff) - 24

All bits left of 0x80 will already be clear. x ^ 0xff will clear all
bits up to the trailing 0 bit of the prefix and set that to 1. 

>
>> p a pointer to the first
>> byte. Some unrelated consistency checks removed.
>>
>>   mask = (1 << (8 - u_len)) - 1; /* all value bits in the first byte set */
>
> That includes one more bit than you want.

Indeed. 8 - u_len is the shift index of the last non-zero prefix bit. It
should have been 7 - u_len or (8 - (u_len + 1)).


[...]

> I don't see why you need to look at the next byte.
>
>>   if (!x) { /* x == 0 implies u_len > 2 */
>
> x == 0 implies an overlong sequence now that you have dealt with the
> length 2 case which can have one bit on x set and still be overlong.

According to the Linux man page, a number in the range 0x800 - 0xffff is
encoded as three bytes:

1110xxxx 10xxxxxx 10xxxxxx

Program encoding 0x800 in this way:

--------
#include <stdio.h>

int main(void)
{
    unsigned u = 0x800;
    unsigned char utf[3], *p;

    p = utf;
    *p++ = 0xe0 | (u >> 12);
    *p++ = 0x80 | ((u >> 6) & 63);
    *p = 0x80 | (u & 63);

    printf("%x %x %x\n", *utf, utf[1], utf[2]);

    return 0;
}
-------

And the output is e0 a0 80. The situation is similar for all other
ranges, including the 4-byte sequence which is actually supposed to be
used: The first number of the range corresponds with a set bit in the second
byte.

I may again have gotten something wrong here. But I've done all the
calculations twice and got the same result.

Back to comp.unix.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-17 22:55 +0000
  Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-18 00:04 +0000
    Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-18 17:15 +0000
      Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-18 23:14 +0000
        Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-19 21:39 +0000
          Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-19 21:53 +0000
          Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-20 03:57 +0000
            Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 18:28 +0000
              Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 19:30 +0000
              Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-20 20:46 +0000
                Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 21:24 +0000
          Re: UTF-8 overlong encodings Nicolas George <nicolas$george@salle-s.org> - 2021-12-20 13:34 +0000
  Re: UTF-8 overlong encodings Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-17 16:37 -0800
    Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-18 17:19 +0000
      Re: UTF-8 overlong encodings John McCue <jmccue@fuzzball.mhome.org> - 2021-12-19 00:49 +0000
        Re: UTF-8 overlong encodings Nicolas George <nicolas$george@salle-s.org> - 2021-12-20 09:42 +0000
          Re: UTF-8 overlong encodings Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-20 10:56 -0800

csiph-web