Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.unix.programmer > #13656

Re: UTF-8 overlong encodings

Show all headers | View raw

Rainer Weikusat <rweikusat@talktalk.net> writes:

> Ben Bacarisse <ben.usenet@bsb.me.uk> writes:
>> Rainer Weikusat <rweikusat@talktalk.net> writes:
>
> [...]
>
>>>>> An encoded sequences has a length of 2, 3, 4, 5 or 6 bytes. When
>>>>> ignoring the initial special case, the shift value relative to the start
>>>>> of the first six bit block for each encoded sequence is 8 -
>>>>> its length:
>>>>>
>>>>> 3	->	5
>>>>> 4	->	4
>>>>> 5	->	3
>>>>> 6	->      2
>>>>>
>>>>> Any corrections or other comments very much welcome.
>>>>
>>>> I was not sure what this part of the description was supposed to add to
>>>> the initial definition.
>>>
>>> I want to calculate that with a general algorithm.
>>
>> I don't know what "that" refers to.  Do you want to calculate the UTF-8
>> sequence length from the code point?  It seems not.  Do you want to
>> determine if a sequence is overlong by looking at the sequence?  It
>> seems not.  What is the algorithm given, and what it its result?
>
> I want to determine if a sequence is overlong using a generalized
> algorithm for that, ie, not by special-casing start byte values.

I don't think I follow what you mean.  Over long sequences are special
case so you have to special-case something.  Why not the first byte?  It
seems to be such a simple method.

> So far,
> the untested (and very likely buggy) code for this looks like follows:
>
> u_len is the length of the sequence in bytes,

How have you calculated u_len?  You can detect and overlong sequence
without knowing it, so there is some risk in using it when it's not
needed.

> p a pointer to the first
> byte. Some unrelated consistency checks removed.
>
>   mask = (1 << (8 - u_len)) - 1; /* all value bits in the first byte set */

That includes one more bit than you want.  In a proper UTF-8 sequence,
that bit will be zero, so it's harmless, but have you already checked
that the sequence is valid (other than possibly being overlong).

By the way, I'd use 0xff >> u_len to get the mask.  It seems more
natural.

>   x = *p & mask;
>   if (u_len == 2) if (x < 2) return U_BIN; /* 2 byte sequence overlong if only the lowest bit set */

(or if no bits are set, but you include that in your test)

>   y = *++p;

I don't see why you need to look at the next byte.

>   if (!x) { /* x == 0 implies u_len > 2 */

x == 0 implies an overlong sequence now that you have dealt with the
length 2 case which can have one bit on x set and still be overlong.

>     mask = ~((1 << (8 - u_len)) - 1); /* all bits down to start bit in 2nd byte set */
>     if ((y & mask) == 0x80) return U_BIN; /* overlong if continuation pattern only */
>   }

-- 
Ben.

Back to comp.unix.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-17 22:55 +0000
  Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-18 00:04 +0000
    Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-18 17:15 +0000
      Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-18 23:14 +0000
        Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-19 21:39 +0000
          Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-19 21:53 +0000
          Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-20 03:57 +0000
            Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 18:28 +0000
              Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 19:30 +0000
              Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-20 20:46 +0000
                Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 21:24 +0000
          Re: UTF-8 overlong encodings Nicolas George <nicolas$george@salle-s.org> - 2021-12-20 13:34 +0000
  Re: UTF-8 overlong encodings Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-17 16:37 -0800
    Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-18 17:19 +0000
      Re: UTF-8 overlong encodings John McCue <jmccue@fuzzball.mhome.org> - 2021-12-19 00:49 +0000
        Re: UTF-8 overlong encodings Nicolas George <nicolas$george@salle-s.org> - 2021-12-20 09:42 +0000
          Re: UTF-8 overlong encodings Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-20 10:56 -0800

csiph-web