Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.unix.programmer > #13654

Re: UTF-8 overlong encodings

Show all headers | View raw

Ben Bacarisse <ben.usenet@bsb.me.uk> writes:
> Rainer Weikusat <rweikusat@talktalk.net> writes:

[...]

>>>> An encoded sequences has a length of 2, 3, 4, 5 or 6 bytes. When
>>>> ignoring the initial special case, the shift value relative to the start
>>>> of the first six bit block for each encoded sequence is 8 -
>>>> its length:
>>>>
>>>> 3	->	5
>>>> 4	->	4
>>>> 5	->	3
>>>> 6	->      2
>>>>
>>>> Any corrections or other comments very much welcome.
>>>
>>> I was not sure what this part of the description was supposed to add to
>>> the initial definition.
>>
>> I want to calculate that with a general algorithm.
>
> I don't know what "that" refers to.  Do you want to calculate the UTF-8
> sequence length from the code point?  It seems not.  Do you want to
> determine if a sequence is overlong by looking at the sequence?  It
> seems not.  What is the algorithm given, and what it its result?

I want to determine if a sequence is overlong using a generalized
algorithm for that, ie, not by special-casing start byte values. So far,
the untested (and very likely buggy) code for this looks like follows:

u_len is the length of the sequence in bytes, p a pointer to the first
byte. Some unrelated consistency checks removed.

  mask = (1 << (8 - u_len)) - 1; /* all value bits in the first byte set */
  x = *p & mask;
  if (u_len == 2) if (x < 2) return U_BIN; /* 2 byte sequence overlong if only the lowest bit set */

  y = *++p;

  if (!x) { /* x == 0 implies u_len > 2 */
    mask = ~((1 << (8 - u_len)) - 1); /* all bits down to start bit in 2nd byte set */
    if ((y & mask) == 0x80) return U_BIN; /* overlong if continuation pattern only */
  }

Back to comp.unix.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-17 22:55 +0000
  Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-18 00:04 +0000
    Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-18 17:15 +0000
      Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-18 23:14 +0000
        Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-19 21:39 +0000
          Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-19 21:53 +0000
          Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-20 03:57 +0000
            Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 18:28 +0000
              Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 19:30 +0000
              Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-20 20:46 +0000
                Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 21:24 +0000
          Re: UTF-8 overlong encodings Nicolas George <nicolas$george@salle-s.org> - 2021-12-20 13:34 +0000
  Re: UTF-8 overlong encodings Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-17 16:37 -0800
    Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-18 17:19 +0000
      Re: UTF-8 overlong encodings John McCue <jmccue@fuzzball.mhome.org> - 2021-12-19 00:49 +0000
        Re: UTF-8 overlong encodings Nicolas George <nicolas$george@salle-s.org> - 2021-12-20 09:42 +0000
          Re: UTF-8 overlong encodings Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-20 10:56 -0800

csiph-web