Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.unix.programmer > #13651

Re: UTF-8 overlong encodings

Show all headers | View raw

Rainer Weikusat <rweikusat@talktalk.net> writes:

> Ben Bacarisse <ben.usenet@bsb.me.uk> writes:
>> Rainer Weikusat <rweikusat@talktalk.net> writes:
>>
>>> As usual with technical terms "everyone understands", it gets thrown
>>> around everywhere but is never defined. The definition I derived is
>>> below.
>>>
>>> The non-ASCII part of UTF-8 is composed of 5 ranges each of which
>>> starts with a number which has only one bit set. The starting numbers
>>> are 0x80, 0x800, 0x10000, 0x200000 and 0x4000000. An encoding is
>>> 'overlong' when this start bit isn't set.
>>
>> I'd express it in terms of magnitude.  An overlong 2-byte sequence will
>> decode to a value than 0x80.  An overlong encoded 3-byte value will be
>> less than 0x800 and so on.  Or going the other way, you need at least
>> two byte if the value to encode is >= 0x80, 3 bytes if it's >= 0x800 and
>> so on.
>
> Yes. That's an error I made: An overlong sequence is one where none of
> the bits between the end of the prefix and the start bit (inclusive) are
> set.
>
> [...]
>
>>> Expressed as (left) shift arguments, the starting bits are 7, 11, 16, 21
>>> and 26.
>>>
>>> Each range is composed of a number of six bit blocks plus a
>>> remainder which gets put into the byte starting the encoded
>>> sequence. Again expressed as (left) shift arguments, the highest bits of
>>> the left-most six bit blocks are 5, 11, 17, 23, 29.
>>>
>>> Subtracting the shift value corresponding with the highest bit in the
>>> first six bit block from the shift value of the starting bit yiels the
>>> position of this starting bit relative to the highest bit in the first
>>> six bit block. The corresponding values are 2, 0, -1, -2 and -3.
>>>
>>> The first case is special because the starting bit is the bit
>>> corresponding with 1 in the first byte. All other start bits are in the
>>> second byte, at positions 5, 4, 3 and 2.
>>>
>>> An encoded sequences has a length of 2, 3, 4, 5 or 6 bytes. When
>>> ignoring the initial special case, the shift value relative to the start
>>> of the first six bit block for each encoded sequence is 8 -
>>> its length:
>>>
>>> 3	->	5
>>> 4	->	4
>>> 5	->	3
>>> 6	->      2
>>>
>>> Any corrections or other comments very much welcome.
>>
>> I was not sure what this part of the description was supposed to add to
>> the initial definition.
>
> I want to calculate that with a general algorithm.

I don't know what "that" refers to.  Do you want to calculate the UTF-8
sequence length from the code point?  It seems not.  Do you want to
determine if a sequence is overlong by looking at the sequence?  It
seems not.  What is the algorithm given, and what it its result?

-- 
Ben.

Back to comp.unix.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-17 22:55 +0000
  Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-18 00:04 +0000
    Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-18 17:15 +0000
      Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-18 23:14 +0000
        Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-19 21:39 +0000
          Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-19 21:53 +0000
          Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-20 03:57 +0000
            Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 18:28 +0000
              Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 19:30 +0000
              Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-20 20:46 +0000
                Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 21:24 +0000
          Re: UTF-8 overlong encodings Nicolas George <nicolas$george@salle-s.org> - 2021-12-20 13:34 +0000
  Re: UTF-8 overlong encodings Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-17 16:37 -0800
    Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-18 17:19 +0000
      Re: UTF-8 overlong encodings John McCue <jmccue@fuzzball.mhome.org> - 2021-12-19 00:49 +0000
        Re: UTF-8 overlong encodings Nicolas George <nicolas$george@salle-s.org> - 2021-12-20 09:42 +0000
          Re: UTF-8 overlong encodings Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-20 10:56 -0800

csiph-web