Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.unix.programmer > #13648

Re: UTF-8 overlong encodings

Show all headers | View raw

Ben Bacarisse <ben.usenet@bsb.me.uk> writes:
> Rainer Weikusat <rweikusat@talktalk.net> writes:
>
>> As usual with technical terms "everyone understands", it gets thrown
>> around everywhere but is never defined. The definition I derived is
>> below.
>>
>> The non-ASCII part of UTF-8 is composed of 5 ranges each of which
>> starts with a number which has only one bit set. The starting numbers
>> are 0x80, 0x800, 0x10000, 0x200000 and 0x4000000. An encoding is
>> 'overlong' when this start bit isn't set.
>
> I'd express it in terms of magnitude.  An overlong 2-byte sequence will
> decode to a value than 0x80.  An overlong encoded 3-byte value will be
> less than 0x800 and so on.  Or going the other way, you need at least
> two byte if the value to encode is >= 0x80, 3 bytes if it's >= 0x800 and
> so on.

Yes. That's an error I made: An overlong sequence is one where none of
the bits between the end of the prefix and the start bit (inclusive) are
set.

[...]

>> Expressed as (left) shift arguments, the starting bits are 7, 11, 16, 21
>> and 26.
>>
>> Each range is composed of a number of six bit blocks plus a
>> remainder which gets put into the byte starting the encoded
>> sequence. Again expressed as (left) shift arguments, the highest bits of
>> the left-most six bit blocks are 5, 11, 17, 23, 29.
>>
>> Subtracting the shift value corresponding with the highest bit in the
>> first six bit block from the shift value of the starting bit yiels the
>> position of this starting bit relative to the highest bit in the first
>> six bit block. The corresponding values are 2, 0, -1, -2 and -3.
>>
>> The first case is special because the starting bit is the bit
>> corresponding with 1 in the first byte. All other start bits are in the
>> second byte, at positions 5, 4, 3 and 2.
>>
>> An encoded sequences has a length of 2, 3, 4, 5 or 6 bytes. When
>> ignoring the initial special case, the shift value relative to the start
>> of the first six bit block for each encoded sequence is 8 -
>> its length:
>>
>> 3	->	5
>> 4	->	4
>> 5	->	3
>> 6	->      2
>>
>> Any corrections or other comments very much welcome.
>
> I was not sure what this part of the description was supposed to add to
> the initial definition.

I want to calculate that with a general algorithm.

Back to comp.unix.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-17 22:55 +0000
  Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-18 00:04 +0000
    Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-18 17:15 +0000
      Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-18 23:14 +0000
        Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-19 21:39 +0000
          Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-19 21:53 +0000
          Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-20 03:57 +0000
            Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 18:28 +0000
              Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 19:30 +0000
              Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-20 20:46 +0000
                Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 21:24 +0000
          Re: UTF-8 overlong encodings Nicolas George <nicolas$george@salle-s.org> - 2021-12-20 13:34 +0000
  Re: UTF-8 overlong encodings Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-17 16:37 -0800
    Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-18 17:19 +0000
      Re: UTF-8 overlong encodings John McCue <jmccue@fuzzball.mhome.org> - 2021-12-19 00:49 +0000
        Re: UTF-8 overlong encodings Nicolas George <nicolas$george@salle-s.org> - 2021-12-20 09:42 +0000
          Re: UTF-8 overlong encodings Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-20 10:56 -0800

csiph-web