Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.unix.programmer > #13646

Re: UTF-8 overlong encodings

Show all headers | View raw

Rainer Weikusat <rweikusat@talktalk.net> writes:
> As usual with technical terms "everyone understands", it gets thrown
> around everywhere but is never defined. The definition I derived is
> below.
>
> The non-ASCII part of UTF-8 is composed of 5 ranges each of which
> starts with a number which has only one bit set. The starting numbers
> are 0x80, 0x800, 0x10000, 0x200000 and 0x4000000. An encoding is
> 'overlong' when this start bit isn't set.
>
> Expressed as (left) shift arguments, the starting bits are 7, 11, 16, 21
> and 26.
>
> Each range is composed of a number of six bit blocks plus a
> remainder which gets put into the byte starting the encoded
> sequence. Again expressed as (left) shift arguments, the highest bits of
> the left-most six bit blocks are 5, 11, 17, 23, 29.
>
> Subtracting the shift value corresponding with the highest bit in the
> first six bit block from the shift value of the starting bit yiels the
> position of this starting bit relative to the highest bit in the first
> six bit block. The corresponding values are 2, 0, -1, -2 and -3.
>
> The first case is special because the starting bit is the bit
> corresponding with 1 in the first byte. All other start bits are in the
> second byte, at positions 5, 4, 3 and 2.
>
> An encoded sequences has a length of 2, 3, 4, 5 or 6 bytes. When
> ignoring the initial special case, the shift value relative to the start
> of the first six bit block for each encoded sequence is 8 -
> its length:
>
> 3	->	5
> 4	->	4
> 5	->	3
> 6	->      2
>
> Any corrections or other comments very much welcome.

Unicode only defines character values up to 0x10fffd, so there are no
valid encodings longer than 4 octets.

Here's a table I came up with a while ago:

        00-7F   (7 bits)  0xxxxxxx
    0080-07FF  (11 bits)  110xxxxx 10xxxxxx
    0800-FFFF  (16 bits)  1110xxxx 10xxxxxx 10xxxxxx
010000-10FFFF  (21 bits)  11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

The character code is determined by concatenating the 'x's together.
A 1-octet encoding has 7 value bits.
A 2-octet encoding has 11 value bits.
And so on.

An octet starting with 0 is always a single-octet character (ASCII compatible).
An octet starting with 11 is always the first octet of a multi-octet encoding.
An octet starting with 10 is always a continuation octet.

Overlong encodings that use more octets than necessary are invalid.  For
example, the letter 'k' is 0x6b or 1101011 and is encoded in a single
octet:
    01101011
     -------
A two-octet encoding of the same character code is invalid:
    11000001 10101011
       -----   ------

You could extrapolate UTF-8 to up to 8-octet encodings, representing up to
42 bits, but that's also not valid UTF-8 (though I can imagine it being
useful for some purposes).

        110000-3FFFFFF (26 bits)  111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
       4000000-7FFFFFF (31 bits)  1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
     8000000-FFFFFFFFF (36 bits)  11111110 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
1000000000-3FFFFFFFFFF (42 bits)  11111111 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

https://en.wikipedia.org/wiki/UTF-8

-- 
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Philips
void Void(void) { Void(); } /* The recursive call of the void */

Back to comp.unix.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-17 22:55 +0000
  Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-18 00:04 +0000
    Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-18 17:15 +0000
      Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-18 23:14 +0000
        Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-19 21:39 +0000
          Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-19 21:53 +0000
          Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-20 03:57 +0000
            Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 18:28 +0000
              Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 19:30 +0000
              Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-20 20:46 +0000
                Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 21:24 +0000
          Re: UTF-8 overlong encodings Nicolas George <nicolas$george@salle-s.org> - 2021-12-20 13:34 +0000
  Re: UTF-8 overlong encodings Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-17 16:37 -0800
    Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-18 17:19 +0000
      Re: UTF-8 overlong encodings John McCue <jmccue@fuzzball.mhome.org> - 2021-12-19 00:49 +0000
        Re: UTF-8 overlong encodings Nicolas George <nicolas$george@salle-s.org> - 2021-12-20 09:42 +0000
          Re: UTF-8 overlong encodings Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-20 10:56 -0800

csiph-web