Path: csiph.com!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail From: Keith Thompson Newsgroups: comp.unix.programmer Subject: Re: UTF-8 overlong encodings Date: Fri, 17 Dec 2021 16:37:00 -0800 Organization: None to speak of Lines: 82 Message-ID: <87lf0i7npv.fsf@nosuchdomain.example.com> References: <874k76zvrc.fsf@doppelsaurus.mobileactivedefense.com> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: reader02.eternal-september.org; posting-host="364d79685ae1098260e47310ff6b1319"; logging-data="28079"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1+ef94JXqEsnsztd+d7laY/" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) Cancel-Lock: sha1:4kzc6gju2KPdIg0NQ1trxyUYIhA= sha1:VOvjidPagqcw2QanmNzeLaXPlAU= Xref: csiph.com comp.unix.programmer:13646 Rainer Weikusat writes: > As usual with technical terms "everyone understands", it gets thrown > around everywhere but is never defined. The definition I derived is > below. > > The non-ASCII part of UTF-8 is composed of 5 ranges each of which > starts with a number which has only one bit set. The starting numbers > are 0x80, 0x800, 0x10000, 0x200000 and 0x4000000. An encoding is > 'overlong' when this start bit isn't set. > > Expressed as (left) shift arguments, the starting bits are 7, 11, 16, 21 > and 26. > > Each range is composed of a number of six bit blocks plus a > remainder which gets put into the byte starting the encoded > sequence. Again expressed as (left) shift arguments, the highest bits of > the left-most six bit blocks are 5, 11, 17, 23, 29. > > Subtracting the shift value corresponding with the highest bit in the > first six bit block from the shift value of the starting bit yiels the > position of this starting bit relative to the highest bit in the first > six bit block. The corresponding values are 2, 0, -1, -2 and -3. > > The first case is special because the starting bit is the bit > corresponding with 1 in the first byte. All other start bits are in the > second byte, at positions 5, 4, 3 and 2. > > An encoded sequences has a length of 2, 3, 4, 5 or 6 bytes. When > ignoring the initial special case, the shift value relative to the start > of the first six bit block for each encoded sequence is 8 - > its length: > > 3 -> 5 > 4 -> 4 > 5 -> 3 > 6 -> 2 > > Any corrections or other comments very much welcome. Unicode only defines character values up to 0x10fffd, so there are no valid encodings longer than 4 octets. Here's a table I came up with a while ago: 00-7F (7 bits) 0xxxxxxx 0080-07FF (11 bits) 110xxxxx 10xxxxxx 0800-FFFF (16 bits) 1110xxxx 10xxxxxx 10xxxxxx 010000-10FFFF (21 bits) 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx The character code is determined by concatenating the 'x's together. A 1-octet encoding has 7 value bits. A 2-octet encoding has 11 value bits. And so on. An octet starting with 0 is always a single-octet character (ASCII compatible). An octet starting with 11 is always the first octet of a multi-octet encoding. An octet starting with 10 is always a continuation octet. Overlong encodings that use more octets than necessary are invalid. For example, the letter 'k' is 0x6b or 1101011 and is encoded in a single octet: 01101011 ------- A two-octet encoding of the same character code is invalid: 11000001 10101011 ----- ------ You could extrapolate UTF-8 to up to 8-octet encodings, representing up to 42 bits, but that's also not valid UTF-8 (though I can imagine it being useful for some purposes). 110000-3FFFFFF (26 bits) 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 4000000-7FFFFFF (31 bits) 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 8000000-FFFFFFFFF (36 bits) 11111110 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 1000000000-3FFFFFFFFFF (42 bits) 11111111 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx https://en.wikipedia.org/wiki/UTF-8 -- Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com Working, but not speaking, for Philips void Void(void) { Void(); } /* The recursive call of the void */