Re: UTF-8 overlong encodings

Path	csiph.com!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail
From	Rainer Weikusat <rweikusat@talktalk.net>
Newsgroups	comp.unix.programmer
Subject	Re: UTF-8 overlong encodings
Date	Sun, 19 Dec 2021 21:53:25 +0000
Lines	37
Message-ID	<87pmps45yi.fsf@doppelsaurus.mobileactivedefense.com> (permalink)
References	<874k76zvrc.fsf@doppelsaurus.mobileactivedefense.com> <87y24ird6y.fsf@bsb.me.uk> <87mtkx6di7.fsf@doppelsaurus.mobileactivedefense.com> <87sfuppku0.fsf@bsb.me.uk> <87tuf446mh.fsf@doppelsaurus.mobileactivedefense.com>
Mime-Version	1.0
Content-Type	text/plain
X-Trace	individual.net KWff+LJnDszJ/BNWjaPAKw4mJSdkhBHbcElzQEvNkTTuI7NKI=
Cancel-Lock	sha1:pERDYD+8dpW+pZOFj7Js2ocAH6o= sha1:NB2+9ISgkPcPmbhQ57X4lchruXE=
User-Agent	Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)
Xref	csiph.com comp.unix.programmer:13655

Show key headers only | View raw

Rainer Weikusat <rweikusat@talktalk.net> writes:
> Ben Bacarisse <ben.usenet@bsb.me.uk> writes:
>> Rainer Weikusat <rweikusat@talktalk.net> writes:
>
> [...]
>
>>>>> An encoded sequences has a length of 2, 3, 4, 5 or 6 bytes. When
>>>>> ignoring the initial special case, the shift value relative to the start
>>>>> of the first six bit block for each encoded sequence is 8 -
>>>>> its length:
>>>>>
>>>>> 3	->	5
>>>>> 4	->	4
>>>>> 5	->	3
>>>>> 6	->      2
>>>>>
>>>>> Any corrections or other comments very much welcome.
>>>>
>>>> I was not sure what this part of the description was supposed to add to
>>>> the initial definition.
>>>
>>> I want to calculate that with a general algorithm.
>>
>> I don't know what "that" refers to.  Do you want to calculate the UTF-8
>> sequence length from the code point?  It seems not.  Do you want to
>> determine if a sequence is overlong by looking at the sequence?  It
>> seems not.  What is the algorithm given, and what it its result?
>
> I want to determine if a sequence is overlong using a generalized
> algorithm for that, ie, not by special-casing start byte values. So far,
> the untested (and very likely buggy) code for this looks like follows:
>
> u_len is the length of the sequence in bytes, p a pointer to the first
> byte. Some unrelated consistency checks removed.

At the moment, I'm convinced that this algorithm is complete
nonsense. :-)

Thread

UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-17 22:55 +0000
  Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-18 00:04 +0000
    Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-18 17:15 +0000
      Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-18 23:14 +0000
        Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-19 21:39 +0000
          Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-19 21:53 +0000
          Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-20 03:57 +0000
            Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 18:28 +0000
              Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 19:30 +0000
              Re: UTF-8 overlong encodings Ben Bacarisse <ben.usenet@bsb.me.uk> - 2021-12-20 20:46 +0000
                Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-20 21:24 +0000
          Re: UTF-8 overlong encodings Nicolas George <nicolas$george@salle-s.org> - 2021-12-20 13:34 +0000
  Re: UTF-8 overlong encodings Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-17 16:37 -0800
    Re: UTF-8 overlong encodings Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-18 17:19 +0000
      Re: UTF-8 overlong encodings John McCue <jmccue@fuzzball.mhome.org> - 2021-12-19 00:49 +0000
        Re: UTF-8 overlong encodings Nicolas George <nicolas$george@salle-s.org> - 2021-12-20 09:42 +0000
          Re: UTF-8 overlong encodings Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2021-12-20 10:56 -0800

csiph-web