Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.misc > #15081

Re: Around the world with Unicode

Show all headers | View raw

Rich <rich@example.invalid>:
> Marko Rauhamaa <marko@pacujo.net> wrote:
>> However, I blame programming languages for introducing Unicode
>> strings. Byte strings containing UTF-8 is a better choice. Of course,
>> the UTF-8 approach has problems,
>
> Yes, and you left out two of those problems:
>
> Programs written by programmers that are told they are receiving UTF-8
> byte strings, but then fail to treat the same as UTF-8 encoded values.
>
> Results can be:
>
> Multiple 'encoding' (i.e., an encoded UTF-8 byte string is again
> passed through a byte string to utf-8 encoder function).

That may be necessary. For example, we are now trying to decide how to
get a file name out of a percent-encoded URL. The percent-encoding
*usually* refers to UTF-8, but that's not guaranteed. Should we just
interpret each percent-encoded byte as an independent Unicode codepoint
before packing it inside JSON? Or should we treat it as binary and
encode it with base64? Or should we just leave the percent signs in? Or
do we just wish for the best and treat it as UTF-8? That last option
will be optimal in 99.9% of the cases, but the remaining 0.1% must be
handled somehow.

> Incorrect string length measurements. The programmers measure the byte
> length and declare that as the glyph count. This results in things
> like "max 80 characters" actually being counted as "max 80 bytes" for
> validations and the like (and so 41 characters can trigger a "too
> long, over 80 characters" error with the right set of glyphs).

Of course, but Unicode doesn't change it in any way. The problem is
worse with Unicode strings because the programmer may equate a glyph
with a codepoint.

> In the end, it comes down to the knowledge and attention to detail of
> the person writing the code. There is no magic bullet, neither
> "unicode strings" (a very bad thing, I agree) nor "treat all as byte
> strings" mean that someone not paying attention to detail can not mess
> things up.

I believe text processing is a special problem area concerning only some
applications. Unicode is not sufficient for those applications. You need
to consider numerous other details: fonts, ligatures, hyphenation,
layout, writing direction etc. For that special need, you will need a
hefty support package where Unicode is a small part.

However, *most* applications don't deal with text processing. For
example, I wrote an SMTP server in Python. Why must I be burdened with
Unicode corner cases when all I want is to parse an EHLO or open a file?

Instead of Unicode strings, Python might benefit from glyph strings
where each glyph is a normalized cluster of Unicode codepoints. Such
"super-codepoints" would be up to 5 * 21 = 105 bits long (most would fit
in 63 bits). That simplistic data type would make it possible to write a
Usenet client for most European languages.

Actually, emacs developers have spent decades on this issue and claim to
have gotten an upper hand. They were lecturing to Guile developers that
Guile should learn from emacs. As for Python, Python's strings are
actually a superset of Unicode because they tolerate surrogate
codepoints, leading to these kinds of surprises:

   >>> a = "\udc12"
   >>> print(a)
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
   UnicodeEncodeError: 'utf-8' codec can't encode character '\udc12' in \
   position 0: surrogates not allowed

Does any programmer expect printing a string to raise an exception? Note
that Python uses the surrogates to encode non-UTF-8 filenames. So that
exception can arise simply by you printing out a directory listing.


Marko

Back to comp.misc | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Around the world with Unicode RS Wood <rsw@therandymon.com> - 2017-12-04 13:34 -0500
  Re: Around the world with Unicode Huge <Huge@nowhere.much.invalid> - 2017-12-04 21:30 +0000
  Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-04 23:53 +0200
    Re: Around the world with Unicode Bruce Horrocks <07.013@scorecrow.com> - 2017-12-05 00:14 +0000
      Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-05 09:07 +0200
        Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-05 10:58 +0000
          Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-05 13:24 +0200
            Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-05 12:46 +0000
              Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-05 15:28 +0200
                Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-05 18:10 +0000
  Re: Around the world with Unicode Roger Blake <rogblake@iname.invalid> - 2017-12-06 03:48 +0000
    Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-06 02:19 -0400
      Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-06 09:31 +0200
      Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-06 11:25 +0000
        Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-06 15:25 +0200
          Re: Around the world with Unicode "Adam H. Kerman" <ahk@chinet.com> - 2017-12-06 18:32 +0000
            Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-07 00:22 +0200
              Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-06 19:57 -0400
          Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-06 18:43 +0000
            Re: Around the world with Unicode Paul Sture <nospam@sture.ch> - 2017-12-06 20:49 +0100
            Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-07 00:25 +0200
              Re: Around the world with Unicode scott@alfter.diespammersdie.us (Scott Alfter) - 2017-12-06 22:56 +0000
              Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-06 22:57 +0000
          Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-06 19:48 -0400
            Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-07 10:02 +0200
              Re: Around the world with Unicode Roger Blake <rogblake@iname.invalid> - 2017-12-07 16:54 +0000
                Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-07 14:55 -0400
              Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-07 14:43 -0400
                Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-07 21:09 +0200
                Re: Around the world with Unicode The Real Bev <bashley101@gmail.com> - 2017-12-07 11:48 -0800
                Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-08 00:43 -0400

csiph-web