Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.misc > #15075 > unrolled thread

Around the world with Unicode

Back to article view | Back to comp.misc

Contents

  Around the world with Unicode RS Wood <rsw@therandymon.com> - 2017-12-04 13:34 -0500
    Re: Around the world with Unicode Huge <Huge@nowhere.much.invalid> - 2017-12-04 21:30 +0000
    Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-04 23:53 +0200
      Re: Around the world with Unicode Bruce Horrocks <07.013@scorecrow.com> - 2017-12-05 00:14 +0000
        Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-05 09:07 +0200
          Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-05 10:58 +0000
            Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-05 13:24 +0200
              Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-05 12:46 +0000
                Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-05 15:28 +0200
                  Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-05 18:10 +0000
    Re: Around the world with Unicode Roger Blake <rogblake@iname.invalid> - 2017-12-06 03:48 +0000
      Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-06 02:19 -0400
        Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-06 09:31 +0200
        Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-06 11:25 +0000
          Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-06 15:25 +0200
            Re: Around the world with Unicode "Adam H. Kerman" <ahk@chinet.com> - 2017-12-06 18:32 +0000
              Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-07 00:22 +0200
                Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-06 19:57 -0400
            Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-06 18:43 +0000
              Re: Around the world with Unicode Paul Sture <nospam@sture.ch> - 2017-12-06 20:49 +0100
              Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-07 00:25 +0200
                Re: Around the world with Unicode scott@alfter.diespammersdie.us (Scott Alfter) - 2017-12-06 22:56 +0000
                Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-06 22:57 +0000
            Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-06 19:48 -0400
              Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-07 10:02 +0200
                Re: Around the world with Unicode Roger Blake <rogblake@iname.invalid> - 2017-12-07 16:54 +0000
                  Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-07 14:55 -0400
                Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-07 14:43 -0400
                  Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-07 21:09 +0200
                  Re: Around the world with Unicode The Real Bev <bashley101@gmail.com> - 2017-12-07 11:48 -0800
                    Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-08 00:43 -0400

Page 1 of 2  [1] 2  Next page →

#15075 — Around the world with Unicode

https://norasandler.com/2017/11/02/Around-the-with-Unicode.html

A pretty funny read (all hail the brakcet!!!):

//-- clip
Unicode is really exciting, because it lets software handle two things:

    Emoji.
    All written human language

The first point is obviously the most important, but the second is also
pretty interesting. It turns out that cultures and languages are
complicated, which means writing systems are complicated, which means
that the Unicode standard is also complicated. It also turns out that
character encoding standards aren’t value-neutral. This blog post is a
very informal survey of some cool, interesting and surprising things
about Unicode.

What is Unicode?

Computers store all data as a series of bits, and this obviously
includes text. So computers need to know which series of bits
corresponds to which character. Back in Ye Olden Times most systems in
English-speaking countries used ASCII, which encoded each character
using 7 bits. 27 is 128, so you could encode 128 characters. That was
plenty for every upper- and lowercase letter, numbers, punctuation, the
space character, and so on. A byte is 8 bits, so one ASCII character
fits conveniently in a single byte. This was less than ideal if your
application had to handle recipes for soufflés, piña coladas, or
spätzle, and it was completely unworkable for people who used non-Latin
alphabets. So there were a bunch of other encodings for different
alphabets, and it was basically impossible to write software that could
handle text in more than one language (unless, maybe, one of those
languages was English).

So now we have Unicode, which maps every single character in every
writing system to a unique number, called a code point. (Each emoji is
assigned a code point too.) There are a few different ways to translate
code points into actual bits, the most common being UTF-82. If you
support Unicode, you don’t need different encodings for different
languages – you can handle text in any language or mix of languages,
uniformly and with no fuss. As long as you use Unicode all the time,
everything is easy!

This is a lie. Nothing in life is easy.

//-- clip


-- 
RS Wood <rsw@therandymon.com>

[toc] | [next] | [standalone]

#15076

On 2017-12-04, RS Wood <rsw@therandymon.com> wrote:
> https://norasandler.com/2017/11/02/Around-the-with-Unicode.html
>
> A pretty funny read (all hail the brakcet!!!):
>
> //-- clip
> Unicode is really exciting, because it lets software handle two things:

I worked for Xerox for a few years at one point, in their Tech Pubs
department. One of my jobs was the care and feeding of the machine
translation systems we used to translate product documentation (mostly
service manuals) between various languages. This stuff still fascinates
me. Thank you for bringing this to my attention.

-- 
Today is Pungenday, the 46th day of The Aftermath in the YOLD 3183
        Ph'nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn.

[toc] | [prev] | [next] | [standalone]

#15077

> As long as you use Unicode all the time, everything is easy!
>
> This is a lie. Nothing in life is easy.

Unfortunately several programming languages have fallen for the lie:
Python, Guile, Java, JavaScript, to name a few. Each of them has
introduced a Unicode string type of sorts. They have sacrificed the
simplicity of classic byte strings. In return, they were supposed to
nail internationalization once and for all. Unfortunately, as the
article illustrated, Unicode didn't solve the problem at all.

Like C, they should have stuck to byte strings.


Marko

[toc] | [prev] | [next] | [standalone]

#15078

On 04/12/2017 21:53, Marko Rauhamaa wrote:
> 
>> As long as you use Unicode all the time, everything is easy!
>>
>> This is a lie. Nothing in life is easy.
> 
> Unfortunately several programming languages have fallen for the lie:
> Python, Guile, Java, JavaScript, to name a few. Each of them has
> introduced a Unicode string type of sorts. They have sacrificed the
> simplicity of classic byte strings. In return, they were supposed to
> nail internationalization once and for all. Unfortunately, as the
> article illustrated, Unicode didn't solve the problem at all.
> 
> Like C, they should have stuck to byte strings.

These (programming) languages may well have problems and, as the article 
points out, Unicode itself has problems, but there is one area where the 
Unicode team have done an astonishing job, which is the the high-level 
of backwards compatibility achieved by the standard library routine 
(re-)writers when implementing Unicode support.

The opportunity to introduce subtle and hard to predict/detect bugs was 
huge but I don't ever recall hearing of a bug introduced as a result of 
recompiling old code in a newer, Unicode-aware compiler and OS. It just 
works.

This bit of Unicode doesn't get any recognition, IMHO, but it certainly 
deserves some. And now that we have been blessed with emoji ligature I 
should make the effort to offer more than just a smiley in praise!  :-)

-- 
Bruce Horrocks
Surrey
England
(bruce at scorecrow dot com)

[toc] | [prev] | [next] | [standalone]

#15079

Bruce Horrocks <07.013@scorecrow.com>:

> On 04/12/2017 21:53, Marko Rauhamaa wrote:
>> Unfortunately several programming languages have fallen for the lie:
>> Python, Guile, Java, JavaScript, to name a few. Each of them has
>> introduced a Unicode string type of sorts. They have sacrificed the
>> simplicity of classic byte strings. In return, they were supposed to
>> nail internationalization once and for all. Unfortunately, as the
>> article illustrated, Unicode didn't solve the problem at all.
>
> These (programming) languages may well have problems and, as the
> article points out, Unicode itself has problems, but there is one area
> where the Unicode team have done an astonishing job, which is the the
> high-level of backwards compatibility achieved by the standard library
> routine (re-)writers when implementing Unicode support.
>
> The opportunity to introduce subtle and hard to predict/detect bugs
> was huge but I don't ever recall hearing of a bug introduced as a
> result of recompiling old code in a newer, Unicode-aware compiler and
> OS. It just works.
>
> This bit of Unicode doesn't get any recognition, IMHO, but it
> certainly deserves some. And now that we have been blessed with emoji
> ligature I should make the effort to offer more than just a smiley in
> praise! :-)

I don't blame Unicode. It's not perfect, but given the problem domain,
perfection is not achievable.

However, I blame programming languages for introducing Unicode strings.
Byte strings containing UTF-8 is a better choice. Of course, the UTF-8
approach has problems, but the UTF-32 approach has the very same
problems:

 * There's no simple way to know the length of the string since both
   UTF-8 and UTF-32 are varying-width encodings.

 * There's no simple way to compare the semantic equality of two
   strings.

 * There's no simple way deal with illegal encodings.

However, the UTF-32 approach has nasty additional drawbacks:

 * File names cannot be represented as UTF-32 strings. Linux file names
   are generally treated as UTF-8, but it is not uncommon to encounter
   file names that are illegal UTF-8. Windows file names have an
   analogous UTF-16 issue.

 * Python has imported the distinction of text vs binary to Linux. Now
   you have to decide which way you open a file. What's more, Python
   assumes your stdin and stdout are Unicode text.

 * Accidental or malicious illegal encodings in text processing can lead
   to unprocessed exceptions that don't show up in tests but cause the
   program to misbehave.


To put it another way, we have three levels:

    bytes — codepoints — glyphs

You often want to operate on bytes. You sometimes want to operate on
glyphs. You virtually never want to operate on codepoints. But
introducing Unicode codepoints to strings and text files, programmers
are enticed to treat codepoints as glyphs and forced to deal with the
side effects.


Marko

[toc] | [prev] | [next] | [standalone]

#15080

Marko Rauhamaa <marko@pacujo.net> wrote:
> However, I blame programming languages for introducing Unicode strings.
> Byte strings containing UTF-8 is a better choice. Of course, the UTF-8
> approach has problems,

Yes, and you left out two of those problems:

Programs written by programmers that are told they are receiving UTF-8
byte strings, but then fail to treat the same as UTF-8 encoded values.

Results can be:

Multiple 'encoding' (i.e., an encoded UTF-8 byte string is again passed
through a byte string to utf-8 encoder function).

Incorrect string length measurements.  The programmers measure the byte
length and declare that as the glyph count.  This results in things
like "max 80 characters" actually being counted as "max 80 bytes" for
validations and the like (and so 41 characters can trigger a "too long,
over 80 characters" error with the right set of glyphs).


In the end, it comes down to the knowledge and attention to detail of
the person writing the code.  There is no magic bullet, neither
"unicode strings" (a very bad thing, I agree) nor "treat all as byte
strings" mean that someone not paying attention to detail can not mess
things up.

[toc] | [prev] | [next] | [standalone]

#15081

Rich <rich@example.invalid>:
> Marko Rauhamaa <marko@pacujo.net> wrote:
>> However, I blame programming languages for introducing Unicode
>> strings. Byte strings containing UTF-8 is a better choice. Of course,
>> the UTF-8 approach has problems,
>
> Yes, and you left out two of those problems:
>
> Programs written by programmers that are told they are receiving UTF-8
> byte strings, but then fail to treat the same as UTF-8 encoded values.
>
> Results can be:
>
> Multiple 'encoding' (i.e., an encoded UTF-8 byte string is again
> passed through a byte string to utf-8 encoder function).

That may be necessary. For example, we are now trying to decide how to
get a file name out of a percent-encoded URL. The percent-encoding
*usually* refers to UTF-8, but that's not guaranteed. Should we just
interpret each percent-encoded byte as an independent Unicode codepoint
before packing it inside JSON? Or should we treat it as binary and
encode it with base64? Or should we just leave the percent signs in? Or
do we just wish for the best and treat it as UTF-8? That last option
will be optimal in 99.9% of the cases, but the remaining 0.1% must be
handled somehow.

> Incorrect string length measurements. The programmers measure the byte
> length and declare that as the glyph count. This results in things
> like "max 80 characters" actually being counted as "max 80 bytes" for
> validations and the like (and so 41 characters can trigger a "too
> long, over 80 characters" error with the right set of glyphs).

Of course, but Unicode doesn't change it in any way. The problem is
worse with Unicode strings because the programmer may equate a glyph
with a codepoint.

> In the end, it comes down to the knowledge and attention to detail of
> the person writing the code. There is no magic bullet, neither
> "unicode strings" (a very bad thing, I agree) nor "treat all as byte
> strings" mean that someone not paying attention to detail can not mess
> things up.

I believe text processing is a special problem area concerning only some
applications. Unicode is not sufficient for those applications. You need
to consider numerous other details: fonts, ligatures, hyphenation,
layout, writing direction etc. For that special need, you will need a
hefty support package where Unicode is a small part.

However, *most* applications don't deal with text processing. For
example, I wrote an SMTP server in Python. Why must I be burdened with
Unicode corner cases when all I want is to parse an EHLO or open a file?

Instead of Unicode strings, Python might benefit from glyph strings
where each glyph is a normalized cluster of Unicode codepoints. Such
"super-codepoints" would be up to 5 * 21 = 105 bits long (most would fit
in 63 bits). That simplistic data type would make it possible to write a
Usenet client for most European languages.

Actually, emacs developers have spent decades on this issue and claim to
have gotten an upper hand. They were lecturing to Guile developers that
Guile should learn from emacs. As for Python, Python's strings are
actually a superset of Unicode because they tolerate surrogate
codepoints, leading to these kinds of surprises:

   >>> a = "\udc12"
   >>> print(a)
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
   UnicodeEncodeError: 'utf-8' codec can't encode character '\udc12' in \
   position 0: surrogates not allowed

Does any programmer expect printing a string to raise an exception? Note
that Python uses the surrogates to encode non-UTF-8 filenames. So that
exception can arise simply by you printing out a directory listing.


Marko

[toc] | [prev] | [next] | [standalone]

#15082

Marko Rauhamaa <marko@pacujo.net> wrote:
> Rich <rich@example.invalid>:
>> Marko Rauhamaa <marko@pacujo.net> wrote:
>>> However, I blame programming languages for introducing Unicode
>>> strings. Byte strings containing UTF-8 is a better choice. Of course,
>>> the UTF-8 approach has problems,
>>
>> Yes, and you left out two of those problems:
>>
>> Programs written by programmers that are told they are receiving UTF-8
>> byte strings, but then fail to treat the same as UTF-8 encoded values.
>>
>> Results can be:
>>
>> Multiple 'encoding' (i.e., an encoded UTF-8 byte string is again
>> passed through a byte string to utf-8 encoder function).
> 
> That may be necessary.

Multiple, sequential, utf-8 encodings [as in
utf8encode(utf8encode(string))] are not ever necessary.  This is the
case I encountered.  A byte string that was already utf-8 encoded (and
that contained some 2-byte utf-8 sequences), was itself treated as a raw
byte string and utf-8 encoded a second time.  Which changes:

So one starts with (hex to the right in case the actual characters get
corrupted):

abc°      61 62 63 b0 (code points)

And should have gotten:

abc°     61 62 63 c2 b0 (1 layer utf-8 encoded)

as the single pass UTF-8 version, but got:

abcà °   61 62 63 c3 82 c2 b0 (2 layer utf-8 encoded)

instead.

Multiple, sequential, different, encodings [as in
percentencode(utf8encode(string))] are sometimes necessary, but
repeating the same one multiple times is usually not necessary.

> For example, we are now trying to decide how to get a file name out
> of a percent-encoded URL.  The percent-encoding *usually* refers to
> UTF-8, but that's not guaranteed.

Actually, no, percent encoding does not in general mean "utf-8"
encoded.  Percent encoding of a url says nothing about the encoding of
the underlying data that was percent encoded.  The underlying data may
be pure 7-bit ASCII (which technically, is also valid utf-8, but is not
/intentionally/ utf-8 encoded and contains no 2-byte utf-8 sequences),
and it was percent encoded simply because it contained URL reserved
characters [1].

> Should we just interpret each percent-encoded byte as an independent
> Unicode codepoint before packing it inside JSON?  Or should we treat
> it as binary and encode it with base64?  Or should we just leave the
> percent signs in?  Or do we just wish for the best and treat it as
> UTF-8?  That last option will be optimal in 99.9% of the cases, but
> the remaining 0.1% must be handled somehow.

If you do not know the encoding used in the value before it was percent
encoded, then the problem is that you don't know what you've got after
you undo the percent encoding.  In which case the only paths that
perserve the values are percent decode and assume binary or leave it
percent encoded.  Both of which simply punt the handling of "how was
this thing encoded before it was percent encoded" off to another
downstream receiver.

But the true problem here is that there is no way to interpret
/unknown/ data in a /known/ way (beyond ignoring and treating it all as
a raw byte string) [2].  And neither utf-8, nor unicode, nor percent
encoding, nor base64, nor uuencode, nor etc.  will help when the
incoming material is of unknown encoding and you want to do more than
handle it as a raw sequence of bytes.  

>> Incorrect string length measurements. The programmers measure the byte
>> length and declare that as the glyph count. This results in things
>> like "max 80 characters" actually being counted as "max 80 bytes" for
>> validations and the like (and so 41 characters can trigger a "too
>> long, over 80 characters" error with the right set of glyphs).
> 
> Of course, but Unicode doesn't change it in any way. The problem is
> worse with Unicode strings because the programmer may equate a glyph
> with a codepoint.

Actually, this one was because the programmer ignored utf-8 encoding
entirely, yet they were receiving a utf-8 encoded string from a
web-form.  They were essentially performing strlen(value) on a 'value'
that was utf-8 encoded, and using the return from strlen as the
"character count" (it is actually the "byte count", which with utf-8 is
not always equal to "character count").  So if the webform held only
7-bit ASCII characters (which, naturally, is what they tested with),
one could enter 80 characters.  But if one entered anything that
created a two byte (or longer) utf-8 sequence (which, of course, is
what I tested with, since the form was specified up front before it was
coded as returning a utf-8 encoded string of Unicode characters), then
one could not enter "80 characters" (where in this case "character" was
meant to be "glyph visible on screen").

>> In the end, it comes down to the knowledge and attention to detail
>> of the person writing the code.  There is no magic bullet, neither
>> "unicode strings" (a very bad thing, I agree) nor "treat all as byte
>> strings" mean that someone not paying attention to detail can not
>> mess things up.
> 
> I believe text processing is a special problem area concerning only some
> applications. Unicode is not sufficient for those applications. You need
> to consider numerous other details: fonts, ligatures, hyphenation,
> layout, writing direction etc. For that special need, you will need a
> hefty support package where Unicode is a small part.
> 
> However, *most* applications don't deal with text processing. For
> example, I wrote an SMTP server in Python. Why must I be burdened with
> Unicode corner cases when all I want is to parse an EHLO or open a file?

Fair point, and you shouldn't.  Which is why these standards should
indicate what the expected encoding of the data are (and if they can
get by with 7-bit ASCII for their control messages, just limit to 7-bit
ASCII and be done).  SMTP being one such protocol that does not need
'unicode' anywhere within the control data.  Message body, fine,
SMTP protocol, not needed.   And then, when the standards indicate what
the expected encoding should be, you've got something to stand on to
reject someone's attempt to send something that does not comply
(presuming that 'reject' is allowable).

As for opening a file, yes, that's a weird case, and it is one where
the Unix/Linux method of "any byte value beyond ASCII null and '/' are
acceptable" is actually reasonable.  You will be able to create the
file, even if the string you have is utf-8 encoded.  It just may look
funny in other programs that are not utf-8 aware.  Not perfect by any
stretch, but a bit more forgiving of more 'edge' cases.


[1] https://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_reserved_characters

[2] I.e., what if the 'thing' before percent escaping was actually old
    IBM EBCDIC - interpreting that as ASCII or Unicode will both
    produce results that look like garbage.

[toc] | [prev] | [next] | [standalone]

#15083

Rich <rich@example.invalid>:

> Marko Rauhamaa <marko@pacujo.net> wrote:
>> Rich <rich@example.invalid>:
>>> Multiple 'encoding' (i.e., an encoded UTF-8 byte string is again
>>> passed through a byte string to utf-8 encoder function).
>> 
>> That may be necessary.
>
> Multiple, sequential, utf-8 encodings [as in
> utf8encode(utf8encode(string))] are not ever necessary.

It is necessary in this case:

   utf8encode(almost_always_utf8encode(string))

> This is the case I encountered. A byte string that was already utf-8
> encoded (and that contained some 2-byte utf-8 sequences), was itself
> treated as a raw byte string and utf-8 encoded a second time. Which
> changes:

Of course things get messed up, but if you can't trust the inner coding
to be valid UTF-8, you need to do something:

   pass_thru_binary(almost_always_utf8encode(string))

   base64_encode(almost_always_utf8encode(string))

   percent_encode(almost_always_utf8encode(string))

   hex_encode(almost_always_utf8encode(string))

>> For example, we are now trying to decide how to get a file name out
>> of a percent-encoded URL.  The percent-encoding *usually* refers to
>> UTF-8, but that's not guaranteed.
>
> Actually, no, percent encoding does not in general mean "utf-8"
> encoded.  Percent encoding of a url says nothing about the encoding of
> the underlying data that was percent encoded.  The underlying data may
> be pure 7-bit ASCII (which technically, is also valid utf-8, but is not
> /intentionally/ utf-8 encoded and contains no 2-byte utf-8 sequences),
> and it was percent encoded simply because it contained URL reserved
> characters [1].

   The generic URI syntax mandates that new URI schemes that provide for
   the representation of character data in a URI must, in effect,
   represent characters from the unreserved set without translation, and
   should convert all other characters to bytes according to UTF-8, and
   then percent-encode those values. This requirement was introduced in
   January 2005 with the publication of RFC 3986. URI schemes introduced
   before this date are not affected.

   <URL: https://en.wikipedia.org/wiki/Percent-encoding#Current_standard>

IOW, it is almost always UTF-8.

> But the true problem here is that there is no way to interpret
> /unknown/ data in a /known/ way (beyond ignoring and treating it all
> as a raw byte string) [2]. And neither utf-8, nor unicode, nor percent
> encoding, nor base64, nor uuencode, nor etc. will help when the
> incoming material is of unknown encoding and you want to do more than
> handle it as a raw sequence of bytes.

Yes, we have determined that we need to handle it as raw bytes. Then, we
need to decide how to encode that in JSON. Should be use base64 (which
is unreadable) or UTF-8 (byte by byte), which looks good much of the
time as quite many filenames contain a lot of ascii characters.


Marko

[toc] | [prev] | [next] | [standalone]

#15084

Marko Rauhamaa <marko@pacujo.net> wrote:
> Rich <rich@example.invalid>:
> 
>> Marko Rauhamaa <marko@pacujo.net> wrote:
>>> Rich <rich@example.invalid>:
>>>> Multiple 'encoding' (i.e., an encoded UTF-8 byte string is again
>>>> passed through a byte string to utf-8 encoder function).
>>> 
>>> That may be necessary.
>>
>> Multiple, sequential, utf-8 encodings [as in
>> utf8encode(utf8encode(string))] are not ever necessary.
> 
> It is necessary in this case:
> 
>   utf8encode(almost_always_utf8encode(string))

Well, no, the proper reaction to "almost utf8" when the spec. was for
utf8 is "reject", not reencode yet again, making the mess worse.  But
upon reading further into your post, I now see why you say this.  You
are using utf-8 as a generic encoding medium to handle possibly almost
binary data but in a mostly human readable manner.  So it makes more
sense to me now why you do the above.

In my case, the origional data was supposed to be proper utf-8 up
front.  That much was explicitly clear to the folks that did the double
encoding (granted, I did not explictly state such here).  And so my
response was "fix your encoding!".  It sounds like your situation does
not offer you that avenue of escape.

>> This is the case I encountered. A byte string that was already utf-8
>> encoded (and that contained some 2-byte utf-8 sequences), was itself
>> treated as a raw byte string and utf-8 encoded a second time. Which
>> changes:
> 
> Of course things get messed up, but if you can't trust the inner coding
> to be valid UTF-8, you need to do something:

In my inner case the inner coding was *supposed* to be utf-8 from the
get-go.  So discovering the double encoding was an error.

Yes, to handle arbitrary data, that you can't reject outright, you have
to do whatever is necessary to fit the square pegs into the round
holes.

>>> For example, we are now trying to decide how to get a file name out
>>> of a percent-encoded URL.  The percent-encoding *usually* refers to
>>> UTF-8, but that's not guaranteed.
>>
>> Actually, no, percent encoding does not in general mean "utf-8"
>> encoded.  Percent encoding of a url says nothing about the encoding of
>> the underlying data that was percent encoded.  The underlying data may
>> be pure 7-bit ASCII (which technically, is also valid utf-8, but is not
>> /intentionally/ utf-8 encoded and contains no 2-byte utf-8 sequences),
>> and it was percent encoded simply because it contained URL reserved
>> characters [1].
> 
>   The generic URI syntax mandates that new URI schemes that provide for
>   the representation of character data in a URI must, in effect,
>   represent characters from the unreserved set without translation, and
>   should convert all other characters to bytes according to UTF-8, and
>   then percent-encode those values. This requirement was introduced in
>   January 2005 with the publication of RFC 3986. URI schemes introduced
>   before this date are not affected.
> 
>   <URL: https://en.wikipedia.org/wiki/Percent-encoding#Current_standard>
> 
> IOW, it is almost always UTF-8.

Ah, ok, in that case, one outcome would be "reject" those that are not. 
But then those that are not might just be ones created pre-2005 and so
are valid.  So yeah, you've not got a nice pretty picture to work with
there.  That is never going to ever be easy to handle.

>> But the true problem here is that there is no way to interpret
>> /unknown/ data in a /known/ way (beyond ignoring and treating it all
>> as a raw byte string) [2]. And neither utf-8, nor unicode, nor percent
>> encoding, nor base64, nor uuencode, nor etc. will help when the
>> incoming material is of unknown encoding and you want to do more than
>> handle it as a raw sequence of bytes.
> 
> Yes, we have determined that we need to handle it as raw bytes. Then, we
> need to decide how to encode that in JSON. Should be use base64 (which
> is unreadable) or UTF-8 (byte by byte), which looks good much of the
> time as quite many filenames contain a lot of ascii characters.

Unwrapping the percent encoding to raw bytes is about all you can do. 
You 'could' scan the raw bytes to categorize them as "only 7-bit ASCII
found" or "valid utf-8 encoding found" and not have to re-do anything
at that point.  But you'll always need to use some encoding to store
things when the result isn't "only ASCII" or "valid utf-8".  Which
means you either need a second flag for "encoded" vs "not encoded" or
you just encode everything and live with it.

I'd suggest that one choice of base64 vs utf-8 is whether you expect
that direct human readablilty is often needed or if someone will
usually interact via a tool.  If humans are often going to directly be
looking, then a blanket "utf-8 encoding" means that only the oddballs
look odd, everything else is readable.  But if interaction is always
through a tool, the tool could handle transparently decoding base64 so
the users never know the underlying storage format.

[toc] | [prev] | [next] | [standalone]

#15086

On 2017-12-04, RS Wood <rsw@therandymon.com> wrote:
> Unicode is really exciting, because it lets software handle two things:

Yuck. I'm sticking with ASCII. :)

-- 
-----------------------------------------------------------------------------
  Roger Blake (Posts from Google Groups killfiled due to excess spam.)

  NSA sedition and treason        -- http://www.DeathToNSAthugs.com
  Don't talk to cops!             -- http://www.DontTalkToCops.com
  Badges don't grant extra rights -- http://www.CopBlock.org
-----------------------------------------------------------------------------

[toc] | [prev] | [next] | [standalone]

#15089

Roger Blake <rogblake@iname.invalid> writes:

> On 2017-12-04, RS Wood <rsw@therandymon.com> wrote:
> > Unicode is really exciting, because it lets software handle two things:
> 
> Yuck. I'm sticking with ASCII. :)

If you need to correspond or create text or print copy in Arabic,
Hebrew, Chinese, Hindi, Klingon or whatever, it's just fine that there
is (or will be RSN :-) a standardized way to do it digitally.

But it really irritates me that some of my correspondents -- the
technically clueless ones -- writing in a venue that is all English
language and an informal context never destined for typesetting or
publishing, use software that gratuitously puts in (what I take to be)
the UTF-8 chars for left & right single and double quotes, long dash,
ellipsis, apostrophe inter alia that can be handled just fine with
ASCII.  Then, because the result is "binary", it's formatted as
quoted-printable. So I have to run their text through assorted macros
to convert it from Dog's Breakfast to readable ASCII.  It's casual
email in English, FFS, not a submission to the Times of London.

-- 
Mike Spencer                  Nova Scotia, Canada

[toc] | [prev] | [next] | [standalone]

#15090

Mike Spencer <mds@bogus.nodomain.nowhere>:

> But it really irritates me that some of my correspondents -- the
> technically clueless ones -- writing in a venue that is all English
> language and an informal context never destined for typesetting or
> publishing, use software that gratuitously puts in (what I take to be)
> the UTF-8 chars for left & right single and double quotes, long dash,
> ellipsis, apostrophe inter alia that can be handled just fine with
> ASCII. Then, because the result is "binary", it's formatted as
> quoted-printable. So I have to run their text through assorted macros
> to convert it from Dog's Breakfast to readable ASCII. It's casual
> email in English, FFS, not a submission to the Times of London.

I don't like those special quotes, either—but it's not because of UTF-8.
Unicode has little to do with typesetting. I see no reason to avoid
Unicode in emails, Usenet, filenames etc. If your tools can't handle it,
I would blame the tools.


Marko

[toc] | [prev] | [next] | [standalone]

#15092

Mike Spencer <mds@bogus.nodomain.nowhere> wrote:
> 
> Roger Blake <rogblake@iname.invalid> writes:
> 
>> On 2017-12-04, RS Wood <rsw@therandymon.com> wrote:
>> > Unicode is really exciting, because it lets software handle two things:
>> 
>> Yuck. I'm sticking with ASCII. :)
> 
> If you need to correspond or create text or print copy in Arabic,
> Hebrew, Chinese, Hindi, Klingon or whatever, it's just fine that
> there is (or will be RSN :-) a standardized way to do it digitally.

Yes, when one has a 'real use' for those extra glyphs, having Unicode
and utf-8 is way superior to the old 'code page' days.

> But it really irritates me that some of my correspondents -- the
> technically clueless ones -- writing in a venue that is all English
> language and an informal context never destined for typesetting or
> publishing, use software that gratuitously puts in (what I take to
> be) the UTF-8 chars for left & right single and double quotes, long
> dash, ellipsis, apostrophe inter alia that can be handled just fine
> with ASCII.  Then, because the result is "binary", it's formatted as
> quoted-printable.  So I have to run their text through assorted
> macros to convert it from Dog's Breakfast to readable ASCII.  It's
> casual email in English, FFS, not a submission to the Times of
> London.

You'd be corresponding with users who are likely using MS OutHouse (aka
MS Outlook).  MS added an auto-conversion of plain ASCII quotes and
dashes into "typographic" quotes and em-dashes some time ago (it
started out being called 'smart quotes' then later was termed
"typographic quotes" or some such).

The individuals sending it are most likely unaware that anything odd is
going on.  All they might notice, if they even notice anything at all
is that their "quotes" look properly curly and balanced, their single
quotes tilt right, their dash is a bit longer, etc.

FWIW, the mutt email client handles decoding quoted printable (and the
other potential mime types) automatically, so with at least it there's
no need for extra work (and macros) to convert.  It might even be
possible to have it auto-filter the 'typographic' stuff back into plain
ASCII as well, but I never ventured to try as I don't get enough of
those to be bothersome.  I get more 'html' only emails (technically a
violation of email rules, and also usually from clueless businesses so
no hope of 'educating' them to do the right thing) than I get "smart
quotes" myself.

[toc] | [prev] | [next] | [standalone]

#15093

Rich <rich@example.invalid>:

> The individuals sending it are most likely unaware that anything odd
> is going on. All they might notice, if they even notice anything at
> all is that their "quotes" look properly curly and balanced, their
> single quotes tilt right, their dash is a bit longer, etc.

Whether such automatic conversions are a good idea or not, there
shouldn't be anything wrong with sending non-ASCII text

> FWIW, the mutt email client handles decoding quoted printable (and the
> other potential mime types) automatically, so with at least it there's
> no need for extra work (and macros) to convert.

That's how it should be. I don't know what takes place under the hood of
my car.

> It might even be possible to have it auto-filter the 'typographic'
> stuff back into plain ASCII as well, but I never ventured to try as I
> don't get enough of those to be bothersome.

Why should any user even *know* what ASCII is, let alone demand it?

What's so special about ASCII?

> I get more 'html' only emails (technically a violation of email rules,
> and also usually from clueless businesses so no hope of 'educating'
> them to do the right thing) than I get "smart quotes" myself.

MIME allows for HTML email. No violation there. There are other reasons
why HTML mail is usually a bad idea.


Marko

[toc] | [prev] | [next] | [standalone]

#15094

Marko Rauhamaa <marko@pacujo.net> wrote:
>Rich <rich@example.invalid>:

>>The individuals sending it are most likely unaware that anything odd
>>is going on. All they might notice, if they even notice anything at
>>all is that their "quotes" look properly curly and balanced, their
>>single quotes tilt right, their dash is a bit longer, etc.

>Whether such automatic conversions are a good idea or not, there
>shouldn't be anything wrong with sending non-ASCII text

There is when informal discussion is taking place in English and there's
no reason to use characters that shouldn't be used outside of a typeset
document.

>>FWIW, the mutt email client handles decoding quoted printable (and the
>>other potential mime types) automatically, so with at least it there's
>>no need for extra work (and macros) to convert.

>That's how it should be. I don't know what takes place under the hood of
>my car.

That's nice. When you break down other than right in front of a garage
that's open and has the spare parts to repair your automobile, you'll
happily feed yourself to the wolves knowing that you're participating in
the cycle of life.

For me, if there's a repair to be made that's not too difficult, that
gives me the choice of surviving a breakdown in an inconvenient location
and not being terribly delayed, possibly avoiding a dangerous situation.

You go play with the pack of wolves.

>>It might even be possible to have it auto-filter the 'typographic'
>>stuff back into plain ASCII as well, but I never ventured to try as I
>>don't get enough of those to be bothersome.

>Why should any user even *know* what ASCII is, let alone demand it?

>What's so special about ASCII?

ASCII, a subset of most of the character sets in use, is universally
displayable and ideal for communicating informally in English. HTH

>>. . . 

[toc] | [prev] | [next] | [standalone]

#15097

"Adam H. Kerman" <ahk@chinet.com>:
> Marko Rauhamaa <marko@pacujo.net> wrote:
>>That's how it should be. I don't know what takes place under the hood
>>of my car.
>
> That's nice. When you break down other than right in front of a garage
> that's open and has the spare parts to repair your automobile, you'll
> happily feed yourself to the wolves knowing that you're participating
> in the cycle of life.
>
> For me, if there's a repair to be made that's not too difficult, that
> gives me the choice of surviving a breakdown in an inconvenient
> location and not being terribly delayed, possibly avoiding a dangerous
> situation.
>
> You go play with the pack of wolves.

Quite a dramatic point.

Our life is compartmentalized and specialized. Blurring interfaces
usually causes more problems than advantages. As a software engineer, I
should not expose my users to implementation details. That'll only give
them bad improvization ideas and tie my hands when I want to improve the
internal designs.

>>What's so special about ASCII?
>
> ASCII, a subset of most of the character sets in use, is universally
> displayable and ideal for communicating informally in English. HTH

I have been interviewing candidates for the position of senior software
engineer. It is quite surprising how unknown ASCII is among even
experienced software developers.

As for non-specialist computer users, they just want their emojis to
show up and wiggle nicely on the screen.


Marko

[toc] | [prev] | [next] | [standalone]

#15102

Marko Rauhamaa <marko@pacujo.net> writes:

> Our life is compartmentalized and specialized. 

Regretable where true.  Counterproductive for the individual albeit it
useful for h{is,er} employer.

> Blurring interfaces usually causes more problems than advantages. As
> a software engineer, I should not expose my users to implementation
> details. That'll only give them bad improvization ideas and tie my
> hands when I want to improve the internal designs.

The automotive industry has yearned for this for a hundred years.
Lock the hood with a secret key, return to the official vendor's shop
for maintanance or repairs.  In an analogue world, an impossible
dream.  People took cars apart, measured things, put them together
again, and knew how they worked thereafter.  Today, they nearly have
the dream because everything about "how it works" is inside digital
black boxes.  The black boxes are, I hear, penetrable but only in ways
utterly different from our evolved skills of learning how thw world
works. 


-- 
Mike Spencer                  Nova Scotia, Canada

[toc] | [prev] | [next] | [standalone]

#15095

Marko Rauhamaa <marko@pacujo.net> wrote:
> Rich <rich@example.invalid>:
>> I get more 'html' only emails (technically a violation of email
>> rules, and also usually from clueless businesses so no hope of
>> 'educating' them to do the right thing) than I get "smart quotes"
>> myself.
> 
> MIME allows for HTML email. No violation there. There are other
> reasons why HTML mail is usually a bad idea.

The violation is that an HTML email is supposed to be accompanied with
a text/plain version that contains the information, but as just text.

Most of the businesses that email in HTML format forget to include the
additonal text/plain copy of the HTML version.

[toc] | [prev] | [next] | [standalone]

#15096

On 2017-12-06, Rich <rich@example.invalid> wrote:
> Marko Rauhamaa <marko@pacujo.net> wrote:
>> Rich <rich@example.invalid>:
>>> I get more 'html' only emails (technically a violation of email
>>> rules, and also usually from clueless businesses so no hope of
>>> 'educating' them to do the right thing) than I get "smart quotes"
>>> myself.
>> 
>> MIME allows for HTML email. No violation there. There are other
>> reasons why HTML mail is usually a bad idea.
>
> The violation is that an HTML email is supposed to be accompanied with
> a text/plain version that contains the information, but as just text.

Agreed.

> Most of the businesses that email in HTML format forget to include the
> additonal text/plain copy of the HTML version.

But occasionally there's a pleasant surprise, which goes along the lines
of:

   "Since you are reading the plain text version of this message, you
   obviously prefer this format..."

 with steps to get text-only mails sent in the future.

-- 
Everybody has a testing environment. Some people are lucky enough to
have a totally separate environment to run production in.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.misc

csiph-web