Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.misc > #15084

Re: Around the world with Unicode

Show all headers | View raw

Marko Rauhamaa <marko@pacujo.net> wrote:
> Rich <rich@example.invalid>:
> 
>> Marko Rauhamaa <marko@pacujo.net> wrote:
>>> Rich <rich@example.invalid>:
>>>> Multiple 'encoding' (i.e., an encoded UTF-8 byte string is again
>>>> passed through a byte string to utf-8 encoder function).
>>> 
>>> That may be necessary.
>>
>> Multiple, sequential, utf-8 encodings [as in
>> utf8encode(utf8encode(string))] are not ever necessary.
> 
> It is necessary in this case:
> 
>   utf8encode(almost_always_utf8encode(string))

Well, no, the proper reaction to "almost utf8" when the spec. was for
utf8 is "reject", not reencode yet again, making the mess worse.  But
upon reading further into your post, I now see why you say this.  You
are using utf-8 as a generic encoding medium to handle possibly almost
binary data but in a mostly human readable manner.  So it makes more
sense to me now why you do the above.

In my case, the origional data was supposed to be proper utf-8 up
front.  That much was explicitly clear to the folks that did the double
encoding (granted, I did not explictly state such here).  And so my
response was "fix your encoding!".  It sounds like your situation does
not offer you that avenue of escape.

>> This is the case I encountered. A byte string that was already utf-8
>> encoded (and that contained some 2-byte utf-8 sequences), was itself
>> treated as a raw byte string and utf-8 encoded a second time. Which
>> changes:
> 
> Of course things get messed up, but if you can't trust the inner coding
> to be valid UTF-8, you need to do something:

In my inner case the inner coding was *supposed* to be utf-8 from the
get-go.  So discovering the double encoding was an error.

Yes, to handle arbitrary data, that you can't reject outright, you have
to do whatever is necessary to fit the square pegs into the round
holes.

>>> For example, we are now trying to decide how to get a file name out
>>> of a percent-encoded URL.  The percent-encoding *usually* refers to
>>> UTF-8, but that's not guaranteed.
>>
>> Actually, no, percent encoding does not in general mean "utf-8"
>> encoded.  Percent encoding of a url says nothing about the encoding of
>> the underlying data that was percent encoded.  The underlying data may
>> be pure 7-bit ASCII (which technically, is also valid utf-8, but is not
>> /intentionally/ utf-8 encoded and contains no 2-byte utf-8 sequences),
>> and it was percent encoded simply because it contained URL reserved
>> characters [1].
> 
>   The generic URI syntax mandates that new URI schemes that provide for
>   the representation of character data in a URI must, in effect,
>   represent characters from the unreserved set without translation, and
>   should convert all other characters to bytes according to UTF-8, and
>   then percent-encode those values. This requirement was introduced in
>   January 2005 with the publication of RFC 3986. URI schemes introduced
>   before this date are not affected.
> 
>   <URL: https://en.wikipedia.org/wiki/Percent-encoding#Current_standard>
> 
> IOW, it is almost always UTF-8.

Ah, ok, in that case, one outcome would be "reject" those that are not. 
But then those that are not might just be ones created pre-2005 and so
are valid.  So yeah, you've not got a nice pretty picture to work with
there.  That is never going to ever be easy to handle.

>> But the true problem here is that there is no way to interpret
>> /unknown/ data in a /known/ way (beyond ignoring and treating it all
>> as a raw byte string) [2]. And neither utf-8, nor unicode, nor percent
>> encoding, nor base64, nor uuencode, nor etc. will help when the
>> incoming material is of unknown encoding and you want to do more than
>> handle it as a raw sequence of bytes.
> 
> Yes, we have determined that we need to handle it as raw bytes. Then, we
> need to decide how to encode that in JSON. Should be use base64 (which
> is unreadable) or UTF-8 (byte by byte), which looks good much of the
> time as quite many filenames contain a lot of ascii characters.

Unwrapping the percent encoding to raw bytes is about all you can do. 
You 'could' scan the raw bytes to categorize them as "only 7-bit ASCII
found" or "valid utf-8 encoding found" and not have to re-do anything
at that point.  But you'll always need to use some encoding to store
things when the result isn't "only ASCII" or "valid utf-8".  Which
means you either need a second flag for "encoded" vs "not encoded" or
you just encode everything and live with it.

I'd suggest that one choice of base64 vs utf-8 is whether you expect
that direct human readablilty is often needed or if someone will
usually interact via a tool.  If humans are often going to directly be
looking, then a blanket "utf-8 encoding" means that only the oddballs
look odd, everything else is readable.  But if interaction is always
through a tool, the tool could handle transparently decoding base64 so
the users never know the underlying storage format.

Back to comp.misc | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Around the world with Unicode RS Wood <rsw@therandymon.com> - 2017-12-04 13:34 -0500
  Re: Around the world with Unicode Huge <Huge@nowhere.much.invalid> - 2017-12-04 21:30 +0000
  Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-04 23:53 +0200
    Re: Around the world with Unicode Bruce Horrocks <07.013@scorecrow.com> - 2017-12-05 00:14 +0000
      Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-05 09:07 +0200
        Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-05 10:58 +0000
          Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-05 13:24 +0200
            Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-05 12:46 +0000
              Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-05 15:28 +0200
                Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-05 18:10 +0000
  Re: Around the world with Unicode Roger Blake <rogblake@iname.invalid> - 2017-12-06 03:48 +0000
    Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-06 02:19 -0400
      Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-06 09:31 +0200
      Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-06 11:25 +0000
        Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-06 15:25 +0200
          Re: Around the world with Unicode "Adam H. Kerman" <ahk@chinet.com> - 2017-12-06 18:32 +0000
            Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-07 00:22 +0200
              Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-06 19:57 -0400
          Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-06 18:43 +0000
            Re: Around the world with Unicode Paul Sture <nospam@sture.ch> - 2017-12-06 20:49 +0100
            Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-07 00:25 +0200
              Re: Around the world with Unicode scott@alfter.diespammersdie.us (Scott Alfter) - 2017-12-06 22:56 +0000
              Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-06 22:57 +0000
          Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-06 19:48 -0400
            Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-07 10:02 +0200
              Re: Around the world with Unicode Roger Blake <rogblake@iname.invalid> - 2017-12-07 16:54 +0000
                Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-07 14:55 -0400
              Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-07 14:43 -0400
                Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-07 21:09 +0200
                Re: Around the world with Unicode The Real Bev <bashley101@gmail.com> - 2017-12-07 11:48 -0800
                Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-08 00:43 -0400

csiph-web