Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.misc > #15083

Re: Around the world with Unicode

Show all headers | View raw

Rich <rich@example.invalid>:

> Marko Rauhamaa <marko@pacujo.net> wrote:
>> Rich <rich@example.invalid>:
>>> Multiple 'encoding' (i.e., an encoded UTF-8 byte string is again
>>> passed through a byte string to utf-8 encoder function).
>> 
>> That may be necessary.
>
> Multiple, sequential, utf-8 encodings [as in
> utf8encode(utf8encode(string))] are not ever necessary.

It is necessary in this case:

   utf8encode(almost_always_utf8encode(string))

> This is the case I encountered. A byte string that was already utf-8
> encoded (and that contained some 2-byte utf-8 sequences), was itself
> treated as a raw byte string and utf-8 encoded a second time. Which
> changes:

Of course things get messed up, but if you can't trust the inner coding
to be valid UTF-8, you need to do something:

   pass_thru_binary(almost_always_utf8encode(string))

   base64_encode(almost_always_utf8encode(string))

   percent_encode(almost_always_utf8encode(string))

   hex_encode(almost_always_utf8encode(string))

>> For example, we are now trying to decide how to get a file name out
>> of a percent-encoded URL.  The percent-encoding *usually* refers to
>> UTF-8, but that's not guaranteed.
>
> Actually, no, percent encoding does not in general mean "utf-8"
> encoded.  Percent encoding of a url says nothing about the encoding of
> the underlying data that was percent encoded.  The underlying data may
> be pure 7-bit ASCII (which technically, is also valid utf-8, but is not
> /intentionally/ utf-8 encoded and contains no 2-byte utf-8 sequences),
> and it was percent encoded simply because it contained URL reserved
> characters [1].

   The generic URI syntax mandates that new URI schemes that provide for
   the representation of character data in a URI must, in effect,
   represent characters from the unreserved set without translation, and
   should convert all other characters to bytes according to UTF-8, and
   then percent-encode those values. This requirement was introduced in
   January 2005 with the publication of RFC 3986. URI schemes introduced
   before this date are not affected.

   <URL: https://en.wikipedia.org/wiki/Percent-encoding#Current_standard>

IOW, it is almost always UTF-8.

> But the true problem here is that there is no way to interpret
> /unknown/ data in a /known/ way (beyond ignoring and treating it all
> as a raw byte string) [2]. And neither utf-8, nor unicode, nor percent
> encoding, nor base64, nor uuencode, nor etc. will help when the
> incoming material is of unknown encoding and you want to do more than
> handle it as a raw sequence of bytes.

Yes, we have determined that we need to handle it as raw bytes. Then, we
need to decide how to encode that in JSON. Should be use base64 (which
is unreadable) or UTF-8 (byte by byte), which looks good much of the
time as quite many filenames contain a lot of ascii characters.


Marko

Back to comp.misc | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Around the world with Unicode RS Wood <rsw@therandymon.com> - 2017-12-04 13:34 -0500
  Re: Around the world with Unicode Huge <Huge@nowhere.much.invalid> - 2017-12-04 21:30 +0000
  Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-04 23:53 +0200
    Re: Around the world with Unicode Bruce Horrocks <07.013@scorecrow.com> - 2017-12-05 00:14 +0000
      Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-05 09:07 +0200
        Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-05 10:58 +0000
          Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-05 13:24 +0200
            Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-05 12:46 +0000
              Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-05 15:28 +0200
                Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-05 18:10 +0000
  Re: Around the world with Unicode Roger Blake <rogblake@iname.invalid> - 2017-12-06 03:48 +0000
    Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-06 02:19 -0400
      Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-06 09:31 +0200
      Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-06 11:25 +0000
        Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-06 15:25 +0200
          Re: Around the world with Unicode "Adam H. Kerman" <ahk@chinet.com> - 2017-12-06 18:32 +0000
            Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-07 00:22 +0200
              Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-06 19:57 -0400
          Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-06 18:43 +0000
            Re: Around the world with Unicode Paul Sture <nospam@sture.ch> - 2017-12-06 20:49 +0100
            Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-07 00:25 +0200
              Re: Around the world with Unicode scott@alfter.diespammersdie.us (Scott Alfter) - 2017-12-06 22:56 +0000
              Re: Around the world with Unicode Rich <rich@example.invalid> - 2017-12-06 22:57 +0000
          Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-06 19:48 -0400
            Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-07 10:02 +0200
              Re: Around the world with Unicode Roger Blake <rogblake@iname.invalid> - 2017-12-07 16:54 +0000
                Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-07 14:55 -0400
              Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-07 14:43 -0400
                Re: Around the world with Unicode Marko Rauhamaa <marko@pacujo.net> - 2017-12-07 21:09 +0200
                Re: Around the world with Unicode The Real Bev <bashley101@gmail.com> - 2017-12-07 11:48 -0800
                Re: Around the world with Unicode Mike Spencer <mds@bogus.nodomain.nowhere> - 2017-12-08 00:43 -0400

csiph-web