Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #110721

Re: Processing text data with different encodings

Show key headers only | View raw

On Wed, Jun 29, 2016 at 1:52 AM, Random832 <random832@fastmail.com> wrote:
> On Tue, Jun 28, 2016, at 06:25, Chris Angelico wrote:
>> For the OP's situation, frankly, I doubt there'll be anything other
>> than UTF-8, Latin-1, and CP-1252. The chances that someone casually
>> mixes CP-1252 with (say) CP-1254 would be vanishingly small. So the
>> simple decode of "UTF-8, or failing that, 1252" is probably going to
>> give correct results for most of the content. The trick is figuring
>> out a correct boundary for the check; line-by-line may be sufficient,
>> or it may not.
>
> For completeness, this can be done character-by-character (i.e. try to
> decode a UTF-8 character, if it fails decode the offending byte as 1252)
> with an error handler:
>
> import codecs
>
> def cp1252_errors(exception):
>     input, idx = exception.object, exception.start
>     byte = input[idx:idx+1]
>     try:
>         return byte.decode('windows-1252'), idx+1
>     except UnicodeDecodeError:
>         # python's cp1252 doesn't accept 0x81, etc
>         return byte.decode('latin1'), idx+1

Yeah, and the decision as to where that boundary should be placed is
thus completely up to the application. I don't know of any situation
where you'd need the byte-by-byte version, but it's there if you want
it.

The reason I chose line-by-line in my MUD client is because of the
nature of MUDs. The server I primarily use is a naive eight-bit one -
whatever bytes it gets, it retransmits. (All the commands that it
responds to are ASCII, so we can assume that all encodings are
ASCII-compatible or the user will have major difficulties.) Some
clients (including mine) send UTF-8. If I send the command "trivia
This is a piece of text\n", the text will be encoded UTF-8, the server
receives those bytes, and then transmit (to everyone who's tuned to
the [trivia] channel) this text: "MyName [trivia] This is a piece of
text\r\n". Simplistic clients (usually on Windows) will do the same
thing, only they'll use their default encoding - usually 1252 - rather
than UTF-8. But the entire command will be encoded the same way, which
means the server will send an entire line (or several lines, if it
wraps) in the same encoding. It's safe to assume that any given line
will be in one single encoding, but consecutive lines could be in
different encodings.

For emails, it might be possible to use a larger section, but
line-by-line would be safe there too, most likely.

ChrisA

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 09:25 +0200
  Re: Processing text data with different encodings Chris Angelico <rosuav@gmail.com> - 2016-06-28 17:46 +1000
    Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 10:35 +0200
      Re: Processing text data with different encodings Steven D'Aprano <steve@pearwood.info> - 2016-06-28 20:29 +1000
        Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 12:37 +0200
          Re: Processing text data with different encodings Chris Angelico <rosuav@gmail.com> - 2016-06-28 21:09 +1000
  Re: Processing text data with different encodings Peter Otten <__peter__@web.de> - 2016-06-28 10:30 +0200
    Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 12:17 +0200
      Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 12:44 +0200
      Re: Processing text data with different encodings Steven D'Aprano <steve@pearwood.info> - 2016-06-28 21:26 +1000
        Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 14:30 +0200
          Re: Processing text data with different encodings Steven D'Aprano <steve@pearwood.info> - 2016-06-29 00:52 +1000
            Re: Processing text data with different encodings Random832 <random832@fastmail.com> - 2016-06-28 11:01 -0400
            Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 17:52 +0200
              Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-29 06:45 +0200
          Re: Processing text data with different encodings Steven D'Aprano <steve@pearwood.info> - 2016-06-29 01:11 +1000
      Re: Processing text data with different encodings Peter Otten <__peter__@web.de> - 2016-06-28 13:31 +0200
        Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 15:16 +0200
  Re: Processing text data with different encodings Chris Angelico <rosuav@gmail.com> - 2016-06-28 20:25 +1000
  Re: Processing text data with different encodings Random832 <random832@fastmail.com> - 2016-06-28 11:52 -0400
  Re: Processing text data with different encodings Chris Angelico <rosuav@gmail.com> - 2016-06-29 04:03 +1000

csiph-web