Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #110667

Re: Processing text data with different encodings

Show all headers | View raw

On Tue, Jun 28, 2016 at 5:25 PM, Michael Welle <mwe012008@gmx.net> wrote:
> I want to use Python 3 to process data, that unfortunately come with
> different encodings. So far I have found ascii, iso-8859, utf-8,
> windows-1252 and maybe some more in the same file (don't ask...). I read
> the data via sys.stdin and the idea is to read a line, detect the
> current encoding, hit it until it looks like utf-8 and then go on with
> the next line of input:
>
>
> import cchardet
>
> for line in sys.stdin.buffer:
>
>     encoding = cchardet.detect(line)['encoding']
>     line = line.decode(encoding, 'ignore')\
>                .encode('UTF-8').decode('UTF-8', 'ignore')
>
>
> After that line should be a string. The logging module and some others
> choke on line: UnicodeEncodeError: 'charmap' codec can't encode
> character. What would be a right approach to tackle that problem
> (assuming that I can't change the input data)?

This is the exact sort of "ewwww" that I have to cope with in my MUD
client. Sometimes it gets sent UTF-8, other times it gets sent...
uhhhh... some eight-bit encoding, most likely either 8859 or 1252 (but
could theoretically be anything). The way I cope with it is to do a
line-by-line decode, similar to what you're doing, but with a much
simpler algorithm - something like this:

for line in <binary source>:
    try:
        line = line.decode("UTF-8")
    except UnicodeDecodeError:
        line = line.decode("1252")
    yield line

There's no need to chardet for UTF-8; if you successfully decode the
text, it's almost certainly correct. (This includes pure ASCII text,
which would also decode successfully and correctly as ISO-8859 or
Windows-1252.)

You shouldn't need this complicated triple-encode dance. Just decode
it once and work with text from there on. Ideally, you should be using
Python 3, where "work[ing] with text" is exactly how most of the code
wants to work; if not, resign yourself to reprs with u-prefixes, and
work with Unicode strings anyway. It'll save you a lot of trouble.

ChrisA

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 09:25 +0200
  Re: Processing text data with different encodings Chris Angelico <rosuav@gmail.com> - 2016-06-28 17:46 +1000
    Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 10:35 +0200
      Re: Processing text data with different encodings Steven D'Aprano <steve@pearwood.info> - 2016-06-28 20:29 +1000
        Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 12:37 +0200
          Re: Processing text data with different encodings Chris Angelico <rosuav@gmail.com> - 2016-06-28 21:09 +1000
  Re: Processing text data with different encodings Peter Otten <__peter__@web.de> - 2016-06-28 10:30 +0200
    Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 12:17 +0200
      Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 12:44 +0200
      Re: Processing text data with different encodings Steven D'Aprano <steve@pearwood.info> - 2016-06-28 21:26 +1000
        Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 14:30 +0200
          Re: Processing text data with different encodings Steven D'Aprano <steve@pearwood.info> - 2016-06-29 00:52 +1000
            Re: Processing text data with different encodings Random832 <random832@fastmail.com> - 2016-06-28 11:01 -0400
            Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 17:52 +0200
              Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-29 06:45 +0200
          Re: Processing text data with different encodings Steven D'Aprano <steve@pearwood.info> - 2016-06-29 01:11 +1000
      Re: Processing text data with different encodings Peter Otten <__peter__@web.de> - 2016-06-28 13:31 +0200
        Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 15:16 +0200
  Re: Processing text data with different encodings Chris Angelico <rosuav@gmail.com> - 2016-06-28 20:25 +1000
  Re: Processing text data with different encodings Random832 <random832@fastmail.com> - 2016-06-28 11:52 -0400
  Re: Processing text data with different encodings Chris Angelico <rosuav@gmail.com> - 2016-06-29 04:03 +1000

csiph-web