Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #110676

Re: Processing text data with different encodings

From Chris Angelico <rosuav@gmail.com>
Newsgroups comp.lang.python
Subject Re: Processing text data with different encodings
Date 2016-06-28 20:25 +1000
Message-ID <mailman.69.1467109546.2358.python-list@python.org> (permalink)
References <ubl94dxa2e.ln2@news.c0t0d0s0.de> <nktcj3$4u8$1@ger.gmane.org> <CAPTjJmrimmk=HQmLbh1D5t-F3nq4gJh=grA7OdBZ_qynXdU6Rg@mail.gmail.com>

Show all headers | View raw

On Tue, Jun 28, 2016 at 6:30 PM, Peter Otten <__peter__@web.de> wrote:
> Does chardet ever return an encoding that fails to decode
> the line? Only in that case the "ignore" error handler would make sense.

Assuming the module the OP is using is functionally identical to the
one I use from the command line (which is implemented in Python), yes
it can. Usually what happens is that it detects something as an
ISO-8859-* when it's actually the corresponding Windows codepage; if
you try to decode it that way, you end up with a handful of byte
values that don't correctly decode. I have a "cdless" command that
does a chardet, decodes the file, re-encodes as UTF-8, and pipes the
result into less(1); great way to figure out what encoding something
is (if it gets it wrong, it's usually really obvious to a human). It
has a magic second parameter "win" to switch from ISO-8859 to Windows
encoding - ISO-8859-1 becomes Windows-1252, -2 becomes 1250, etc.
Additionally, chardet often returns "MacCyrillic" for files that are
actually encoded Windows-1256 (Arabic). So, yes, it's definitely
possible for chardet to pick something that you can't actually decode
with.

For the OP's situation, frankly, I doubt there'll be anything other
than UTF-8, Latin-1, and CP-1252. The chances that someone casually
mixes CP-1252 with (say) CP-1254 would be vanishingly small. So the
simple decode of "UTF-8, or failing that, 1252" is probably going to
give correct results for most of the content. The trick is figuring
out a correct boundary for the check; line-by-line may be sufficient,
or it may not.

ChrisA

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 09:25 +0200
  Re: Processing text data with different encodings Chris Angelico <rosuav@gmail.com> - 2016-06-28 17:46 +1000
    Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 10:35 +0200
      Re: Processing text data with different encodings Steven D'Aprano <steve@pearwood.info> - 2016-06-28 20:29 +1000
        Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 12:37 +0200
          Re: Processing text data with different encodings Chris Angelico <rosuav@gmail.com> - 2016-06-28 21:09 +1000
  Re: Processing text data with different encodings Peter Otten <__peter__@web.de> - 2016-06-28 10:30 +0200
    Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 12:17 +0200
      Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 12:44 +0200
      Re: Processing text data with different encodings Steven D'Aprano <steve@pearwood.info> - 2016-06-28 21:26 +1000
        Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 14:30 +0200
          Re: Processing text data with different encodings Steven D'Aprano <steve@pearwood.info> - 2016-06-29 00:52 +1000
            Re: Processing text data with different encodings Random832 <random832@fastmail.com> - 2016-06-28 11:01 -0400
            Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 17:52 +0200
              Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-29 06:45 +0200
          Re: Processing text data with different encodings Steven D'Aprano <steve@pearwood.info> - 2016-06-29 01:11 +1000
      Re: Processing text data with different encodings Peter Otten <__peter__@web.de> - 2016-06-28 13:31 +0200
        Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 15:16 +0200
  Re: Processing text data with different encodings Chris Angelico <rosuav@gmail.com> - 2016-06-28 20:25 +1000
  Re: Processing text data with different encodings Random832 <random832@fastmail.com> - 2016-06-28 11:52 -0400
  Re: Processing text data with different encodings Chris Angelico <rosuav@gmail.com> - 2016-06-29 04:03 +1000

csiph-web