Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #110666 > unrolled thread

Processing text data with different encodings

Back to article view | Back to comp.lang.python

Contents

  Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 09:25 +0200
    Re: Processing text data with different encodings Chris Angelico <rosuav@gmail.com> - 2016-06-28 17:46 +1000
      Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 10:35 +0200
        Re: Processing text data with different encodings Steven D'Aprano <steve@pearwood.info> - 2016-06-28 20:29 +1000
          Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 12:37 +0200
            Re: Processing text data with different encodings Chris Angelico <rosuav@gmail.com> - 2016-06-28 21:09 +1000
    Re: Processing text data with different encodings Peter Otten <__peter__@web.de> - 2016-06-28 10:30 +0200
      Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 12:17 +0200
        Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 12:44 +0200
        Re: Processing text data with different encodings Steven D'Aprano <steve@pearwood.info> - 2016-06-28 21:26 +1000
          Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 14:30 +0200
            Re: Processing text data with different encodings Steven D'Aprano <steve@pearwood.info> - 2016-06-29 00:52 +1000
              Re: Processing text data with different encodings Random832 <random832@fastmail.com> - 2016-06-28 11:01 -0400
              Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 17:52 +0200
                Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-29 06:45 +0200
            Re: Processing text data with different encodings Steven D'Aprano <steve@pearwood.info> - 2016-06-29 01:11 +1000
        Re: Processing text data with different encodings Peter Otten <__peter__@web.de> - 2016-06-28 13:31 +0200
          Re: Processing text data with different encodings Michael Welle <mwe012008@gmx.net> - 2016-06-28 15:16 +0200
    Re: Processing text data with different encodings Chris Angelico <rosuav@gmail.com> - 2016-06-28 20:25 +1000
    Re: Processing text data with different encodings Random832 <random832@fastmail.com> - 2016-06-28 11:52 -0400
    Re: Processing text data with different encodings Chris Angelico <rosuav@gmail.com> - 2016-06-29 04:03 +1000

Page 1 of 2  [1] 2  Next page →

#110666 — Processing text data with different encodings

Hello,

I want to use Python 3 to process data, that unfortunately come with
different encodings. So far I have found ascii, iso-8859, utf-8,
windows-1252 and maybe some more in the same file (don't ask...). I read
the data via sys.stdin and the idea is to read a line, detect the
current encoding, hit it until it looks like utf-8 and then go on with
the next line of input:


import cchardet

for line in sys.stdin.buffer:

    encoding = cchardet.detect(line)['encoding']
    line = line.decode(encoding, 'ignore')\
               .encode('UTF-8').decode('UTF-8', 'ignore')


After that line should be a string. The logging module and some others
choke on line: UnicodeEncodeError: 'charmap' codec can't encode
character. What would be a right approach to tackle that problem
(assuming that I can't change the input data)?

Regards
hmw

[toc] | [next] | [standalone]

#110667

On Tue, Jun 28, 2016 at 5:25 PM, Michael Welle <mwe012008@gmx.net> wrote:
> I want to use Python 3 to process data, that unfortunately come with
> different encodings. So far I have found ascii, iso-8859, utf-8,
> windows-1252 and maybe some more in the same file (don't ask...). I read
> the data via sys.stdin and the idea is to read a line, detect the
> current encoding, hit it until it looks like utf-8 and then go on with
> the next line of input:
>
>
> import cchardet
>
> for line in sys.stdin.buffer:
>
>     encoding = cchardet.detect(line)['encoding']
>     line = line.decode(encoding, 'ignore')\
>                .encode('UTF-8').decode('UTF-8', 'ignore')
>
>
> After that line should be a string. The logging module and some others
> choke on line: UnicodeEncodeError: 'charmap' codec can't encode
> character. What would be a right approach to tackle that problem
> (assuming that I can't change the input data)?

This is the exact sort of "ewwww" that I have to cope with in my MUD
client. Sometimes it gets sent UTF-8, other times it gets sent...
uhhhh... some eight-bit encoding, most likely either 8859 or 1252 (but
could theoretically be anything). The way I cope with it is to do a
line-by-line decode, similar to what you're doing, but with a much
simpler algorithm - something like this:

for line in <binary source>:
    try:
        line = line.decode("UTF-8")
    except UnicodeDecodeError:
        line = line.decode("1252")
    yield line

There's no need to chardet for UTF-8; if you successfully decode the
text, it's almost certainly correct. (This includes pure ASCII text,
which would also decode successfully and correctly as ISO-8859 or
Windows-1252.)

You shouldn't need this complicated triple-encode dance. Just decode
it once and work with text from there on. Ideally, you should be using
Python 3, where "work[ing] with text" is exactly how most of the code
wants to work; if not, resign yourself to reprs with u-prefixes, and
work with Unicode strings anyway. It'll save you a lot of trouble.

ChrisA

[toc] | [prev] | [next] | [standalone]

#110672

Hello,

Chris Angelico <rosuav@gmail.com> writes:

> On Tue, Jun 28, 2016 at 5:25 PM, Michael Welle <mwe012008@gmx.net> wrote:
[...]
> This is the exact sort of "ewwww" that I have to cope with in my MUD
> client. Sometimes it gets sent UTF-8, other times it gets sent...
> uhhhh... some eight-bit encoding, most likely either 8859 or 1252 (but
> could theoretically be anything).
my original data is email. The mail header says it's utf-8, but you will
find three or four different encodings in one email. I think at the
sending side they just glue different text fragments from different
sources together without thinking about the encoding.


> The way I cope with it is to do a
> line-by-line decode, similar to what you're doing, but with a much
> simpler algorithm - something like this:
>
> for line in <binary source>:
>     try:
>         line = line.decode("UTF-8")
>     except UnicodeDecodeError:
>         line = line.decode("1252")
>     yield line
>
> There's no need to chardet for UTF-8; if you successfully decode the
> text, it's almost certainly correct. (This includes pure ASCII text,
> which would also decode successfully and correctly as ISO-8859 or
> Windows-1252.)
>
> You shouldn't need this complicated triple-encode dance. Just decode
> it once and work with text from there on. Ideally, you should be using
yepp, makes sense.

Interestingly, using .decode('utf-8') until it fails works better than
looking at the encoding and using a specific .decode(enc). I have to dive
a bit deeper into it to find out why. I looked at a few samples that
cchardet detected as 1252 encoding and at the first glance they looked
like 1252.

Thanks
hmw

[toc] | [prev] | [next] | [standalone]

#110677

On Tue, 28 Jun 2016 06:35 pm, Michael Welle wrote:

> my original data is email. The mail header says it's utf-8, but you will
> find three or four different encodings in one email. I think at the
> sending side they just glue different text fragments from different
> sources together without thinking about the encoding.

Is this spam? In my experience, the only email that is that badly
constructed is spam. I can't imagine how it could be email from a person,
coming from a mail client like Thunderbird or Outlook.





-- 
Steven
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

[toc] | [prev] | [next] | [standalone]

#110680

Hello,

Steven D'Aprano <steve@pearwood.info> writes:

> On Tue, 28 Jun 2016 06:35 pm, Michael Welle wrote:
>
>> my original data is email. The mail header says it's utf-8, but you will
>> find three or four different encodings in one email. I think at the
>> sending side they just glue different text fragments from different
>> sources together without thinking about the encoding.
>
> Is this spam? In my experience, the only email that is that badly
> constructed is spam. I can't imagine how it could be email from a person,
> coming from a mail client like Thunderbird or Outlook.
it's mail from an international company. It's not generated by a person
using an ordinary email client. Other than that your are right ;).

Regards
hmw

[toc] | [prev] | [next] | [standalone]

#110683

On Tue, Jun 28, 2016 at 8:37 PM, Michael Welle <mwe012008@gmx.net> wrote:
> Steven D'Aprano <steve@pearwood.info> writes:
>
>> On Tue, 28 Jun 2016 06:35 pm, Michael Welle wrote:
>>
>>> my original data is email. The mail header says it's utf-8, but you will
>>> find three or four different encodings in one email. I think at the
>>> sending side they just glue different text fragments from different
>>> sources together without thinking about the encoding.
>>
>> Is this spam? In my experience, the only email that is that badly
>> constructed is spam. I can't imagine how it could be email from a person,
>> coming from a mail client like Thunderbird or Outlook.
> it's mail from an international company. It's not generated by a person
> using an ordinary email client. Other than that your are right ;).

So..... buggy commercial email. Great. Just brilliant.

Can you bill them for your time developing this hack?

ChrisA

[toc] | [prev] | [next] | [standalone]

#110670

Michael Welle wrote:

> Hello,
> 
> I want to use Python 3 to process data, that unfortunately come with
> different encodings. So far I have found ascii, iso-8859, utf-8,
> windows-1252 and maybe some more in the same file (don't ask...). I read
> the data via sys.stdin and the idea is to read a line, detect the
> current encoding, hit it until it looks like utf-8 and then go on with
> the next line of input:
> 
> 
> import cchardet
> 
> for line in sys.stdin.buffer:
> 
>     encoding = cchardet.detect(line)['encoding']
>     line = line.decode(encoding, 'ignore')\
>                .encode('UTF-8').decode('UTF-8', 'ignore')

Here the last decode('UTF-8', 'ignore') undoes the preceding 
encode('UTF-8'); therefore

      line = line.decode(encoding, 'ignore')

should suffice. Does chardet ever return an encoding that fails to decode 
the line? Only in that case the "ignore" error handler would make sense. I 
expect that

for line in sys.stdin.buffer:
    encoding = cchardet.detect(line)['encoding']
    line = line.decode(encoding)

will work if you don't want to use the alternative suggested by Chris.

> After that line should be a string. The logging module and some others
> choke on line: UnicodeEncodeError: 'charmap' codec can't encode
> character. What would be a right approach to tackle that problem
> (assuming that I can't change the input data)?

It looks like you are trying to write the unicode you have generated above 
into a file using iso-8859-1 or similar:

$ cat log_unicode.py
import logging
LOGGER = logging.getLogger()
LOGGER.addHandler(logging.FileHandler("tmp.txt", encoding="ISO-8859-1"))
LOGGER.critical("\N{PILE OF POO}")
$ python3 log_unicode.py 
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.4/logging/__init__.py", line 980, in emit
    stream.write(msg)
UnicodeEncodeError: 'latin-1' codec can't encode character '\U0001f4a9' in 
position 0: ordinal not in range(256)
Call stack:
  File "log_unicode.py", line 5, in <module>
    LOGGER.critical("\N{PILE OF POO}")
Message: '💩'
Arguments: ()

If my assumption is correct you can either change the target file's encoding 
to UTF-8 or change the error handling strategy to ignore or something else. 
I didn't find an official way, so here's a minimal example:

$ rm tmp.txt
$ cat log_unicode.py
import logging

class FileHandler(logging.FileHandler):
    def _open(self):
        return open(
            self.baseFilename, self.mode, encoding=self.encoding,
            errors="xmlcharrefreplace")

LOGGER = logging.getLogger()
LOGGER.addHandler(FileHandler("tmp.txt", encoding="ISO-8859-1"))
LOGGER.critical("\N{PILE OF POO}")
$ python3 log_unicode.py 
$ cat tmp.txt
&#128169;

A real program would of course override the initializer...

[toc] | [prev] | [next] | [standalone]

#110675

Hello,

Peter Otten <__peter__@web.de> writes:

> Michael Welle wrote:
>
>> Hello,
>> 
>> I want to use Python 3 to process data, that unfortunately come with
>> different encodings. So far I have found ascii, iso-8859, utf-8,
>> windows-1252 and maybe some more in the same file (don't ask...). I read
>> the data via sys.stdin and the idea is to read a line, detect the
>> current encoding, hit it until it looks like utf-8 and then go on with
>> the next line of input:
>> 
>> 
>> import cchardet
>> 
>> for line in sys.stdin.buffer:
>> 
>>     encoding = cchardet.detect(line)['encoding']
>>     line = line.decode(encoding, 'ignore')\
>>                .encode('UTF-8').decode('UTF-8', 'ignore')
>
> Here the last decode('UTF-8', 'ignore') undoes the preceding 
> encode('UTF-8'); therefore
>
>       line = line.decode(encoding, 'ignore')
>
> should suffice.
yepp.


> Does chardet ever return an encoding that fails to decode 
> the line? Only in that case the "ignore" error handler would make sense. I 
> expect that
>
> for line in sys.stdin.buffer:
>     encoding = cchardet.detect(line)['encoding']
>     line = line.decode(encoding)
>
> will work if you don't want to use the alternative suggested by Chris.
After a bit more 'fiddling' I found out that all test cases work if I
use .decode('utf-8') on the incoming bytes. In my first approach I tried
to find out at what I was looking and then used a specific .decode, e.g.
.decode('windows-1252'). That were the trouble makers.  

With your help, I fixed logging. Somehow I had in mind that the
logging module would do the right thing if I don't specify the encoding.
Well, setting the encoding explicitly to utf-8 changes the behaviour.

If I use decode('windows-1252') on a bit of text I still have trouble to
understand what's happening. For instance, there is an u umlaut in the
1252 encoded portion of the input text. That character is 0xfc in hex.
After applying .decode('windows-1252') and logging it, the log contains
a mangled character with hex codes 0xc3 0x20. If I do the same with
.decode('utf-8'), the result is a working u umlaut with 0xfc in the log.

On the other hand, if I try the following in the interactive
interpreter:

Here I have a few bytes that can be interpreted as a 1252 encoded string
and I command the interpreter to show me the string, right?

>>> e=b'\xe4'
>>> e.decode('1252')
'ä'

Now, I can't to this, because 0xe4 isn't valid utf-8:
>>> e.decode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 0:
unexpected end of data

But why is it different in my actual script? I guess the assumption that
what I am reading from sys.stdin.buffer is the same as what is in the
file, that I pipe into the script, is wrong?

Regards
hmw

[toc] | [prev] | [next] | [standalone]

#110681

Hello,

Michael Welle <mwe012008@gmx.net> writes:
[...]
> understand what's happening. For instance, there is an u umlaut in the
> 1252 encoded portion of the input text. That character is 0xfc in hex.
> After applying .decode('windows-1252') and logging it, the log contains
> a mangled character with hex codes 0xc3 0x20. If I do the same with
> .decode('utf-8'), the result is a working u umlaut with 0xfc in the log.
>
> On the other hand, if I try the following in the interactive
> interpreter:
>
> Here I have a few bytes that can be interpreted as a 1252 encoded string
> and I command the interpreter to show me the string, right?
>
>>>> e=b'\xe4'
>>>> e.decode('1252')
> 'ä'
>
> Now, I can't to this, because 0xe4 isn't valid utf-8:
>>>> e.decode('utf-8')
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 0:
> unexpected end of data
>
> But why is it different in my actual script? I guess the assumption that
> what I am reading from sys.stdin.buffer is the same as what is in the
> file, that I pipe into the script, is wrong?
I made a minimal example and that revealed that the input 'ü' and
.decode('windows-1252') works as expected. The input 'üx' results in a
mangled output. Hmm.

Regards
hmw

[toc] | [prev] | [next] | [standalone]

#110684

On Tue, 28 Jun 2016 08:17 pm, Michael Welle wrote:

> After a bit more 'fiddling' I found out that all test cases work if I
> use .decode('utf-8') on the incoming bytes. In my first approach I tried
> to find out at what I was looking and then used a specific .decode, e.g.
> .decode('windows-1252'). That were the trouble makers.

Remember that chardet's detection is based on statistics and heuristics and
cannot be considered 100% reliable. Normally I would expect that chardet
would guess two or three encodings. If the first fails, you might want to
check the others.

Also remember that chardet works best with large amounts of text, like an
entire webpage. If you pass it a single byte, or even a few bytes, the
results will likely be no better than whatever encoding the chardet
developer decided to use as the default:

"If there's not enough data to guess, just return Win-1252, because that's
pretty common..."



> With your help, I fixed logging. Somehow I had in mind that the
> logging module would do the right thing if I don't specify the encoding.
> Well, setting the encoding explicitly to utf-8 changes the behaviour.

I would expect that logging will do the right thing if you pass it text
strings and have set the encoding to UTF-8.



> If I use decode('windows-1252') on a bit of text 

You cannot decode text. Text is ENCODED to bytes, and bytes are DECODED to
text.

> I still have trouble to understand what's happening.

> For instance, there is an u umlaut in the 
> 1252 encoded portion of the input text.

You don't know that. If the input is *bytes*, then all you know is the byte
values. What they mean is anyone's guess unless the name of the encoding is
transmitted separately. You can be reasonably sure that the bytes are
mostly ASCII, because it's email and nobody sends email in EBCDIC, so if
you see a byte 0x41, you can be sure it represents an 'A'. But outside of
the ASCII range, you're on shaky ground.

If the specified encoding is correct, then everything works well: the email
says it is UTF-8, and sure enough it is UTF-8. But if the specified
encoding is wrong, you're in trouble.

You only think the encoding is Windows-1252 because Chardet has guessed
that. But it's not infallible and maybe it has got it wrong. Especially if
your input is made up of a lot of bytes from all sorts of different
encodings, that may be confusing Chardet.


> That character is 0xfc in hex. 

No. Byte 0xFC represents ü if your guess about the encoding is correct. If
the encoding truly is Windows-1252, or Latin-1, then byte 0xFC will mean ü.
(And some others.) If the source is Western European, that might be a good
guess.

But if the encoding actually is (let's say):

- ISO-8859-5 (Cyrillic), then the byte represents ќ

- ISO-8859-7 (Greek), then the byte represents ό

- MacRoman (Apple Macintosh), then the byte represents ¸

(That last one is not a comma, but a cedilla.)


> After applying .decode('windows-1252') and logging it, the log contains
> a mangled character with hex codes 0xc3 0x20. 

I think you are misunderstanding what you are looking at. How are you seeing
that? 0x20 will be a space in most encodings.

(1) How are you opening the log file in Python? Do you specify an encoding?

(2) How are you writing to the log file?

(3) What are you using to read the log file outside of Python? How do you
know the hex codes?


I don't know any way you can start with the character ü and write it to a
file and get bytes 0xc3 0x20. Maybe somebody else will think of something,
but to me, that seems impossible.


> If I do the same with 
> .decode('utf-8'), the result is a working u umlaut with 0xfc in the log.

That suggests that you have opened the log file using Latin-1 or
Windows-1252 as the encoding. You shouldn't do that. Unless you have a good
reason to do otherwise (in other words, for experts only) you should always
use UTF-8 for writing.


> On the other hand, if I try the following in the interactive
> interpreter:
> 
> Here I have a few bytes that can be interpreted as a 1252 encoded string
> and I command the interpreter to show me the string, right?
> 
>>>> e=b'\xe4'

That's ONE byte, not a few.

>>>> e.decode('1252')
> 'ä'

Right -- that means that byte 0xE4 represents ä in Windows-1252, also in
Latin-1 and some others. But:

py> e.decode('iso-8859-7')  # Greek
'δ'
py> e.decode('iso-8859-8')  # Hebrew
'ה'
py> e.decode('iso-8859-6')  # Arabic
'ل'
py> e.decode('MacRoman')
'‰'
py> e.decode('iso-8859-5')
'ф'

So if you find a byte 0xE4 in a file, and don't know where it came from, you
don't know what it means. If you can guess it came from Russia, then it
might be a ф. If you think it came from a Macintosh prior to OS X, then it
probably means a per-mill sign ‰.


> Now, I can't to this, because 0xe4 isn't valid utf-8:
>>>> e.decode('utf-8')
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 0:
> unexpected end of data

Correct.


> But why is it different in my actual script? 

Without seeing your script, it's hard to say what you are actually doing.


> I guess the assumption that 
> what I am reading from sys.stdin.buffer is the same as what is in the
> file, that I pipe into the script, is wrong?

I wouldn't rule that out, but more likely the issue lies elsewhere, in your
own code.



-- 
Steven
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

[toc] | [prev] | [next] | [standalone]

#110690

Hello,

Steven D'Aprano <steve@pearwood.info> writes:

> On Tue, 28 Jun 2016 08:17 pm, Michael Welle wrote:
>
>> After a bit more 'fiddling' I found out that all test cases work if I
>> use .decode('utf-8') on the incoming bytes. In my first approach I tried
>> to find out at what I was looking and then used a specific .decode, e.g.
>> .decode('windows-1252'). That were the trouble makers.
>
> Remember that chardet's detection is based on statistics and heuristics and
> cannot be considered 100% reliable. Normally I would expect that chardet
> would guess two or three encodings. If the first fails, you might want to
> check the others.
jepp, I considered that and checked it with a hex editor.


> Also remember that chardet works best with large amounts of text, like an
> entire webpage. If you pass it a single byte, or even a few bytes, the
> results will likely be no better than whatever encoding the chardet
> developer decided to use as the default:
>
> "If there's not enough data to guess, just return Win-1252, because that's
> pretty common..."
That could be a problem, indeed. The text samples I checked in the real
application are quite small. But my hand-crafted minimal example (see my
other mail/posting) don't utilise cchardet.


>> With your help, I fixed logging. Somehow I had in mind that the
>> logging module would do the right thing if I don't specify the encoding.
>> Well, setting the encoding explicitly to utf-8 changes the behaviour.
>
> I would expect that logging will do the right thing if you pass it text
> strings and have set the encoding to UTF-8.
>
>
>
>> If I use decode('windows-1252') on a bit of text 
>
> You cannot decode text. Text is ENCODED to bytes, and bytes are DECODED to
> text.
Sorry, sloppy wording. I had a piece of the input test in mind.


>> I still have trouble to understand what's happening.
>
>> For instance, there is an u umlaut in the 
>> 1252 encoded portion of the input text.
>
> You don't know that. If the input is *bytes*, then all you know is the byte
> values. What they mean is anyone's guess unless the name of the encoding is
> transmitted separately. You can be reasonably sure that the bytes are
> mostly ASCII, because it's email and nobody sends email in EBCDIC, so if
> you see a byte 0x41, you can be sure it represents an 'A'. But outside of
> the ASCII range, you're on shaky ground.
I look at the hex values of the bytes, get the win-1252 table and
translate the bytes to chars. If the result makes sense, it's win-1252
(and maybe others, if the tables overlap). So in that sense I know what
I have. I least for this experiments, when I can control the input.


> If the specified encoding is correct, then everything works well: the email
> says it is UTF-8, and sure enough it is UTF-8. But if the specified
> encoding is wrong, you're in trouble.
>
> You only think the encoding is Windows-1252 because Chardet has guessed
> that.
No, I think that because I looked at the input in a hex editor and
translated the bytes manually using a character table and got something
that makes sense. In that case cchardet came to the same conclusion. But
it's clear that it can't guess everything.


> But it's not infallible and maybe it has got it wrong. Especially if
> your input is made up of a lot of bytes from all sorts of different
> encodings, that may be confusing Chardet.
Sure.


>> That character is 0xfc in hex. 
>
> No. Byte 0xFC represents ü if your guess about the encoding is
> correct. If
> the encoding truly is Windows-1252, or Latin-1, then byte 0xFC will mean ü.
> (And some others.) If the source is Western European, that might be a good
> guess.
That's what I said. u umlaut == ü. I have input, that is an u umlaut in
windows-1252 encoding and the hex value is 0xfc.


> But if the encoding actually is (let's say):
>
> - ISO-8859-5 (Cyrillic), then the byte represents ќ
>
> - ISO-8859-7 (Greek), then the byte represents ό
>
> - MacRoman (Apple Macintosh), then the byte represents ¸
>
> (That last one is not a comma, but a cedilla.)
Yes, but that doesn't matter. Because in this example I am sure that I
deal with windows-1252.


>> After applying .decode('windows-1252') and logging it, the log contains
>> a mangled character with hex codes 0xc3 0x20. 
>
> I think you are misunderstanding what you are looking at. How are you seeing
> that? 0x20 will be a space in most encodings.
Again, I used an hex editor and hd.


> (1) How are you opening the log file in Python? Do you specify an
> encoding?
Well, I use the logging module. In my very first posting I didn't
specify an encoding. Later I changed the encoding to utf-8. The details
of opening the log file can be found somewhere in the logging module


> (2) How are you writing to the log file?
I use the provided functions for writing records with the given
severity.


> (3) What are you using to read the log file outside of Python? How do you
> know the hex codes?
I use emacs, hexlify-mode and hd.


> I don't know any way you can start with the character ü and write it to a
> file and get bytes 0xc3 0x20. Maybe somebody else will think of something,
> but to me, that seems impossible.
>
>
>> If I do the same with 
>> .decode('utf-8'), the result is a working u umlaut with 0xfc in the log.
>
> That suggests that you have opened the log file using Latin-1 or
> Windows-1252 as the encoding. You shouldn't do that. Unless you have a good
> reason to do otherwise (in other words, for experts only) you should always
> use UTF-8 for writing.
That's how I open the log:

LOGGER.addHandler(logging.FileHandler("tmp.txt", encoding="utf-8"))


>> On the other hand, if I try the following in the interactive
>> interpreter:
>> 
>> Here I have a few bytes that can be interpreted as a 1252 encoded string
>> and I command the interpreter to show me the string, right?
>> 
>>>>> e=b'\xe4'
>
> That's ONE byte, not a few.
>
>>>>> e.decode('1252')
>> 'ä'
>
> Right -- that means that byte 0xE4 represents ä in Windows-1252, also in
> Latin-1 and some others. But:
>
> py> e.decode('iso-8859-7')  # Greek
> 'δ'
> py> e.decode('iso-8859-8')  # Hebrew
> 'ה'
> py> e.decode('iso-8859-6')  # Arabic
> 'ل'
> py> e.decode('MacRoman')
> '‰'
> py> e.decode('iso-8859-5')
> 'ф'
>
> So if you find a byte 0xE4 in a file, and don't know where it came from, you
> don't know what it means. If you can guess it came from Russia, then it
> might be a ф. If you think it came from a Macintosh prior to OS X, then it
> probably means a per-mill sign ‰.
Yes, as I said above. I wonder if I'm the only one, who likes to go from a
general problem to a specialised one, to find out what the problem is?
It's clear that in the general problem I don't know what the input data
is. But when I use hand-crafted data like 0xe4 for test, I have a
certain encoding in mind. How should I come up with 0xe4 else? Sorry for
not making that clear enough.


>> Now, I can't to this, because 0xe4 isn't valid utf-8:
>>>>> e.decode('utf-8')
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 0:
>> unexpected end of data
>
> Correct.
At least I know how decode and encode should work ;).


>> But why is it different in my actual script? 
>
> Without seeing your script, it's hard to say what you are actually doing.

I changed the code from my initial mail to:

LOGGER = logging.getLogger()
LOGGER.addHandler(logging.FileHandler("tmp.txt", encoding="utf-8"))

for l in sys.stdin.buffer:
    l = l.decode('utf-8')
    LOGGER.critical(l)
    

>> I guess the assumption that 
>> what I am reading from sys.stdin.buffer is the same as what is in the
>> file, that I pipe into the script, is wrong?
>
> I wouldn't rule that out, but more likely the issue lies elsewhere, in your
> own code.
Yepp, I think so, unfortunately. I think I will put it aside for a few
hours and then come back with fresh eyes.

Thanks
hmw

[toc] | [prev] | [next] | [standalone]

#110705

On Tue, 28 Jun 2016 10:30 pm, Michael Welle wrote:

> I look at the hex values of the bytes, get the win-1252 table and
> translate the bytes to chars. If the result makes sense, it's win-1252
> (and maybe others, if the tables overlap). So in that sense I know what
> I have. I least for this experiments, when I can control the input.

So let me see if I understand what you are doing.

You create an input file for testing, let's call it "test.txt". In that test
file, you control the input, so you place an ü and save it using
Windows-1252 encoding. Then you open the test file and see a byte 0xFC.

Then you open an email, encoded using a completely unknown encoding, but
claiming to be UTF-8, and see a byte 0xFC. And from this you conclude that
the encoding must be Windows-1252 and the unknown character must be ü.

Is that right?

Maybe I'm misunderstanding you, but frankly from your description I have
very little confidence that the unknown encoding is Windows-1252,
especially given your earlier comment:

    "you will find THREE OR FOUR different encodings in one email. 
    I think at the sending side they just glue different text 
    fragments from different sources together without thinking 
    about the encoding"

But I'm not going to argue any more. Maybe I've misunderstood you, and what
you have done makes perfect sense. Maybe there's only one encoding, 1252,
not three or four. Windows-1252 is a very common encoding, so perhaps you
are right. The worst that will happen is that you will get mojibake.

So I will accept that the encoding is Windows-1252.


>>> After applying .decode('windows-1252') and logging it, the log contains
>>> a mangled character with hex codes 0xc3 0x20.
>>
>> I think you are misunderstanding what you are looking at. How are you
>> seeing that? 0x20 will be a space in most encodings.
>
> Again, I used an hex editor and hd.

hd? Also known as hexdump?

http://www.unix.com/man-page/Linux/1/hd/

I ask because there is also another tool, hdtool, sometimes called hd, used
for formatting hard disks. I assume you're not using that :-)


>> (1) How are you opening the log file in Python? Do you specify an
>> encoding?
>
> Well, I use the logging module. In my very first posting I didn't
> specify an encoding. Later I changed the encoding to utf-8. The details
> of opening the log file can be found somewhere in the logging module

The mystery 0xC3 0x20 hex codes -- what encoding were you using at the time
you logged them? Because there is no way that I can see that you can get
0xC3 0x20 using UTF-8. It is an invalid sequence of bytes. If the logger
was using UTF-8, that implies either a bug in the logger, or disk
corruption.

Are you *sure* it is 0x20? If it were 0xC3 0xBC that would make perfect
sense. But 0xC3 0x20 is invalid UTF-8.


>> (2) How are you writing to the log file?
>
> I use the provided functions for writing records with the given
> severity.

Are you passing it a text string? A bytes string? Something else?


>> (3) What are you using to read the log file outside of Python? How do you
>> know the hex codes?
> 
> I use emacs, hexlify-mode and hd.

Okay.



>> I don't know any way you can start with the character ü and write it to a
>> file and get bytes 0xc3 0x20. Maybe somebody else will think of
>> something, but to me, that seems impossible.

That comment still stands.


>>> If I do the same with
>>> .decode('utf-8'), the result is a working u umlaut with 0xfc in the log.

But that is, I think, impossible. You must be misinterpreting what you are
seeing, or confusing output in the log when it used a different encoding.

py> 'ü'.encode('utf-8')
b'\xc3\xbc'

not 0xFC.


More to follow...




-- 
Steven
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

[toc] | [prev] | [next] | [standalone]

#110706

On Tue, Jun 28, 2016, at 10:52, Steven D'Aprano wrote:
>     "you will find THREE OR FOUR different encodings in one email. 
>     I think at the sending side they just glue different text 
>     fragments from different sources together without thinking 
>     about the encoding"
> 
> But I'm not going to argue any more. Maybe I've misunderstood you

Well, given that the other encodings he named were us-ascii and latin-1,
I think he was just counting windows-1252 as three different encodings
depending on what characters are present, as hyperbole.

[toc] | [prev] | [next] | [standalone]

#110713

Hello,

Steven D'Aprano <steve@pearwood.info> writes:

> On Tue, 28 Jun 2016 10:30 pm, Michael Welle wrote:
>
>> I look at the hex values of the bytes, get the win-1252 table and
>> translate the bytes to chars. If the result makes sense, it's win-1252
>> (and maybe others, if the tables overlap). So in that sense I know what
>> I have. I least for this experiments, when I can control the input.
>
> So let me see if I understand what you are doing.
>
> You create an input file for testing, let's call it "test.txt". In that test
> file, you control the input, so you place an ü and save it using
> Windows-1252 encoding. Then you open the test file and see a byte
> 0xFC.
yes.


> Then you open an email, encoded using a completely unknown encoding, but
> claiming to be UTF-8, and see a byte 0xFC. And from this you conclude that
> the encoding must be Windows-1252 and the unknown character must be ü.
>
> Is that right?
Noyes ;). No in case you think there is a (mental) connection between
creating an input file and the 'real' data. Yes, if you think I look at
a bunch of hex values, let's say enough values so that they can form a
word. Then I get the win-1252 table (or any table, utf-8 etc.) and
translate that bytes to characters. If the characters form a valid word
in the expected context, then it's safe to assume that I look at a piece
of text that is win-1252 encoded. Yes, there could be another encoding
that would give me a different, but still valid word. But the chances of
this to happen are very slim, I guess.

And yes, if I only had a byte, the 0xfc, and I would find it's a ü
interpreted as win-1252 and it's a smiley interpreted as another
encoding, and only the ü would make sense in the context (and
considering  very small blocks of bytes/text), then it's safe to assume
the block I'm looking at is win-1252. What's not safe to assume is to
guess the encoding of a bigger block of, let's say, 1k bytes just
because one of the 1k bytes is a 0xfc. 


> Maybe I'm misunderstanding you, but frankly from your description I have
> very little confidence that the unknown encoding is Windows-1252,
> especially given your earlier comment:
>
>     "you will find THREE OR FOUR different encodings in one email. 
>     I think at the sending side they just glue different text 
>     fragments from different sources together without thinking 
>     about the encoding"
Again, we (or at least I ;) are talking about different things in this
thread. My initial problem resulted from dealing with that crappy
emails. In the course of the discussion I broke that down to understand
if my expectation of encode/decode is wrong or if there is some other
error. I think it's reasonable not to use a bunch of 'random' bytes for
that process, but a defined set of bytes, that could be translated into a
valid text using win-1252. 


> But I'm not going to argue any more. Maybe I've misunderstood you, and what
> you have done makes perfect sense. Maybe there's only one encoding, 1252,
> not three or four. Windows-1252 is a very common encoding, so perhaps you
> are right. The worst that will happen is that you will get mojibake.
>
> So I will accept that the encoding is Windows-1252.
For me it makes perfect sense, but it could well be a brain fart ;).


>>>> After applying .decode('windows-1252') and logging it, the log contains
>>>> a mangled character with hex codes 0xc3 0x20.
>>>
>>> I think you are misunderstanding what you are looking at. How are you
>>> seeing that? 0x20 will be a space in most encodings.
>>
>> Again, I used an hex editor and hd.
>
> hd? Also known as hexdump?
>
> http://www.unix.com/man-page/Linux/1/hd/
>
> I ask because there is also another tool, hdtool, sometimes called hd, used
> for formatting hard disks. I assume you're not using that :-)
That would explain why my terminal all the sudden started reacting
funny ;). Yes, hexdump, od and the like are the tools I'm talking about. 



>>> (1) How are you opening the log file in Python? Do you specify an
>>> encoding?
>>
>> Well, I use the logging module. In my very first posting I didn't
>> specify an encoding. Later I changed the encoding to utf-8. The details
>> of opening the log file can be found somewhere in the logging module
>
> The mystery 0xC3 0x20 hex codes -- what encoding were you using at the time
> you logged them? Because there is no way that I can see that you can get
> 0xC3 0x20 using UTF-8. It is an invalid sequence of bytes. If the logger
> was using UTF-8, that implies either a bug in the logger, or disk
> corruption.
>
> Are you *sure* it is 0x20? If it were 0xC3 0xBC that would make perfect
> sense. But 0xC3 0x20 is invalid UTF-8.
Thinking about it, I'm puzzled, too. Unfortunately I have removed the data
files of the experiments. The logger should have been switched to utf-8
at that time, because it was after Peter's mail. Well, 0x20 might be a
space. 0xc3 could be valid win-1252, but why would it in the log?
There is a chance, of course, that I just made an error while looking at
the values. On the other hand I had the log open in an text editor and
there I saw something that wasn't readable.


>>> (2) How are you writing to the log file?
>>
>> I use the provided functions for writing records with the given
>> severity.
>
> Are you passing it a text string? A bytes string? Something else?
A text string.


[...]
>>> I don't know any way you can start with the character ü and write it to a
>>> file and get bytes 0xc3 0x20. Maybe somebody else will think of
>>> something, but to me, that seems impossible.
>
> That comment still stands.
Yepp.


>>>> If I do the same with
>>>> .decode('utf-8'), the result is a working u umlaut with 0xfc in the log.
>
> But that is, I think, impossible. You must be misinterpreting what you are
> seeing, or confusing output in the log when it used a different encoding.
Yes, I have the feeling I'm making a fool out of myself. But I just
don't know how. Yet. That's why I put that aside for some time. It's
evening in my timezone and I hope that I can grok it tomorrow morning
with fresh mind and eyes.

Thanks
hmw

[toc] | [prev] | [next] | [standalone]

#110747

Hello,


Michael Welle <mwe012008@gmx.net> writes:

> Hello,
>
> Steven D'Aprano <steve@pearwood.info> writes:
[...]
>>>>> If I do the same with
>>>>> .decode('utf-8'), the result is a working u umlaut with 0xfc in the log.
>>
>> But that is, I think, impossible. You must be misinterpreting what you are
>> seeing, or confusing output in the log when it used a different encoding.
> Yes, I have the feeling I'm making a fool out of myself. But I just
> don't know how. Yet. That's why I put that aside for some time. It's
> evening in my timezone and I hope that I can grok it tomorrow morning
> with fresh mind and eyes.
so, morning has come and I feel like a fool ;(. I've cleaned up yesterday's
experiments from the code, restarted the Python interpreter. I run the
tests in a non-unicode shell environment and... can't reproduce the problem.
The only change that is needed after my initial posting is switching the
logger to utf-8.

During development I usually use one Python instance per project, which
I rarely restart. Maybe I should, so that all the cruft that accumulates
over time gets reset.

Thank you for your help, Chris, Steven, Peter and random.

Regards
hmw

[toc] | [prev] | [next] | [standalone]

#110707

On Tue, 28 Jun 2016 10:30 pm, Michael Welle wrote:

> I changed the code from my initial mail to:
> 
> LOGGER = logging.getLogger()
> LOGGER.addHandler(logging.FileHandler("tmp.txt", encoding="utf-8"))
> 
> for l in sys.stdin.buffer:
>     l = l.decode('utf-8')
>     LOGGER.critical(l)

I imagine you're running this over input known to contain UTF-8 text?
Because if you run it over your emails with non-UTF8 content, you'll get an
exception.

I would try this:


for l in sys.stdin.buffer:
    l = l.decode('utf-8', errors='surrogateescape')
    print(repr(l))  # or log it, whichever you prefer



If I try simulating that, you'll see the output:


py> buffer = []
py> buffer.append('abüd\n'.encode('utf-8'))
py> buffer.append('abüd\n'.encode('utf-8'))
py> buffer.append('abüd\n'.encode('latin-1'))
py> buffer.append('abüd\n'.encode('utf-8'))
py> buffer
[b'ab\xc3\xbcd\n', b'ab\xc3\xbcd\n', b'ab\xfcd\n', b'ab\xc3\xbcd\n']
py> for l in buffer:  #sys.stdin.buffer:
...     l = l.decode('utf-8', errors='surrogateescape')
...     print(repr(l))
...
'abüd\n'
'abüd\n'
'ab\udcfcd\n'
'abüd\n'


See the second last line? The \udcfc code point is a surrogate, encoding
the "bad byte" \xfc. See the docs for further details.

Alternatively, you could try:

for l in sys.stdin.buffer:
    try:
        l = l.decode('utf-8', errors='strict')
    except UnicodeDecodeError:
        l = l.decode('latin1')  # May generate mojibake.
    print(repr(l))  # or log it, whichever you prefer


This version should give satisfactory results if the email actually does
contain lines of Latin-1 (or Windows-1252 if you prefer) mixed in with the
UTF-8. If not, it will generate mojibake, which may be acceptable to your
users.





-- 
Steven
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

[toc] | [prev] | [next] | [standalone]

#110686

Michael Welle wrote:

> With your help, I fixed logging. Somehow I had in mind that the
> logging module would do the right thing if I don't specify the encoding.

The default encoding depends on the environment (and platform):

$ touch tmp.txt
$ python3 -c 'print(open("tmp.txt").encoding)'
UTF-8
$ LANG=C python3 -c 'print(open("tmp.txt").encoding)'
ANSI_X3.4-1968

> Well, setting the encoding explicitly to utf-8 changes the behaviour.
> 
> If I use decode('windows-1252') on a bit of text I still have trouble to
> understand what's happening. For instance, there is an u umlaut in the
> 1252 encoded portion of the input text. That character is 0xfc in hex.
> After applying .decode('windows-1252') and logging it, the log contains
> a mangled character with hex codes 0xc3 0x20. If I do the same with
> .decode('utf-8'), the result is a working u umlaut with 0xfc in the log.
> 
> On the other hand, if I try the following in the interactive
> interpreter:
> 
> Here I have a few bytes that can be interpreted as a 1252 encoded string
> and I command the interpreter to show me the string, right?
> 
>>>> e=b'\xe4'
>>>> e.decode('1252')
> 'ä'
> 
> Now, I can't to this, because 0xe4 isn't valid utf-8:
>>>> e.decode('utf-8')
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 0:
> unexpected end of data
> 
> But why is it different in my actual script? I guess the assumption that
> what I am reading from sys.stdin.buffer is the same as what is in the
> file, that I pipe into the script, is wrong?

The situation is simple; the string consists of code points, but the file 
may only contain bytes. When reading a string from a file the bytes read 
need decoding, and before writing a string to a file it must be encoded.

What byte sequence denotes a specific code point depends on the encoding.

This is always the case, i. e. if you look at a UTF-8-encoded file with an 
editor that expects cp1252 you will see

>>> in_the_file = "ä".encode("utf-8")
>>> in_the_file
b'\xc3\xa4'
>>> what_the_editor_shows = in_the_file.decode("cp1252")
>>> print(what_the_editor_shows)
ä

On the other hand if you look at a cp1252-encoded file decoding the data as 
UTF-8 you will likely get an error because the byte

>>> "ä".encode("cp1252")
b'\xe4'

alone is not valid UTF-8. As part of a sequence the data may still be 
ambiguous. If you were to write an a-umlaut followed by two euro signs using 
cp1252

>>> in_the_file = '䀀'.encode("cp1252")

an editor expecting UTF-8 would show

>>> in_the_file.decode("utf-8")
'䀀'

[toc] | [prev] | [next] | [standalone]

#110689

Hello,

Peter Otten <__peter__@web.de> writes:

> Michael Welle wrote:
>
>> With your help, I fixed logging. Somehow I had in mind that the
>> logging module would do the right thing if I don't specify the encoding.
>
> The default encoding depends on the environment (and platform):
>
> $ touch tmp.txt
> $ python3 -c 'print(open("tmp.txt").encoding)'
> UTF-8
> $ LANG=C python3 -c 'print(open("tmp.txt").encoding)'
> ANSI_X3.4-1968
I see. Seems to make sense to always make the encoding explicite.
Imagine running a script in a user context and from a scheduler.

Thanks
hmw

[toc] | [prev] | [next] | [standalone]

#110676

On Tue, Jun 28, 2016 at 6:30 PM, Peter Otten <__peter__@web.de> wrote:
> Does chardet ever return an encoding that fails to decode
> the line? Only in that case the "ignore" error handler would make sense.

Assuming the module the OP is using is functionally identical to the
one I use from the command line (which is implemented in Python), yes
it can. Usually what happens is that it detects something as an
ISO-8859-* when it's actually the corresponding Windows codepage; if
you try to decode it that way, you end up with a handful of byte
values that don't correctly decode. I have a "cdless" command that
does a chardet, decodes the file, re-encodes as UTF-8, and pipes the
result into less(1); great way to figure out what encoding something
is (if it gets it wrong, it's usually really obvious to a human). It
has a magic second parameter "win" to switch from ISO-8859 to Windows
encoding - ISO-8859-1 becomes Windows-1252, -2 becomes 1250, etc.
Additionally, chardet often returns "MacCyrillic" for files that are
actually encoded Windows-1256 (Arabic). So, yes, it's definitely
possible for chardet to pick something that you can't actually decode
with.

For the OP's situation, frankly, I doubt there'll be anything other
than UTF-8, Latin-1, and CP-1252. The chances that someone casually
mixes CP-1252 with (say) CP-1254 would be vanishingly small. So the
simple decode of "UTF-8, or failing that, 1252" is probably going to
give correct results for most of the content. The trick is figuring
out a correct boundary for the check; line-by-line may be sufficient,
or it may not.

ChrisA

[toc] | [prev] | [next] | [standalone]

#110712

On Tue, Jun 28, 2016, at 06:25, Chris Angelico wrote:
> For the OP's situation, frankly, I doubt there'll be anything other
> than UTF-8, Latin-1, and CP-1252. The chances that someone casually
> mixes CP-1252 with (say) CP-1254 would be vanishingly small. So the
> simple decode of "UTF-8, or failing that, 1252" is probably going to
> give correct results for most of the content. The trick is figuring
> out a correct boundary for the check; line-by-line may be sufficient,
> or it may not.

For completeness, this can be done character-by-character (i.e. try to
decode a UTF-8 character, if it fails decode the offending byte as 1252)
with an error handler:

import codecs

def cp1252_errors(exception):
    input, idx = exception.object, exception.start
    byte = input[idx:idx+1]
    try:
        return byte.decode('windows-1252'), idx+1
    except UnicodeDecodeError:
        # python's cp1252 doesn't accept 0x81, etc
        return byte.decode('latin1'), idx+1

codecs.register_error('cp1252', cp1252_errors)

assert b"t\xe9st\xc3\xadng".decode('utf-8', errors='cp1252') ==
"t\u00e9st\u00edng"

This is probably sufficient for most purposes; byte sequences that
happen to be valid UTF-8 characters but mean something sensible in
cp-1252 are rare. Just be fortunate that that's all you have to deal
with - the equivalent problem for Japanese encodings, for instance, is
much harder (you'd probably want the boundary to be "per run of
non-ASCII* characters" if lines don't suffice, and detecting the
difference between UTF-8, Shift-JIS, and EUC-JP is nontrivial). There's
a reason the word "mojibake" comes from Japanese.

*well, JIS X 0201, which is ASCII but for 0x5C and 0x7E. And unless
you've got ISO-2022 codes to provide context for that, you've just got
to guess what those two bytes mean. Fortunately (fsvo), many
environments' fonts display the relevant ASCII characters as their JIS
alternatives, taking that choice away from you.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web