Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #196306 > unrolled thread

Decoding bytes to text strings in Python 2

Back to article view | Back to comp.lang.python

Contents

  Decoding bytes to text strings in Python 2 Rayner Lucas <usenet202101@magic-cookie.co.ukNOSPAMPLEASE> - 2024-06-21 16:49 +0100
    Re: Decoding bytes to text strings in Python 2 Chris Angelico <rosuav@gmail.com> - 2024-06-22 03:42 +1000
      Re: Decoding bytes to text strings in Python 2 Rayner Lucas <usenet202101@magic-cookie.co.ukNOSPAMPLEASE> - 2024-06-22 13:13 +0100
        Re: Decoding bytes to text strings in Python 2 (Posting On Python-List Prohibited) Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-06-22 23:19 +0000
        Re: Decoding bytes to text strings in Python 2 Chris Angelico <rosuav@gmail.com> - 2024-06-24 09:30 +1000
        Re: Decoding bytes to text strings in Python 2 MRAB <python@mrabarnett.plus.com> - 2024-06-24 01:14 +0100
        Re: Decoding bytes to text strings in Python 2 Chris Angelico <rosuav@gmail.com> - 2024-06-24 11:43 +1000
        Tkinter and astral characters (was: Decoding bytes to text strings in Python 2) "Peter J. Holzer" <hjp-python@hjp.at> - 2024-06-24 13:03 +0200
    Re: Decoding bytes to text strings in Python 2 Rayner Lucas <usenet202101@magic-cookie.co.ukNOSPAMPLEASE> - 2024-06-22 13:26 +0100

#196306 — Decoding bytes to text strings in Python 2

I'm curious about something I've encountered while updating a very old 
Tk app (originally written in Python 1, but I've ported it to Python 2 
as a first step towards getting it running on modern systems). The app 
downloads emails from a POP server and displays them. At the moment, the 
code is completely unaware of character encodings (which is something I 
plan to fix), and I have found that I don't understand what Python is 
doing when no character encoding is specified.

To demonstrate, I have written this short example program that displays 
a variety of UTF-8 characters to check whether they are decoded 
properly:

---- Example Code ----
import Tkinter as tk

window = tk.Tk()

mytext = """
  \xc3\xa9 LATIN SMALL LETTER E WITH ACUTE
  \xc5\x99 LATIN SMALL LETTER R WITH CARON
  \xc4\xb1 LATIN SMALL LETTER DOTLESS I
  \xef\xac\x84 LATIN SMALL LIGATURE FFL
  \xe2\x84\x9a DOUBLE-STRUCK CAPITAL Q
  \xc2\xbd VULGAR FRACTION ONE HALF
  \xe2\x82\xac EURO SIGN
  \xc2\xa5 YEN SIGN
  \xd0\x96 CYRILLIC CAPITAL LETTER ZHE
  \xea\xb8\x80 HANGUL SYLLABLE GEUL
  \xe0\xa4\x93 DEVANAGARI LETTER O
  \xe5\xad\x97 CJK UNIFIED IDEOGRAPH-5B57
  \xe2\x99\xa9 QUARTER NOTE
  \xf0\x9f\x90\x8d SNAKE
  \xf0\x9f\x92\x96 SPARKLING HEART
"""

mytext = mytext.decode(encoding="utf-8")
greeting = tk.Label(text=mytext)
greeting.pack()

window.mainloop()
---- End Example Code ----

This works exactly as expected, with all the characters displaying 
correctly.

However, if I comment out the line 'mytext = mytext.decode
(encoding="utf-8")', the program still displays *almost* everything 
correctly. All of the characters appear correctly apart from the two 
four-byte emoji characters at the end, which instead display as four 
characters. For example, the "SNAKE" character actually displays as:
U+00F0 LATIN SMALL LETTER ETH
U+FF9F HALFWIDTH KATAKANA SEMI-VOICED SOUND MARK
U+FF90 HALFWIDTH KATAKANA LETTER MI
U+FF8D HALFWIDTH KATAKANA LETTER HE

What's Python 2 doing here? sys.getdefaultencoding() returns 'ascii', 
but it's clearly not attempting to display the bytes as ASCII (or 
cp1252, or ISO-8859-1). How is it deciding on some sort of almost-but-
not-quite UTF-8 decoding?

I am using Python 2.7.18 on a Windows 10 system. If there's any other 
relevant information I should provide please let me know.

Many thanks,
Rayner

[toc] | [next] | [standalone]

#196307

On Sat, 22 Jun 2024 at 03:28, Rayner Lucas via Python-list
<python-list@python.org> wrote:
> I'm curious about something I've encountered while updating a very old
> Tk app (originally written in Python 1, but I've ported it to Python 2
> as a first step towards getting it running on modern systems).
>
> I am using Python 2.7.18 on a Windows 10 system. If there's any other
> relevant information I should provide please let me know.

Unfortunately, you're running into one of the most annoying problems
from Python 2 and Windows: "narrow builds". You don't actually have
proper Unicode support. You have a broken implementation that works
for UCS-2 but doesn't actually support astral characters.

If you switch to a Linux system, it should work correctly, and you'll
be able to migrate the rest of the way onto Python 3. Once you achieve
that, you'll be able to operate on Windows or Linux equivalently,
since Python 3 solved this problem. At least, I *think* it will; my
current system has a Python 2 installed, but doesn't have tkinter
(because I never bothered to install it), and it's no longer available
from the upstream Debian repos, so I only tested it in the console.
But the decoding certainly worked.

ChrisA

[toc] | [prev] | [next] | [standalone]

#196309

In article <mailman.159.1718991773.2909.python-list@python.org>, 
rosuav@gmail.com says...
> 
> If you switch to a Linux system, it should work correctly, and you'll
> be able to migrate the rest of the way onto Python 3. Once you achieve
> that, you'll be able to operate on Windows or Linux equivalently,
> since Python 3 solved this problem. At least, I *think* it will; my
> current system has a Python 2 installed, but doesn't have tkinter
> (because I never bothered to install it), and it's no longer available
> from the upstream Debian repos, so I only tested it in the console.
> But the decoding certainly worked.

Thank you for the idea of trying it on a Linux system. I did so, and my 
example code generated the error:

_tkinter.TclError: character U+1f40d is above the range (U+0000-U+FFFF) 
allowed by Tcl

So it looks like the problem is ultimately due to a limitation of 
Tcl/Tk. I'm still not sure why it doesn't give an error on Windows and 
instead either works (when UTF-8 encoding is specified) or converts the 
out-of-range characters to ones it can display (when the encoding isn't 
specified). But now I know what the root of the problem is, I can deal 
with it appropriately (and my curiosity is at least partly satisfied).

This has given me a much better understanding of what I need to do in 
order to migrate to Python 3 and add proper support for non-ASCII 
characters, so I'm very grateful for your help!

Thanks,
Rayner

[toc] | [prev] | [next] | [standalone]

#196312 — Re: Decoding bytes to text strings in Python 2 (Posting On Python-List Prohibited)

On Sat, 22 Jun 2024 13:13:28 +0100, Rayner Lucas wrote:

> I'm still not sure why it doesn't give an error on Windows and 
> instead either works (when UTF-8 encoding is specified) or converts the 
> out-of-range characters to ones it can display ...

Windows can be so helpful, can’t it ...
<https://arstechnica.com/security/2024/06/php-vulnerability-allows-attackers-to-run-malicious-code-on-windows-servers/>

[toc] | [prev] | [next] | [standalone]

#196320

On Mon, 24 Jun 2024 at 08:20, Rayner Lucas via Python-list
<python-list@python.org> wrote:
>
> In article <mailman.159.1718991773.2909.python-list@python.org>,
> rosuav@gmail.com says...
> >
> > If you switch to a Linux system, it should work correctly, and you'll
> > be able to migrate the rest of the way onto Python 3. Once you achieve
> > that, you'll be able to operate on Windows or Linux equivalently,
> > since Python 3 solved this problem. At least, I *think* it will; my
> > current system has a Python 2 installed, but doesn't have tkinter
> > (because I never bothered to install it), and it's no longer available
> > from the upstream Debian repos, so I only tested it in the console.
> > But the decoding certainly worked.
>
> Thank you for the idea of trying it on a Linux system. I did so, and my
> example code generated the error:
>
> _tkinter.TclError: character U+1f40d is above the range (U+0000-U+FFFF)
> allowed by Tcl
>
> So it looks like the problem is ultimately due to a limitation of
> Tcl/Tk.
Yep, that seems to be the case. Not sure if that's still true on a
more recent Python, but it does look like you won't get astral
characters in tkinter on the one you're using.

> I'm still not sure why it doesn't give an error on Windows and

Because of the aforementioned weirdness of old (that is: pre-3.3)
Python versions on Windows. They were built to use a messy, buggy
hybrid of UCS-2 and UTF-16. Sometimes this got you around problems, or
at least masked them; but it wouldn't be reliable. That's why, in
Python 3.3, all that was fixed :)

> instead either works (when UTF-8 encoding is specified) or converts the
> out-of-range characters to ones it can display (when the encoding isn't
> specified). But now I know what the root of the problem is, I can deal
> with it appropriately (and my curiosity is at least partly satisfied).

Converting out-of-range characters is fairly straightforward, at least
as long as your Python interpreter is correctly built (so, Python 3,
or a Linux build of Python 2).

"".join(c if ord(c) < 65536 else "?" for c in text)

> This has given me a much better understanding of what I need to do in
> order to migrate to Python 3 and add proper support for non-ASCII
> characters, so I'm very grateful for your help!
>

Excellent. Hopefully all this mess is just a transitional state and
you'll get to something that REALLY works, soon!

ChrisA

[toc] | [prev] | [next] | [standalone]

#196321

On 2024-06-24 00:30, Chris Angelico via Python-list wrote:
> On Mon, 24 Jun 2024 at 08:20, Rayner Lucas via Python-list
> <python-list@python.org> wrote:
>>
>> In article <mailman.159.1718991773.2909.python-list@python.org>,
>> rosuav@gmail.com says...
>> >
>> > If you switch to a Linux system, it should work correctly, and you'll
>> > be able to migrate the rest of the way onto Python 3. Once you achieve
>> > that, you'll be able to operate on Windows or Linux equivalently,
>> > since Python 3 solved this problem. At least, I *think* it will; my
>> > current system has a Python 2 installed, but doesn't have tkinter
>> > (because I never bothered to install it), and it's no longer available
>> > from the upstream Debian repos, so I only tested it in the console.
>> > But the decoding certainly worked.
>>
>> Thank you for the idea of trying it on a Linux system. I did so, and my
>> example code generated the error:
>>
>> _tkinter.TclError: character U+1f40d is above the range (U+0000-U+FFFF)
>> allowed by Tcl
>>
>> So it looks like the problem is ultimately due to a limitation of
>> Tcl/Tk.
> Yep, that seems to be the case. Not sure if that's still true on a
> more recent Python, but it does look like you won't get astral
> characters in tkinter on the one you're using.
> 
[snip]
Tkinter in recent versions of Python can handle astral characters, at 
least back to Python 3.8, the oldest I have on my Windows PC.

[toc] | [prev] | [next] | [standalone]

#196322

On Mon, 24 Jun 2024 at 10:18, MRAB via Python-list
<python-list@python.org> wrote:
> Tkinter in recent versions of Python can handle astral characters, at
> least back to Python 3.8, the oldest I have on my Windows PC.

Good to know, thanks! I was hoping that would be the case, but I don't
have a Windows system to check on, so I didn't want to speak without
facts.

ChrisA

[toc] | [prev] | [next] | [standalone]

#196329 — Tkinter and astral characters (was: Decoding bytes to text strings in Python 2)

[Multipart message — attachments visible in raw view] — view raw

On 2024-06-24 01:14:22 +0100, MRAB via Python-list wrote:
> Tkinter in recent versions of Python can handle astral characters, at least
> back to Python 3.8, the oldest I have on my Windows PC.

I just tried modifying
https://docs.python.org/3/library/tkinter.html#a-hello-world-program
to display "Hello World \N{ROCKET}" instead (Python 3.10.12 as included
with Ubuntu 22.04). I don't get a warning or error, but the emoji isn't
displayed either.

I suspect that the default font doesn't include emojis and Tk isn't
smart enough to fall back to a different font (unlike xfce4-terminal
which shows the emoji just fine).

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp@hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"

[toc] | [prev] | [next] | [standalone]

#196310

In article <Text-20240621184010@ram.dialup.fu-berlin.de>, ram@zedat.fu-
berlin.de says...
> 
>   I didn't really do a super thorough deep dive on this,
>   but I'm just giving the initial impression without 
>   actually being familiar with Tkinter under Python 2,
>   so I might be wrong!
> 
>   The Text widget typically expects text in Tcl encoding,
>   which is usually UTF-8. 
> 
>   This is independent of the result returned by sys.get-
>   defaultencoding()! 
> 
>   If a UTF-8 string is inserted directly as a bytes object,
>   its code points will be displayed correctly by the Text 
>   widget as long as they are in the BMP (Basic Multilingual
>   Plane), as you already found out yourself.

Many thanks, you've helped me greatly in understanding what's happening. 
When I tried running my example code on a different system (Python 
2.7.18 on Linux, with Tcl/Tk 8.5), I got the error:

_tkinter.TclError: character U+1f40d is above the range (U+0000-U+FFFF) 
allowed by Tcl

So, as your reply suggests, the problem is ultimately a limitation of 
Tcl/Tk itself. Perhaps I should have spent more time studying the docs 
for that instead of puzzling over the details of character encodings in 
Python! I'm not sure why it doesn't give the same error on Windows, but 
at least now I know where the root of the issue is.

I am now much better informed about how to migrate the code I'm working 
on, so I am very grateful for your help.

Thanks,
Rayner

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web