Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #75625 > unrolled thread

cmd.exe on WIndows - problem with displaying some Unicode characters

Back to article view | Back to comp.lang.python

Contents

  cmd.exe on WIndows - problem with displaying some Unicode characters Wiktor <look@signature.invalid> - 2014-08-04 00:52 +0200
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Chris Angelico <rosuav@gmail.com> - 2014-08-04 09:08 +1000
      Re: cmd.exe on WIndows - problem with displaying some Unicode characters Christian Gollwitzer <auriocus@gmx.de> - 2014-08-04 22:35 +0200
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-08-04 00:20 +0100
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Andrew Berg <aberg010@my.hennepintech.edu> - 2014-08-03 18:25 -0500
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-08-04 00:29 +0100
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Chris Angelico <rosuav@gmail.com> - 2014-08-04 09:39 +1000
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Chris Angelico <rosuav@gmail.com> - 2014-08-04 10:14 +1000
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Glenn Linderman <v+python@g.nevcal.com> - 2014-08-03 17:17 -0700
      Re: cmd.exe on WIndows - problem with displaying some Unicode characters wxjmfauth@gmail.com - 2014-08-04 01:47 -0700
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Glenn Linderman <v+python@g.nevcal.com> - 2014-08-03 21:14 -0700
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Andrew Berg <aberg010@my.hennepintech.edu> - 2014-08-04 00:06 -0500
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Terry Reedy <tjreedy@udel.edu> - 2014-08-04 04:39 -0400
      Re: cmd.exe on WIndows - problem with displaying some Unicode characters Wiktor <look@signature.invalid> - 2014-08-04 17:00 +0200
        Re: cmd.exe on WIndows - problem with displaying some Unicode characters Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> - 2014-08-04 17:43 +0200
          Re: cmd.exe on WIndows - problem with displaying some Unicode characters Wiktor <look@signature.invalid> - 2014-08-04 18:48 +0200
            Re: cmd.exe on WIndows - problem with displaying some Unicode characters Chris Angelico <rosuav@gmail.com> - 2014-08-05 03:06 +1000
              Re: cmd.exe on WIndows - problem with displaying some Unicode characters Wiktor <look@signature.invalid> - 2014-08-04 19:22 +0200
                Re: cmd.exe on WIndows - problem with displaying some Unicode characters Terry Reedy <tjreedy@udel.edu> - 2014-08-04 15:43 -0400
        Re: cmd.exe on WIndows - problem with displaying some Unicode characters Terry Reedy <tjreedy@udel.edu> - 2014-08-04 15:17 -0400
          Re: cmd.exe on WIndows - problem with displaying some Unicode characters Wiktor <look@signature.invalid> - 2014-08-04 22:24 +0200
            Re: cmd.exe on WIndows - problem with displaying some Unicode characters Akira Li <4kir4.1i@gmail.com> - 2014-08-05 04:51 +0400
              Re: cmd.exe on WIndows - problem with displaying some Unicode characters wxjmfauth@gmail.com - 2014-08-05 00:38 -0700
              Re: cmd.exe on WIndows - problem with displaying some Unicode characters Wiktor <look@signature.invalid> - 2014-08-05 11:39 +0200
        Re: cmd.exe on WIndows - problem with displaying some Unicode characters Chris Angelico <rosuav@gmail.com> - 2014-08-05 06:11 +1000
        Re: cmd.exe on WIndows - problem with displaying some Unicode characters giacomo boffi <g@boffi.net> - 2014-08-04 22:53 +0200
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Chris “Kwpolska” Warrick <kwpolska@gmail.com> - 2014-08-04 10:46 +0200
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Glenn Linderman <v+python@g.nevcal.com> - 2014-08-04 02:46 -0700
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Glenn Linderman <v+python@g.nevcal.com> - 2014-08-04 02:53 -0700
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> - 2014-08-04 12:24 +0200
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Andrew Berg <aberg010@my.hennepintech.edu> - 2014-08-04 05:33 -0500
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Glenn Linderman <v+python@g.nevcal.com> - 2014-08-04 11:04 -0700
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Glenn Linderman <v+python@g.nevcal.com> - 2014-08-04 11:15 -0700
      Re: cmd.exe on WIndows - problem with displaying some Unicode characters Grant Edwards <invalid@invalid.invalid> - 2014-08-04 19:30 +0000
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Terry Reedy <tjreedy@udel.edu> - 2014-08-04 15:25 -0400
    Re: cmd.exe on WIndows - problem with displaying some Unicode characters Tony the Tiger <tony@tiger.invalid> - 2014-08-05 20:26 +0000
      Re: cmd.exe on WIndows - problem with displaying some Unicode characters Grant Edwards <invalid@invalid.invalid> - 2014-08-05 20:41 +0000
      Re: cmd.exe on WIndows - problem with displaying some Unicode characters Wiktor <look@signature.invalid> - 2014-08-06 00:16 +0200
        Re: cmd.exe on WIndows - problem with displaying some Unicode characters Tony the Tiger <tony@tiger.invalid> - 2014-08-06 18:27 +0000
      Re: cmd.exe on WIndows - problem with displaying some Unicode characters wxjmfauth@gmail.com - 2014-08-05 23:30 -0700

Page 1 of 2  [1] 2  Next page →

#75625 — cmd.exe on WIndows - problem with displaying some Unicode characters

Hi,

as OO programming exercise, I'm trying to port to Python one of my favorite
game from early'90 (Atari 65XL/XE) - Kolony (here's video from original
version on C64 https://www.youtube.com/watch?v=UFycYOp2cbE, and here's
video from modern rewritten (for Atari emulators) version: Kolony 2106
https://www.youtube.com/watch?v=eX20Qqqm5eg - you get the idea? ;-)).

OO Design is one thing, but I want to make it look as near as possible to
the original (those windows-like menus in console window). I tried to use
'standard' Unicode characters (I can see that most of my Windows monospaced
fonts have them) to draw frame around menu. Something like this:

 ┌──────────────╖
 │ Construction ║
 │ Production   ║
 │ Research     ║
 │ Exploration  ║
 ├··············╢
 │ Next turn    ║
 ╘══════════════╝

(I like the look of double lines on right and at the bottom)
But when I try to print those characters, I get an error:

| Traceback (most recent call last):
|   File "E:\Moje dokumenty\python\kolony\menu.py", line 14, in <module>
|     """
|   File "C:\Python34\lib\encodings\cp852.py", line 19, in encode
|     return codecs.charmap_encode(input,self.errors,encoding_map)[0]
| UnicodeEncodeError: 'charmap' codec can't encode character '\u2556' in position 1
| 6: character maps to <undefined>

Now I know what that means. Code page that my cmd.exe is using (852)
doesn't have "╖", "╘", "╢" and "·" symbols. Changing code page to Unicode
(65001) doesn't really help, because all is messed up:
 ┌──────────────╖
 │ Construction ║
 │ Production   ║
 │ Research     ║
 │ Exploration  ║
 ├··············╢
 │ Next turn    ║
 ╘══════════════╝
    �·········╢
 │ Next turn    ║
 ╘══════════════╝
    ��════════════╝
    ��═════╝
    ═╝
(I believe that's cmd.exe bug with Unicode support, not Python fault)


Before I drop entirely this idea of using double lines on right and bottom
edges, and make it look like this 
    ┌──────────────┐
    │ Construction │
    ├--------------┤
    │ Next turn    │
    └──────────────┘
I have to ask - is there a way to make that original concept work? I know,
that CP437 has symbols "╖", "╢" and "╘", but does not have polish letters -
and I need to display them too. 
I also know, that cmd.exe can display those Unicode characters (by
copy/paste them in command line or by listing filenames containing that
characters), no matter what CP is set. How does it manage to do it? Can I
exploit that writing my Python program?
Wiktor

-- 
Best regards,     Wiktor Matuszewski
'py{}@wu{}em.pl'.format('wkm', 'ka')

[toc] | [next] | [standalone]

#75627

On Mon, Aug 4, 2014 at 8:52 AM, Wiktor <look@signature.invalid> wrote:
> I have to ask - is there a way to make that original concept work? I know,
> that CP437 has symbols "╖", "╢" and "╘", but does not have polish letters -
> and I need to display them too.

Yeah, that's exactly the problem with codepages :)

The best way to do it is to use the Unicode codepage, but cmd.exe just
plain has issues. There are underlying Windows APIs for displaying
text that have problems with astral characters (I think that's what it
is), so ultimately, you're largely stuck.

One option would be to render the whole thing graphically, abandoning
cmd.exe altogether. That would be how a lot of telnet and SSH clients
will do the work. Get a proper Unicode-supporting toolkit (Tkinter has
issues with astral characters too, AIUI), and yes, you'll have to do a
lot of work yourself. Or maybe, grab an actual telnet client, and
write this as a socket server. I'd be delighted to help you with that
option - I'm a MUDder and have spent innumerable dev hours on telnet
clients!

ChrisA

[toc] | [prev] | [next] | [standalone]

#75710

Am 04.08.14 01:08, schrieb Chris Angelico:
> On Mon, Aug 4, 2014 at 8:52 AM, Wiktor <look@signature.invalid> wrote:
>> I have to ask - is there a way to make that original concept work? I know,
>> that CP437 has symbols "╖", "╢" and "╘", but does not have polish letters -
>> and I need to display them too.
>
> Yeah, that's exactly the problem with codepages :)
>
> The best way to do it is to use the Unicode codepage,

Agreed.

> but cmd.exe just
> plain has issues.

It's not cmd.exe, it's the terminal that is shit. You can't even 
interactively resize the width in the standard terminal.

> There are underlying Windows APIs for displaying
> text that have problems with astral characters (I think that's what it
> is), so ultimately, you're largely stuck.
>
> One option would be to render the whole thing graphically, abandoning
> cmd.exe altogether. That would be how a lot of telnet and SSH clients
> will do the work. Get a proper Unicode-supporting toolkit (Tkinter has
> issues with astral characters too, AIUI), and yes, you'll have to do a
> lot of work yourself.

Tkinter only supports the BMP currently. But neither Polish nor box 
drawing does require more: All those box drawing symbols are in the BMP:

	http://www.unicode.org/charts/PDF/U2500.pdf

So you could use a Tkinter text widget and put your data there - or even 
a simple label would do.

	Christian

[toc] | [prev] | [next] | [standalone]

#75628

On 03/08/2014 23:52, Wiktor wrote:
>
> Hi,
>
> as OO programming exercise, I'm trying to port to Python one of my favorite
> game from early'90 (Atari 65XL/XE) - Kolony (here's video from original
> version on C64 https://www.youtube.com/watch?v=UFycYOp2cbE, and here's
> video from modern rewritten (for Atari emulators) version: Kolony 2106
> https://www.youtube.com/watch?v=eX20Qqqm5eg - you get the idea? ;-)).
>
> OO Design is one thing, but I want to make it look as near as possible to
> the original (those windows-like menus in console window). I tried to use
> 'standard' Unicode characters (I can see that most of my Windows monospaced
> fonts have them) to draw frame around menu. Something like this:
>
>   ┌──────────────╖
>   │ Construction ║
>   │ Production   ║
>   │ Research     ║
>   │ Exploration  ║
>   ├··············╢
>   │ Next turn    ║
>   ╘══════════════╝
>
> (I like the look of double lines on right and at the bottom)
> But when I try to print those characters, I get an error:
>
> | Traceback (most recent call last):
> |   File "E:\Moje dokumenty\python\kolony\menu.py", line 14, in <module>
> |     """
> |   File "C:\Python34\lib\encodings\cp852.py", line 19, in encode
> |     return codecs.charmap_encode(input,self.errors,encoding_map)[0]
> | UnicodeEncodeError: 'charmap' codec can't encode character '\u2556' in position 1
> | 6: character maps to <undefined>
>
> Now I know what that means. Code page that my cmd.exe is using (852)
> doesn't have "╖", "╘", "╢" and "·" symbols. Changing code page to Unicode
> (65001) doesn't really help, because all is messed up:
>   ┌──────────────╖
>   │ Construction ║
>   │ Production   ║
>   │ Research     ║
>   │ Exploration  ║
>   ├··············╢
>   │ Next turn    ║
>   ╘══════════════╝
>      �·········╢
>   │ Next turn    ║
>   ╘══════════════╝
>      ��════════════╝
>      ��═════╝
>      ═╝
> (I believe that's cmd.exe bug with Unicode support, not Python fault)
>
>
> Before I drop entirely this idea of using double lines on right and bottom
> edges, and make it look like this
>      ┌──────────────┐
>      │ Construction │
>      ├--------------┤
>      │ Next turn    │
>      └──────────────┘
> I have to ask - is there a way to make that original concept work? I know,
> that CP437 has symbols "╖", "╢" and "╘", but does not have polish letters -
> and I need to display them too.
> I also know, that cmd.exe can display those Unicode characters (by
> copy/paste them in command line or by listing filenames containing that
> characters), no matter what CP is set. How does it manage to do it? Can I
> exploit that writing my Python program?
> Wiktor
>

There are multiple known problems with cmd.exe and unicode.  A solution 
might be to use powershell, but windows being windows who really knows? :)

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#75629

On 2014.08.03 18:08, Chris Angelico wrote:
> The best way to do it is to use the Unicode codepage, but cmd.exe just
> plain has issues. There are underlying Windows APIs for displaying
> text that have problems with astral characters (I think that's what it
> is), so ultimately, you're largely stuck.
That is not quite true. The terminal has these issues, not the shell. Using
cp65001 does make Unicode in a Windows terminal possible, but using a better
terminal[1] makes it almost perfect (my experience has been that input can be
difficult, but output works well). I personally have used an IRC bot written in
Python with logging output containing Unicode characters that display just fine
(both locally and over SSH).

[1] I recommend ConEmu: https://code.google.com/p/conemu-maximus5/

[toc] | [prev] | [next] | [standalone]

#75630

On 04/08/2014 00:25, Andrew Berg wrote:
> On 2014.08.03 18:08, Chris Angelico wrote:
>> The best way to do it is to use the Unicode codepage, but cmd.exe just
>> plain has issues. There are underlying Windows APIs for displaying
>> text that have problems with astral characters (I think that's what it
>> is), so ultimately, you're largely stuck.
> That is not quite true. The terminal has these issues, not the shell. Using
> cp65001 does make Unicode in a Windows terminal possible, but using a better
> terminal[1] makes it almost perfect (my experience has been that input can be
> difficult, but output works well). I personally have used an IRC bot written in
> Python with logging output containing Unicode characters that display just fine
> (both locally and over SSH).
>
> [1] I recommend ConEmu: https://code.google.com/p/conemu-maximus5/
>

*facepalm* forgot all about ConEmu, but then I only use it on a daily 
basis :)

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#75631

On Mon, Aug 4, 2014 at 9:25 AM, Andrew Berg
<aberg010@my.hennepintech.edu> wrote:
> On 2014.08.03 18:08, Chris Angelico wrote:
>> The best way to do it is to use the Unicode codepage, but cmd.exe just
>> plain has issues. There are underlying Windows APIs for displaying
>> text that have problems with astral characters (I think that's what it
>> is), so ultimately, you're largely stuck.
> That is not quite true. The terminal has these issues, not the shell. Using
> cp65001 does make Unicode in a Windows terminal possible, but using a better
> terminal[1] makes it almost perfect (my experience has been that input can be
> difficult, but output works well). I personally have used an IRC bot written in
> Python with logging output containing Unicode characters that display just fine
> (both locally and over SSH).
>
> [1] I recommend ConEmu: https://code.google.com/p/conemu-maximus5/

Sorry, yeah, my terminology was sloppy. It's not technically cmd.exe,
but the default console.

I just played around with a CP-437 decode of everything 128-255,
rendered in various different fonts, all using my MUD client on
Windows. (For what it's worth, it renders using GTK2 and Pango. But I
suspect this is more a font issue than a display engine one.) Most
fonts had those little boxes-with-numbers for most of the line drawing
characters, but Lucida Sans Unicode, Meiryo, and Tahoma all managed to
display all the characters. However, I wasn't able to visually
distinguish between the single-line and double-line characters, so you
may want to play around with it a bit more.

Your simplest solution may still be to abandon those double lines and
just go with single.

ChrisA

[toc] | [prev] | [next] | [standalone]

#75633

On Mon, Aug 4, 2014 at 9:39 AM, Chris Angelico <rosuav@gmail.com> wrote:
> I just played around with a CP-437 decode of everything 128-255,
> rendered in various different fonts, all using my MUD client on
> Windows. (For what it's worth, it renders using GTK2 and Pango. But I
> suspect this is more a font issue than a display engine one.) Most
> fonts had those little boxes-with-numbers for most of the line drawing
> characters, but Lucida Sans Unicode, Meiryo, and Tahoma all managed to
> display all the characters. However, I wasn't able to visually
> distinguish between the single-line and double-line characters, so you
> may want to play around with it a bit more.

Hmm. Actually... I'd mucked up my CP-437 decode. For instance, I'd
represented 0xC8 as U+251D BOX DRAWINGS VERTICAL LIGHT AND RIGHT
HEAVY, where it's actually U+255A BOX DRAWINGS DOUBLE UP AND RIGHT.
When I get the right characters, the default fonts on Windows work
just fine. (There's still the issue that the supposedly "HEAVY" lines
are coming out with the exact same thickness as the "LIGHT" lines, but
that's unlikely to bother you as you don't need those characters.)

So, conclusion: If you do the drawing yourself, all those characters
are fine. Go ahead and use them!

ChrisA

[toc] | [prev] | [next] | [standalone]

#75634

[Multipart message — attachments visible in raw view] — view raw

On 8/3/2014 4:25 PM, Andrew Berg wrote:
> On 2014.08.03 18:08, Chris Angelico wrote:
>> The best way to do it is to use the Unicode codepage, but cmd.exe just
>> plain has issues. There are underlying Windows APIs for displaying
>> text that have problems with astral characters (I think that's what it
>> is), so ultimately, you're largely stuck.
> That is not quite true. The terminal has these issues, not the shell. Using
> cp65001 does make Unicode in a Windows terminal possible, but using a better
> terminal[1] makes it almost perfect (my experience has been that input can be
> difficult, but output works well). I personally have used an IRC bot written in
> Python with logging output containing Unicode characters that display just fine
> (both locally and over SSH).
>
> [1] I recommend ConEmu: https://code.google.com/p/conemu-maximus5/
I will be reading more about conemu, thanks for the reference.

http://bugs.python.org/issue1602  describes 7 years worth of discussion 
of the problems with the console/terminal used by default by cmd.exe and 
other Windows command line programs, versus Python.

The recent insights in the last couple weeks have given me hope that 
Python might be able to be fixed to work properly with the default 
Windows console at long last... at least for non-astral characters (I'm 
not sure whether or not the Windows console supports non-BMP characters).

For this OP problem, it is mostly a matter of finding a fixed-width font 
that supports the box drawing characters and the Polish characters that 
are desired.  Lucida Console has a fair repertoire, and Consolas has a 
fair repertoire, in the fixed-width font arena. There may be others, 
documented on Polish language web sites that I wouldn't know about, and 
I don't know enough Polish to be sure those I mentioned suffice.

And then, the workarounds mentioned in the above-referenced bug or on 
the GitHub or PyPi sites mentioned should provide any needed additional 
solutions... and hopefully something along this line finally integrated 
into Python so that it can finally be said that Python supports Unicode 
properly on Windows (or at least as properly as Windows allows... but it 
is pretty clear that Windows supports Unicode, even for the console, 
using different APIs that Python is presently using, and that mismatch 
between APIs is really the source of the problems with using Unicode in 
Python on Windows).

Glenn

[toc] | [prev] | [next] | [standalone]

#75674

Le lundi 4 août 2014 02:17:59 UTC+2, Glenn Linderman a écrit :
> On 8/3/2014 4:25 PM, Andrew Berg wrote:
> 
>     
>     
>       On 2014.08.03 18:08, Chris Angelico wrote:
> 
>       
>         The best way to do it is to use the Unicode codepage, but cmd.exe just
> plain has issues. There are underlying Windows APIs for displaying
> text that have problems with astral characters (I think that's what it
> is), so ultimately, you're largely stuck.
> 
>       
>       That is not quite true. The terminal has these issues, not the shell. Using
> cp65001 does make Unicode in a Windows terminal possible, but using a better
> terminal[1] makes it almost perfect (my experience has been that input can be
> difficult, but output works well). I personally have used an IRC bot written in
> Python with logging output containing Unicode characters that display just fine
> (both locally and over SSH).
> 
> [1] I recommend ConEmu: https://code.google.com/p/conemu-maximus5/
> 
>     
>     I will be reading more about conemu, thanks for the reference.
> 
>     
> 
>     http://bugs.python.org/issue1602  describes 7 years worth of
>     discussion of the problems with the console/terminal used by default
>     by cmd.exe and other Windows command line programs, versus Python.
> 
>     
> 
>     The recent insights in the last couple weeks have given me hope that
>     Python might be able to be fixed to work properly with the default
>     Windows console at long last... at least for non-astral characters
>     (I'm not sure whether or not the Windows console supports non-BMP
>     characters).
> 
>     
> 
>     For this OP problem, it is mostly a matter of finding a fixed-width
>     font that supports the box drawing characters and the Polish
>     characters that are desired.  Lucida Console has a fair repertoire,
>     and Consolas has a fair repertoire, in the fixed-width font arena.
>     There may be others, documented on Polish language web sites that I
>     wouldn't know about, and I don't know enough Polish to be sure those
>     I mentioned suffice.
> 
>     
> 
>     And then, the workarounds mentioned in the above-referenced bug or
>     on the GitHub or PyPi sites mentioned should provide any needed
>     additional solutions... and hopefully something along this line
>     finally integrated into Python so that it can finally be said that
>     Python supports Unicode properly on Windows (or at least as properly
>     as Windows allows... but it is pretty clear that Windows supports
>     Unicode, even for the console, using different APIs that Python is
>     presently using, and that mismatch between APIs is really the source
>     of the problems with using Unicode in Python on Windows).
> 


1) A lot of confusion and imprecisions.

2) Unicode will never work properly because its handling
is wrong by design.

3) From my interactive interpreter:
>>> me.centralwidget.shell.font().rawName()
'Consolas'
>>> me.centralwidget.shell.fontMetrics().width('—')
9
>>> me.centralwidget.shell.fontMetrics().width('㑖')
17
>>> # note:
>>> me.centralwidget.shell.fontMetrics().width('\t')
80
>>>

(When I think I will thow away all this work...)

4) Already posted:

Fun with win_unicode_console

NB Modulo .notdef glyph in font.

D:\>D:\conuni\build\exe.win32-3.2\jmtest.exe
3.2.5 (default, May 15 2013, 23:06:03) [MSC v.1500 32 bit (Intel)]
Quelques caractères: «abc需ßÜÆŸçñö»
Loop: empty string => quit
—>for ascii users: abc
Votre entrée était : for ascii users: abc  20 caractère(s)
—>abc需ß
Votre entrée était : abcéœ€ß  7 caractère(s)
—>*€*\u20ac*
Votre entrée était : *€*€*  5 caractère(s)
—>\\\
Votre entrée était : \\\  3 caractère(s)
—>\\u0066
Votre entrée était : \f  2 caractère(s)
—>aϕЯ①ǢṺijṏz
Votre entrée était : aϕЯ①ǢṺijṏz  9 caractère(s)
—>D:\jm\Москва\Zürich\Αθήνα\œdipe
Votre entrée était : D:\jm\Москва\Zürich\Αθήνα\œdipe  31 caractère(s)
—>a\u123z
Wahrsheinlich falsches \uxxxx
—>r'\'
Votre entrée était : r'\'  4 caractère(s)
—>é\u1234\u3456z
Votre entrée était : éሴ㑖z  4 caractère(s)
—>
Fin


Addendum: there is a "but".

5) I'm aware about the discussions on the subject, see 1).

jmf

[toc] | [prev] | [next] | [standalone]

#75658

[Multipart message — attachments visible in raw view] — view raw

On 8/3/2014 5:17 PM, Glenn Linderman wrote:
> On 8/3/2014 4:25 PM, Andrew Berg wrote:
>> On 2014.08.03 18:08, Chris Angelico wrote:
>>> The best way to do it is to use the Unicode codepage, but cmd.exe just
>>> plain has issues. There are underlying Windows APIs for displaying
>>> text that have problems with astral characters (I think that's what it
>>> is), so ultimately, you're largely stuck.
>> That is not quite true. The terminal has these issues, not the shell. Using
>> cp65001 does make Unicode in a Windows terminal possible, but using a better
>> terminal[1] makes it almost perfect (my experience has been that input can be
>> difficult, but output works well). I personally have used an IRC bot written in
>> Python with logging output containing Unicode characters that display just fine
>> (both locally and over SSH).
>>
>> [1] I recommend ConEmu:https://code.google.com/p/conemu-maximus5/
> I will be reading more about conemu, thanks for the reference.

Having read a bit about ConEmu, it seems that it is a "pretty face" 
built on top of Windows Console, by screen scraping the real (but 
hidden) Windows Console, and providing a number of interesting display 
features and modes. So while it adds functionality to the Windows 
Console interface, it doesn't seem like it is likely to fix bugs or 
resolve issues with code pages, font selection, or Unicode character 
repertoires, which are the issues of this thread and the bug I 
referenced earlier.

Can anyone with ConEmu installed refute this interpretation of its 
functionality?

[toc] | [prev] | [next] | [standalone]

#75659

On 2014.08.03 23:14, Glenn Linderman wrote:
> Having read a bit about ConEmu, it seems that it is a "pretty face" built on
> top of Windows Console, by screen scraping the real (but hidden) Windows
> Console, and providing a number of interesting display features and modes. So
> while it adds functionality to the Windows Console interface, it doesn't seem
> like it is likely to fix bugs or resolve issues with code pages, font
> selection, or Unicode character repertoires, which are the issues of this
> thread and the bug I referenced earlier.
> 
> Can anyone with ConEmu installed refute this interpretation of its functionality?
> 
If you run cmd in it, you will still need to use cp65001. This is not necessary
for  (or applicable to) other applications (such as a Python interpreter) run
directly. ConEmu can use any arbitrary font available on the system. As I have
said, I have been able to display Unicode output on it from an application
written in Python. No mojibake, no replacement characters, just the exact
characters one would expect.
I do not know the internals of ConEmu and how it interacts with conhost and
whatever else, but I have not found a need to since it has just worked for me.

[toc] | [prev] | [next] | [standalone]

#75670

On 8/3/2014 6:52 PM, Wiktor wrote:
>
> Hi,
>
> as OO programming exercise, I'm trying to port to Python one of my favorite
> game from early'90 (Atari 65XL/XE) - Kolony (here's video from original
> version on C64 https://www.youtube.com/watch?v=UFycYOp2cbE, and here's

This appears to be an actual text screen, no graphics.

> video from modern rewritten (for Atari emulators) version: Kolony 2106
> https://www.youtube.com/watch?v=eX20Qqqm5eg - you get the idea? ;-)).

This appears to be text boxes on a graphics screen.

> OO Design is one thing, but I want to make it look as near as possible to
> the original (those windows-like menus in console window).

Which original? the C64 or Atari.  The important characteristic of both 
is that both have multiple overlapping popup boxes. This means that 
either you or a widget framework much keep track of stacking order and 
how to restore what was hidden when a box goes away or is moved down in 
the stacking order. I would not be surprised if the Atari had at least a 
rudimentary widget framework.

 > I tried to use
> 'standard' Unicode characters (I can see that most of my Windows monospaced
> fonts have them) to draw frame around menu. Something like this:
>
>   ┌──────────────╖
>   │ Construction ║
>   │ Production   ║
>   │ Research     ║
>   │ Exploration  ║
>   ├··············╢
>   │ Next turn    ║
>   ╘══════════════╝
>
> (I like the look of double lines on right and at the bottom)
> But when I try to print those characters, I get an error:
>
> | Traceback (most recent call last):
> |   File "E:\Moje dokumenty\python\kolony\menu.py", line 14, in <module>
> |     """
> |   File "C:\Python34\lib\encodings\cp852.py", line 19, in encode
> |     return codecs.charmap_encode(input,self.errors,encoding_map)[0]
> | UnicodeEncodeError: 'charmap' codec can't encode character '\u2556' in position 1
> | 6: character maps to <undefined>

You have two separate problems with running in the windows console.

1. The character issue. If you run a program to just print the above 
from an Idle editor, so that the output is printed to the Idle shell, 
there should be no problem.
 >>> print('\u2556'*10)
╖╖╖╖╖╖╖╖╖╖
But characters are not your real issue.

2. The random access issue. The MS console in normal use is like a 
serial printer or terminal. Once a line is printed, it cannot be 
changed. I looked at the video and the program randomly accesses a 24 or 
25 line x 80 column screen, overprinting existing characters at will and 
reversing black on white versus white of black at will.  MSDOS screens 
recognized standard ANSI screen control codes once the ANSI.SYS driver 
was installed, which was fairly normal. But cmd.exe is actually a 
regression from MS-DOS in that it apparently will not allow this.  Or it 
is a hugh pain.

You could get a program that emulates a full-screen ANSI terminal, and 
learn to use ANSI control codes.  Or you could use a tkinter (tk) Text 
widget. People have written at least serial terminal emulators for Text, 
but I did not find a full-screen.

Using tkinter, I would try making each box a separate text box placed in 
a frameand let tkinter worry about displaying them correctly and 
detecting which box get a mouse click or <enter> key.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#75695

  Hi,
first, thank you all for responses. I decided to just use single line frame
around menu. Yes, those double+single line corners work fine in ConEmu, but
I don't want this Python script to be dependent on external program. Maybe
one day it will be worth of showing to others, and it's silly to tell them
'It is pure console based game/framework but works only in ConEmu'...


  Now, to Terry's post:

On 8/3/2014 6:52 PM, Wiktor wrote:

>> as OO programming exercise, I'm trying to port to Python one of my favorite
>> game from early'90 (Atari 65XL/XE) - Kolony (here's video from original
>> version on C64 https://www.youtube.com/watch?v=UFycYOp2cbE, and here's
> 
> This appears to be an actual text screen, no graphics.
> 
>> video from modern rewritten (for Atari emulators) version: Kolony 2106
>> https://www.youtube.com/watch?v=eX20Qqqm5eg - you get the idea? ;-)).
> 
> This appears to be text boxes on a graphics screen.
> 
>> OO Design is one thing, but I want to make it look as near as possible to
>> the original (those windows-like menus in console window).
> 
> Which original? the C64 or Atari.  The important characteristic of both 
> is that both have multiple overlapping popup boxes. This means that 
> either you or a widget framework much keep track of stacking order and 
> how to restore what was hidden when a box goes away or is moved down in 
> the stacking order. I would not be surprised if the Atari had at least a 
> rudimentary widget framework.

  Yes, I'm aware that first link is to the text based game, and second to
graphic based game. I provided both links, because I couldn't find screen
cast from original Atari game (which is also text based, but IMO looks
better than C64's version), and this modern game is translated to English,
so is better for you to understand character of game.
  Yes, I'd like to make text game, that looks like window-based, with popup
boxes, inactive windows grayed out and all this stuff. And all this running
on standard console window (cmd.exe).

  I'm not starting from scratch. I'm using packages 'termcolor', 'colorama'
and 'colorconsole' - they provide functions to print text at desired
position on screen, and change color of foreground/background of this text.
With those packages I already developed some classes that allow me to
construct some simple menus for my console programs. Short demo of silly
program calculating degree of n-th root:
  http://youtu.be/V8ilLhHAT_k
  (I link to the video, because there's too much code lines to paste them
here. Also it's dependent upon those third party packages, and still
work-in-progress).

  So, I'm not worry about randomly access, colors, overprinting existing
characters. At this point I know how to do that. 
  I'm taking next step, so I tried to draw nice frame around menu (that's
why I posted yesterday).
  Next step would be to manage those widgets to draw one over another, to
keep track which one window opens which, and when the other window must be
closed and when only grayed out. At this point I still don't know how to do
this right, but I'm thinking about this very hard. :-) Probably one day
I'll ask it here, if I don't figure it out. :-)

Wiktor

-- 
Best regards,     Wiktor Matuszewski
'py{}@wu{}em.pl'.format('wkm', 'ka')

[toc] | [prev] | [next] | [standalone]

#75697

On 08/04/2014 05:00 PM, Wiktor wrote:
>    Hi,
> first, thank you all for responses. I decided to just use single line frame
> around menu. Yes, those double+single line corners work fine in ConEmu, but
> I don't want this Python script to be dependent on external program. Maybe
> one day it will be worth of showing to others, and it's silly to tell them
> 'It is pure console based game/framework but works only in ConEmu'...
>
>
>    Now, to Terry's post:
>
>
>    I'm not starting from scratch. I'm using packages 'termcolor', 'colorama'
> and 'colorconsole' - they provide functions to print text at desired
> position on screen, and change color of foreground/background of this text.

Thanks for pointing out these packages! Since you say you're using all 
three of them: where do colorama and colorconsole differ. From a quick 
look, I can see that termcolor is a bit different, but colorama and 
colorconsole seem pretty similar in scope.

Thanks,
Wolfgang

[toc] | [prev] | [next] | [standalone]

#75698

On Mon, 04 Aug 2014 17:43:41 +0200, Wolfgang Maier wrote:

>>    I'm not starting from scratch. I'm using packages 'termcolor', 'colorama'
>> and 'colorconsole' - they provide functions to print text at desired
>> position on screen, and change color of foreground/background of this text.
> 
> Thanks for pointing out these packages! Since you say you're using all 
> three of them: where do colorama and colorconsole differ. From a quick 
> look, I can see that termcolor is a bit different, but colorama and 
> colorconsole seem pretty similar in scope.

  From colorama I just use one function - init(). Without this
initialization all those ansii escape characters (used by colorama itself,
but also by termcolor.colored()) don't work in cmd.exe. At least I couldn't
make it work.
  All coloring work I make in termcolor.colored() function, because it
returns string, which I can work on (store and/or send it to print_at()
function later).
  And colorconsole is helpful with its screen.print_at() function [where
screen = colorconsole.terminal.get_terminal()].
  
  So, yes, it's matter of (probably bad) design, but now I need all three
packages. Maybe if I resign from storing my colored strings, and color them
just while sending them to printing function, I could get rid of colorama
and termcolor...
  Well, thanks for asking, because now, during writing this response, I see
that maybe redesign is worth of trying...

Wiktor

-- 
Best regards,     Wiktor Matuszewski
'py{}@wu{}em.pl'.format('wkm', 'ka')

[toc] | [prev] | [next] | [standalone]

#75699

On Tue, Aug 5, 2014 at 2:48 AM, Wiktor <look@signature.invalid> wrote:
>   From colorama I just use one function - init(). Without this
> initialization all those ansii escape characters (used by colorama itself,
> but also by termcolor.colored()) don't work in cmd.exe. At least I couldn't
> make it work.

I dug into colorama's source code, and it seems that "just one
function" is a little dismissive :) When you call colorama's init(),
it replaces stdout with a wrapper that parses ANSI sequences and turns
them into API calls. So, yeah, without that anything that outputs ANSI
sequences isn't going to work.

ChrisA

[toc] | [prev] | [next] | [standalone]

#75700

On Tue, 5 Aug 2014 03:06:41 +1000, Chris Angelico wrote:

> On Tue, Aug 5, 2014 at 2:48 AM, Wiktor <look@signature.invalid> wrote:
>>   From colorama I just use one function - init(). Without this
>> initialization all those ansii escape characters (used by colorama itself,
>> but also by termcolor.colored()) don't work in cmd.exe. At least I couldn't
>> make it work.
> 
> I dug into colorama's source code, and it seems that "just one
> function" is a little dismissive :) When you call colorama's init(),
> it replaces stdout with a wrapper that parses ANSI sequences and turns
> them into API calls. So, yeah, without that anything that outputs ANSI
> sequences isn't going to work.

  Maybe I didn't write it clear. :-) What I meant was, that even though I
don't use any other functions from colorama (I color all the strings with
termcolor) - I still have to use init() function from colorama. 
  termcolor doesn't want to work alone, even though its described as OS
independent. I guess it works fine on Linux terminal without init()
function from colorama. In cmd.exe I need colorama just for this.

-- 
Best regards,     Wiktor Matuszewski
'py{}@wu{}em.pl'.format('wkm', 'ka')

[toc] | [prev] | [next] | [standalone]

#75707

On 8/4/2014 1:22 PM, Wiktor wrote:
> On Tue, 5 Aug 2014 03:06:41 +1000, Chris Angelico wrote:
>
>> On Tue, Aug 5, 2014 at 2:48 AM, Wiktor <look@signature.invalid> wrote:
>>>    From colorama I just use one function - init(). Without this
>>> initialization all those ansii escape characters (used by colorama itself,
>>> but also by termcolor.colored()) don't work in cmd.exe. At least I couldn't
>>> make it work.
>>
>> I dug into colorama's source code, and it seems that "just one
>> function" is a little dismissive :) When you call colorama's init(),
>> it replaces stdout with a wrapper that parses ANSI sequences and turns
>> them into API calls. So, yeah, without that anything that outputs ANSI
>> sequences isn't going to work.
>
>    Maybe I didn't write it clear. :-) What I meant was, that even though I
> don't use any other functions from colorama (I color all the strings with
> termcolor) - I still have to use init() function from colorama.
>    termcolor doesn't want to work alone, even though its described as OS
> independent.

Termcolor says "ANSI Color formatting for output in terminal."
https://pypi.python.org/pypi/termcolor/1.1.0

It is OS-independent but depends on support of standard ANSI screen 
command codes. Microsoft removed that support from cmd.exe. If you look 
at the Terminal properties box on the page above, the only thing 
termcolor can do on Windows, by itself, is reversed text.  Colorama.init 
adds back (at least some of) the ANSI to API translation omitted from 
cmd.exe.

> I guess it works fine on Linux terminal without init()

Because linux terminals translate ANSI to whatever api calls are needed.

> function from colorama. In cmd.exe I need colorama just for this.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#75703

On 8/4/2014 11:00 AM, Wiktor wrote:

>    Yes, I'd like to make text game, that looks like window-based, with popup
> boxes, inactive windows grayed out and all this stuff. And all this running
> on standard console window (cmd.exe).

Your problem doing this is that cmd.exe is not a standard since 30 years 
ago full-screen console window, , but is intentionally crippled to stop 
people from doing what you are trying to do. Some as MS would like to 
delete it altogether.

>    I'm not starting from scratch. I'm using packages 'termcolor', 'colorama'
> and 'colorconsole' -

All on pypi.python.org, I see.

 > they provide functions to print text at desired
> position on screen, and change color of foreground/background of this text.
> With those packages I already developed some classes that allow me to
> construct some simple menus for my console programs. Short demo of silly
> program calculating degree of n-th root:
>    http://youtu.be/V8ilLhHAT_k

I am impressed with what the authors of those packages managed to do.

>    I'm taking next step, so I tried to draw nice frame around menu (that's
> why I posted yesterday).

Is there no working codepage with ascii text and the line chars? I 
suppose I am not surprised if not.

>    Next step would be to manage those widgets to draw one over another, to
> keep track which one window opens which, and when the other window must be
> closed and when only grayed out. At this point I still don't know how to do
> this right, but I'm thinking about this very hard. :-) Probably one day
> I'll ask it here, if I don't figure it out. :-)

You may have to settle for using different background colors to 
delineate different boxes.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web