Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'python,': 0.02; 'scripts': 0.03; 'broken': 0.04; 'interpreter': 0.05; '*not*': 0.07; '-*-': 0.07; 'encoded': 0.07; 'utf-8': 0.07; 'string': 0.09; 'coding:': 0.09; 'line:': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'type,': 0.09; 'python': 0.11; 'jan': 0.12; '2.7': 0.14; 'posted': 0.15; '3.0.': 0.16; 'buggy': 0.16; 'code?': 0.16; 'itself,': 0.16; 'literals': 0.16; 'non-ascii': 0.16; 'received:80.91.229.3': 0.16; 'received:plane.gmane.org': 0.16; 'reedy': 0.16; 'script,': 0.16; 'subject:unicode': 0.16; 'underlying': 0.16; 'unicode.': 0.16; 'utf8': 0.16; 'subject:python': 0.16; 'fix': 0.17; 'wrote:': 0.18; 'do.': 0.18; 'passing': 0.19; 'entered': 0.20; 'code,': 0.22; 'example': 0.22; 'header:User-Agent:1': 0.23; '2.x': 0.24; 'bytes': 0.24; 'case.': 0.24; 'focusing': 0.24; 'text.': 0.24; 'unicode': 0.24; 'script': 0.25; 'logging': 0.26; 'second': 0.26; 'pass': 0.26; 'gets': 0.27; 'header:X-Complaints-To:1': 0.27; 'header:In-Reply-To:1': 0.27; 'url:bugs': 0.29; 'related': 0.29; 'characters': 0.30; "i'm": 0.30; 'program,': 0.31; 'code': 0.31; 'posting': 0.31; "skip:' 10": 0.31; 'embedding': 0.31; 'extending': 0.31; 'text': 0.33; 'sense': 0.34; 'subject:from': 0.34; 'problem': 0.35; 'but': 0.35; 'there': 0.35; 'c++': 0.36; 'done': 0.36; "didn't": 0.36; 'possible': 0.36; 'wrong': 0.37; 'too': 0.37; 'clear': 0.37; 'being': 0.38; 'to:addr:python-list': 0.38; 'pm,': 0.38; 'does': 0.39; 'subject:can': 0.39; 'sure': 0.39; 'to:addr:python.org': 0.39; 'received:org': 0.40; 'even': 0.60; 'subject: / ': 0.60; 'conversion': 0.61; 'received:173': 0.61; 'skip:* 10': 0.61; 'first': 0.61; 'back': 0.62; 'happen': 0.63; 'great': 0.65; 'here': 0.66; 'url:png': 0.68; 'reverse': 0.68; 'pictures': 0.72; 'inline': 0.74; 'printing,': 0.74; 'subject:get': 0.81; 'characters,': 0.84; 'received:fios.verizon.net': 0.84; 'routines': 0.84; 'recover': 0.91; 'before?': 0.93 X-Injected-Via-Gmane: http://gmane.org/ To: python-list@python.org From: Terry Reedy Subject: Re: can't get utf8 / unicode strings from embedded python Date: Sun, 25 Aug 2013 14:59:36 -0400 References: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Gmane-NNTP-Posting-Host: pool-173-75-251-66.phlapa.fios.verizon.net User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 In-Reply-To: X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 43 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1377457194 news.xs4all.nl 15993 [2001:888:2000:d::a6]:37030 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:52982 On 8/25/2013 1:57 PM, David M. Cotter wrote: > i'm sorry this is so confusing, let me try to re-state the problem in as clear a way as i can. > > I have a C++ program, with very well tested unicode support. All logging is done in utf8. I have conversion routines that work flawlessly, so i can assure you there is nothing wrong with logging and unicode support in the underlying program. > I am embedding python 2.7 into the program, and extending python with routines in my C++ program. If you want 'well-tested' (correct) unicode support from Python, use 3.3. Unicode in 2.x is somewhat buggy and definitely flakey. The first fix was to make unicode *the* text type, in 3.0. The second was to redesign the internals in 3.3. It is possible that 2.7 is too broken for what you want to do. > I have a script, encoded in utf8, and *marked* as utf8 with this line: > # -*- coding: utf-8 -*- > > In that script, i have inline unicode text. The example scripts that you posted pictures of do *not* have unicode text. They have bytestring literals with (encoded) non-ascii chars inside them. This is not a great idea. I am not sure what bytes you end up with. Apparently, not what you expect. To make them 'unicode text', you must prepend the literals with 'u'. Didn't someone say this before? > When I pass that text to my C++ program, the Python interpreter decides that these bytes are macRoman, and handily "converts" them to unicode. To compensate, i must "convert" these "macRoman" characters encoded as utf8, back to macRoman, then "interpret" them as utf8. In this way i can recover the original unicode. > > When i return a unicode string back to python, i must do the reverse so that Python gets back what it expects. > > This is not related to printing, or sys.stdout, it does happen with that too but focusing on that is a red-herring. Let's focus on just passing a string into C++ then back out. > > This would all actually make sense IF my script was marked as being "macRoman" even tho i entered UTF8 Characters, but that is not the case. > > Let's prove my statements. Here is the script, *interpreted* as MacRoman: > http://karaoke.kjams.com/screenshots/bugs/python_unicode/script_as_macroman.png Why are you posting pictures of code, instead of the (runnable) code itself, as you did with C code? -- Terry Jan Reedy