Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder2.enfer-du-nord.net!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.005 X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'escape': 0.04; 'ascii': 0.07; 'bytes.': 0.07; 'encoded': 0.07; 'outcome': 0.07; "editor's": 0.09; 'encoding.': 0.09; 'exception.': 0.09; 'presume': 0.09; 'utf-8': 0.09; 'output': 0.12; 'am,': 0.14; 'wrote:': 0.14; '(1).': 0.16; '(hopefully': 0.16; 'coded': 0.16; 'codes,': 0.16; 'consult': 0.16; 'cp1252': 0.16; 'hex': 0.16; 'ignores': 0.16; 'received:203.24': 0.16; 'renders': 0.16; 'subject:unicode': 0.16; 'sys.stdout': 0.16; 'input': 0.18; 'bytes': 0.19; 'command': 0.19; 'seeing': 0.21; '(or': 0.22; 'code': 0.22; 'header:In-Reply-To:1': 0.22; '(but': 0.22; 'e.g.': 0.22; 'file,': 0.22; 'itself.': 0.22; 'thu,': 0.22; 'incorrect': 0.23; 'sequences.': 0.23; 'byte': 0.25; 'detect': 0.25; 'extract': 0.25; 'specify': 0.25; 'assume': 0.25; 'windows': 0.26; "i'm": 0.26; "doesn't": 0.28; 'thanks': 0.29; 'string': 0.29; 'problem': 0.29; 'be.': 0.29; 'depends': 0.29; 'unicode': 0.29; 'blocks': 0.31; 'character.': 0.31; 'digits': 0.31; 'does': 0.31; "can't": 0.31; 'match': 0.31; 'to:addr:python-list': 0.32; 'another': 0.32; '...': 0.32; '(including': 0.33; 'character': 0.33; 'headers': 0.33; 'using': 0.34; 'skip:" 10': 0.34; 'there': 0.35; 'file': 0.35; 'characters': 0.35; 'editor': 0.35; 'open': 0.35; 'header :User-Agent:1': 0.35; 'point': 0.35; 'typical': 0.35; 'else': 0.37; 'should': 0.37; 'sequence': 0.38; 'less': 0.38; 'files': 0.38; 'anything': 0.38; 'unless': 0.38; 'affect': 0.39; 'docs': 0.39; 'likely': 0.39; 'skip:s 30': 0.39; 'to:addr:python.org': 0.39; 'how': 0.39; 'necessarily': 0.40; 'box,': 0.60; 'unusual': 0.60; 'best': 0.60; '2011': 0.62; 'american': 0.65; 'nothing.': 0.68; 'obtained': 0.68; 'reply-to:no real name:2**0': 0.72; 'header:Reply-To:1': 0.72; 'order,': 0.73; 'spaces': 0.73; 'stream': 0.73; 'consumer': 0.80; 'western': 0.82; 'encoding,': 0.84; 'encoding?': 0.84; 'symptom': 0.84; 'window,': 0.84; 'inability': 0.91; 'points,': 0.91; 'render': 0.91; 'you).': 0.91; 'on...': 0.93 In-Reply-To: References: Date: Thu, 12 May 2011 13:54:20 +1000 Subject: Re: unicode by default From: "John Machin" To: python-list@python.org User-Agent: SquirrelMail/1.4.21 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: sjmachin@lexicon.net List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 73 NNTP-Posting-Host: 82.94.164.166 X-Trace: 1305172465 news.xs4all.nl 81474 [::ffff:82.94.164.166]:51120 X-Complaints-To: abuse@xs4all.nl Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:5191 On Thu, May 12, 2011 11:22 am, harrismh777 wrote: > John Machin wrote: >> (1) You cannot work without using bytes sequences. Files are byte >> sequences. Web communication is in bytes. You need to (know / assume / >> be >> able to extract / guess) the input encoding. You need to encode your >> output using an encoding that is expected by the consumer (or use an >> output method that will do it for you). >> >> (2) You don't need to use bytes to specify a Unicode code point. Just >> use >> an escape sequence e.g. "\u0404" is a Cyrillic character. >> > > Thanks John. In reverse order, I understand point (2). I'm less clear > on point (1). > > If I generate a string of characters that I presume to be ascii/utf-8 > (no \u0404 type characters) > and write them to a file (stdout) how does > default encoding affect that file.by default..? I'm not seeing that > there is anything unusual going on... About """characters that I presume to be ascii/utf-8 (no \u0404 type characters)""": All Unicode characters (including U+0404) are encodable in bytes using UTF-8. The result of sys.stdout.write(unicode_characters) to a TERMINAL depends mostly on sys.stdout.encoding. This is likely to be UTF-8 on a linux/OSX/platform. On a typical American / Western European /[former] colonies Windows box, this is likely to be cp850 on a Command Prompt window, and cp1252 in IDLE. UTF-8: All Unicode characters are encodable in UTF-8. Only problem arises if the terminal can't render the character -- you'll get spaces or blobs or boxes with hex digits in them or nothing. Windows (Command Prompt window): only a small subset of characters can be encoded in e.g. cp850; anything else causes an exception. Windows (IDLE): ignores sys.stdout.encoding and renders the characters itself. Same outcome as *x/UTF-8 above. If you write directly (or sys.stdout is redirected) to a FILE, the default encoding is obtained by sys.getdefaultencoding() and is AFAIK ascii unless the machine's site.py has been fiddled with to make it UTF-8 or something else. > If I open the file with vi? If > I open the file with gedit? emacs? Any editor will have a default encoding; if that doesn't match the file encoding, you have a (hopefully obvious) problem if the editor doesn't detect the mismatch. Consult your editor's docs or HTFF1K. > Another question... in mail I'm receiving many small blocks that look > like sprites with four small hex codes, scattered about the mail... > mostly punctuation, maybe? ... guessing, are these unicode code > points, yes > and if so what is the best way to 'guess' the encoding? google("chardet") or rummage through the mail headers (but 4 hex digits in a box are a symptom of inability to render, not necessarily caused by an incorrect decoding) ... is > it coded in the stream somewhere...protocol? Should be.