Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #38793

Re: UnicodeEncodeError when not running script from IDE

Show all headers | View raw

Thanks a lot Steven, you gave me a good AHA experience! :)

Now I understand why I had to use encoding when calling the urllib2! So basically Eclipse PyDev does this in the background for me, and its console supports utf-8, so thats why i never had to think about it before (and why some scripts tends to fail with unicode errors when run outside of the Eclipse IDE).

cheers
Magnus

> Start here:
> 
> 
> 
> "The Absolute Minimum Every Software Developer Absolutely, Positively Must
> 
> Know About Unicode and Character Sets (No Excuses!)"
> 
> 
> 
> http://www.joelonsoftware.com/articles/Unicode.html
> 
> 
> 
> 
> 
> Basically, Unicode is an in-memory data format. Python knows about Unicode
> 
> characters (to be technical: code points), but files on disk do not.
> 
> Neither do network protocols, or terminals, or other simple devices. They
> 
> only understand bytes.
> 
> 
> 
> So when you have Unicode text, and you want to write it to a file on disk,
> 
> or print it, or send it over the network to another machine, it has to be
> 
> *encoded* into bytes, and then *decoded* back into Unicode when you read it
> 
> from the file again. Sometimes the system will "helpfully" do that encoding
> 
> and decoding automatically for you, which is fine when it works but when it
> 
> doesn't it can be perplexing.
> 
> 
> 
> There are many, many, many different *encoding schemes*. ASCII is one. UTF-8
> 
> is another. And then there are about a bazillion legacy encodings which, if
> 
> you are lucky, you will never need to care about. Only some encodings can
> 
> deal with the entire range of Unicode characters, most can only deal with a
> 
> (typically small) subset of possible characters. E.g. ASCII only knows
> 
> about 127 characters out of the million-plus that Unicode deals with.
> 
> Latin-1 can handle close to 256 different characters. If you have a say in
> 
> the matter, always use UTF-8, since it can handle the full set of Unicode
> 
> characters in the most efficient manner.
> 
> 
> 
> 
> 
> -- 
> 
> Steven

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

UnicodeEncodeError when not running script from IDE Magnus Pettersson <magpettersson@gmail.com> - 2013-02-12 02:43 -0800
  Re: UnicodeEncodeError when not running script from IDE Andrew Berg <bahamutzero8825@gmail.com> - 2013-02-12 05:01 -0600
    Re: UnicodeEncodeError when not running script from IDE Magnus Pettersson <magpettersson@gmail.com> - 2013-02-12 06:24 -0800
      Re: UnicodeEncodeError when not running script from IDE Peter Otten <__peter__@web.de> - 2013-02-12 15:49 +0100
        Re: UnicodeEncodeError when not running script from IDE Magnus Pettersson <magpettersson@gmail.com> - 2013-02-12 07:29 -0800
          Re: UnicodeEncodeError when not running script from IDE Peter Otten <__peter__@web.de> - 2013-02-12 16:48 +0100
          Re: UnicodeEncodeError when not running script from IDE Dave Angel <davea@davea.name> - 2013-02-12 10:58 -0500
            Re: UnicodeEncodeError when not running script from IDE Magnus Pettersson <magpettersson@gmail.com> - 2013-02-12 09:12 -0800
              Re: UnicodeEncodeError when not running script from IDE Fabio Zadrozny <fabiofz@gmail.com> - 2013-02-12 18:04 -0200
              Re: UnicodeEncodeError when not running script from IDE Dave Angel <davea@davea.name> - 2013-02-12 15:51 -0500
                Re: UnicodeEncodeError when not running script from IDE Magnus Pettersson <magpettersson@gmail.com> - 2013-02-12 16:20 -0800
                Re: UnicodeEncodeError when not running script from IDE Dave Angel <davea@davea.name> - 2013-02-12 22:51 -0500
              Re: UnicodeEncodeError when not running script from IDE Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-02-13 11:21 +1100
                Re: UnicodeEncodeError when not running script from IDE Magnus Pettersson <magpettersson@gmail.com> - 2013-02-12 16:40 -0800
        Re: UnicodeEncodeError when not running script from IDE Magnus Pettersson <magpettersson@gmail.com> - 2013-02-12 07:29 -0800
      Re: UnicodeEncodeError when not running script from IDE MRAB <python@mrabarnett.plus.com> - 2013-02-12 21:03 +0000
    Re: UnicodeEncodeError when not running script from IDE Magnus Pettersson <magpettersson@gmail.com> - 2013-02-12 06:24 -0800
  Re: UnicodeEncodeError when not running script from IDE Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-02-12 22:43 +1100
    Re: UnicodeEncodeError when not running script from IDE Magnus Pettersson <magpettersson@gmail.com> - 2013-02-12 04:34 -0800
      Re: UnicodeEncodeError when not running script from IDE Terry Reedy <tjreedy@udel.edu> - 2013-02-12 11:07 -0500

csiph-web