Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #38761

Re: UnicodeEncodeError when not running script from IDE

Show all headers | View raw

> Are you sure you are writing the same data? That would mean that pydev 
> 
> changes the default encoding -- which is evil.
> 
> 
> 
> A portable approach would be to use codecs.open() or io.open() instead of 
> 
> the built-in:
> 
> 
> 
> import io
> 
> with io.open(filepath, "a") as f:
> 
>     ...
> 
> 
> 
> io.open() uses UTF-8 by default, but you can specify other encodings with
> 
> io.open(filepath, mode, encoding=whatever).


Interesting. Pydev must be doing something behind the scenes because when i changed open() to io.open() i get error inside of eclipse now:

f.write(card+"\n")
  File "C:\python27\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character u'\u53c8' in position 32: character maps to <undefined>

....

io.open(filepath, "a", encoding="UTF-8") as f: 

Then it works in eclipse. But I seem to be having an encoding problem all over the place that works in eclipse but dosnt work outside of eclipse pydev.

Here is the flow of my data, im terrible at using unicode/encode/decode so could use some help here:

kanji_anki_gui.py:

def on_addButton_clicked(self):
    #code
    # self.kanji.text() comes from a kanji letter written into a pyqt4 QLineEdit
    kanji = unicode(self.kanji.text())
    card = kanji_anki.scrapeKanji(kanji,tags)
    #more code

kanji_anki.py:

def scrapeKanji(kanji, tags="", onlymeaning=False):
    baseurl = unicode("http://www.romajidesu.com/kanji/")
    url = unicode(baseurl+kanji)
    #test to write out url to disk, works outside of eclipse now
    savefile([url])
    
    #getting webpage works fine in eclipse, prints "Oh no..." in terminal
    try:
        page = urllib2.urlopen(url)
    except:
        print "OH no website dont work"
	return None

    #Code that does some scraping and returns a string containing kanji letters
    return card

def savefile(cardlist,filepath="D:/iknow_kanji.txt"):
    with io.open(filepath, "a") as f:
        for card in cardlist:
            f.write(card+"\n")
    return True

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

UnicodeEncodeError when not running script from IDE Magnus Pettersson <magpettersson@gmail.com> - 2013-02-12 02:43 -0800
  Re: UnicodeEncodeError when not running script from IDE Andrew Berg <bahamutzero8825@gmail.com> - 2013-02-12 05:01 -0600
    Re: UnicodeEncodeError when not running script from IDE Magnus Pettersson <magpettersson@gmail.com> - 2013-02-12 06:24 -0800
      Re: UnicodeEncodeError when not running script from IDE Peter Otten <__peter__@web.de> - 2013-02-12 15:49 +0100
        Re: UnicodeEncodeError when not running script from IDE Magnus Pettersson <magpettersson@gmail.com> - 2013-02-12 07:29 -0800
          Re: UnicodeEncodeError when not running script from IDE Peter Otten <__peter__@web.de> - 2013-02-12 16:48 +0100
          Re: UnicodeEncodeError when not running script from IDE Dave Angel <davea@davea.name> - 2013-02-12 10:58 -0500
            Re: UnicodeEncodeError when not running script from IDE Magnus Pettersson <magpettersson@gmail.com> - 2013-02-12 09:12 -0800
              Re: UnicodeEncodeError when not running script from IDE Fabio Zadrozny <fabiofz@gmail.com> - 2013-02-12 18:04 -0200
              Re: UnicodeEncodeError when not running script from IDE Dave Angel <davea@davea.name> - 2013-02-12 15:51 -0500
                Re: UnicodeEncodeError when not running script from IDE Magnus Pettersson <magpettersson@gmail.com> - 2013-02-12 16:20 -0800
                Re: UnicodeEncodeError when not running script from IDE Dave Angel <davea@davea.name> - 2013-02-12 22:51 -0500
              Re: UnicodeEncodeError when not running script from IDE Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-02-13 11:21 +1100
                Re: UnicodeEncodeError when not running script from IDE Magnus Pettersson <magpettersson@gmail.com> - 2013-02-12 16:40 -0800
        Re: UnicodeEncodeError when not running script from IDE Magnus Pettersson <magpettersson@gmail.com> - 2013-02-12 07:29 -0800
      Re: UnicodeEncodeError when not running script from IDE MRAB <python@mrabarnett.plus.com> - 2013-02-12 21:03 +0000
    Re: UnicodeEncodeError when not running script from IDE Magnus Pettersson <magpettersson@gmail.com> - 2013-02-12 06:24 -0800
  Re: UnicodeEncodeError when not running script from IDE Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-02-12 22:43 +1100
    Re: UnicodeEncodeError when not running script from IDE Magnus Pettersson <magpettersson@gmail.com> - 2013-02-12 04:34 -0800
      Re: UnicodeEncodeError when not running script from IDE Terry Reedy <tjreedy@udel.edu> - 2013-02-12 11:07 -0500

csiph-web