Path: csiph.com!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: Peter Otten <__peter__@web.de>
Newsgroups: comp.lang.python
Subject: Re: trying to force stdout to utf-8 with errors='ignore' or 'replace'
Date: Fri, 11 Dec 2015 12:37:01 +0100
Organization: None
Lines: 57
Message-ID: <mailman.131.1449833844.12405.python-list@python.org>
References: <k6nqjcx307.ln2@news.ducksburg.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7Bit
User-Agent: KNode/4.13.3
Precedence: list
Xref: csiph.com comp.lang.python:100257

Adam Funk wrote:

> I'm fiddling with a program that reads articles in the news spool
> using email.parser (standard library) &
> email_reply_parser.EmailReplyParser (installed with pip).  Reading is
> fine, & I don't get any errors writing output extracted from article
> bodies *until* I try to suppress invalid characters.  This works:
> 
>     if message.is_multipart():
>         body = message.get_payload(0, True)
>     else:
>         body = message.get_payload()
>     main_body = EmailReplyParser.parse_reply(body)
>     # fix quoted-printable stuff
>     if equals_regex.search(main_body):
>         main_body = quopri.decodestring(main_body)
>     # suppress attribution before quoted text
>     main_body = attrib_regex.sub('>', main_body)
>     # suppress sig
>     main_body = sig_regex.sub('\n', main_body)
>     main_body.strip()
>     stdout.write(main_body + '\n\n')
> 
> but the stdout includes invalid characters.  I tried adding this at
> the beginning
> 
>     if stdout.encoding is None:
>        writer = codecs.getwriter("utf-8")
>        stdout = writer(stdout, errors='replace')
> 
> and changing the output line to
> 
>     stdout.write(main_body.encode('utf-8', errors='replace') + '\n\n')
> 
> but with either or both of those, I get the dreaded
> "UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
> 562: ordinal not in range(128)".  How can I force the output to be in
> UTF-8 & silently suppress invalid characters?

(I'm assuming you are using Python 2 and that main_body is a unicode 
instance)

Note that you are getting a *decode* error.

Either feed unicode to the modified stdout

>     writer = codecs.getwriter("utf-8")
>     stdout = writer(stdout, errors='replace')

      stdout.write(main_body + u'\n\n')

or encode manually and write to the original sys.stdout:

      sys.stdout.write(main_body.encode('utf-8', errors='replace') + '\n\n')