Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed3a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Peter Otten <__peter__@web.de>
Subject: Re: Unicode in cgi-script with apache2
Date: Sat, 16 Aug 2014 13:17:16 +0200
Organization: None
References: <53EE4D11.7040604@ramaekers-stassart.be>
Mime-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7Bit
User-Agent: KNode/4.13.2
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.13050.1408187854.18130.python-list@python.org>
Lines: 68
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:76404

Dominique Ramaekers wrote:

> I've got a little script:
> 
> #!/usr/bin/env python3
> print("Content-Type: text/html")
> print("Cache-Control: no-cache, must-revalidate")    # HTTP/1.1
> print("Expires: Sat, 26 Jul 1997 05:00:00 GMT") # Date in the past
> print("")
> f = open("/var/www/cgi-data/index.html", "r")
> for line in f:
>      print(line,end='')
> 
> If I run the script in the terminal, it nicely prints the webpage
> 'index.html'.
> 
> If access the script through a webbrowser, apache gives an error:
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
> 1791: ordinal not in range(128)
> 
> I've done a hole afternoon of reading on fora and blogs, I don't have a
> solution.
> 
> Can anyone help me?

If the input and output encoding are the same you can avoid the byte-to-text 
(and subsequent text-to-byte conversion) and serve the binary contents of 
the index.html file directly:

#!/usr/bin/env python3
import sys

print("Content-Type: text/html")
print("Cache-Control: no-cache, must-revalidate")    # HTTP/1.1
print("Expires: Sat, 26 Jul 1997 05:00:00 GMT") # Date in the past
print("")
sys.stdout.flush()
with open("/var/www/cgi-data/index.html", "rb") as f:
    for line in f:
        sys.stdout.buffer.write(line)

The flush() is necessary to write pending data before accessing the lowlevel 
stdout.buffer. Instead of the loop you can use any of these:

sys.stdout.buffer.write(f.read()) # not for huge files, but should be OK for
                                  # typical html file sizes
sys.stdout.buffer.writelines(f)
shutil.copyfileobj(f, sys.stdout.buffer) # show off your knowledge 
                                         # of the stdlib ;)


Alternatively you could choose an encoding via the locale:

#!/usr/bin/env python3
import locale
locale.setlocale(locale.LC_ALL, "en_US.UTF-8")

print("Content-Type: text/html")
print("Cache-Control: no-cache, must-revalidate")    # HTTP/1.1
print("Expires: Sat, 26 Jul 1997 05:00:00 GMT") # Date in the past
print("")
with open("/var/www/cgi-data/index.html") as f:
    for line in f:
        print(line, end='')

Python should then use UTF-8 as the default for i/o and the resulting 
scripts looks more familiar.