Path: csiph.com!eternal-september.org!feeder.eternal-september.org!border1.nntp.ams1.giganews.com!nntp.giganews.com!newsfeed.xs4all.nl!newsfeed8.news.xs4all.nl!nzpost1.xs4all.net!not-for-mail
MIME-Version: 1.0
In-Reply-To: <c085c6af-31f6-480c-a9b4-90f46441fdd1@googlegroups.com>
References: <c085c6af-31f6-480c-a9b4-90f46441fdd1@googlegroups.com>
From: Chris Kaynor <ckaynor@zindagigames.com>
Date: Tue, 25 Aug 2015 14:28:36 -0700
Subject: Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
To: "python-list@python.org" <python-list@python.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.34.1440538145.11709.python-list@python.org>
Lines: 83
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:95641

On Tue, Aug 25, 2015 at 2:19 PM, RAH <rene.heymans@gmail.com> wrote:
> Dear All,
>
> I experienced an incomprehensible behavior (I've spent already many hours=
 on this subject): the `file.write('string')` provides an error in run mode=
 and not when interpreted at the console. The string must contain non-ASCII=
 characters. If all ASCII, there is no error.
>
> The following example shows what I can see. I must overlook something bec=
ause I cannot think Python makes a difference between interpreted and run m=
odes and yet ... Can someone please check that subject.
>
> Thank you in advance.
> Ren=C3=A9
>
> Code extract from WSGI application (reply.py)
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
>     request_body =3D environ['wsgi.input'].read(request_body_size)    # b=
ytes
>     rb =3D request_body.decode()                                      # s=
tring
>     d =3D parse_qs(rb)                                                # d=
ict
>
>     f =3D open('logbytes', 'ab')
>     g =3D open('logstr', 'a')
>     h =3D open('logdict', 'a')
>
>     f.write(request_body)
>     g.write(str(type(request_body)) + '\t' + str(type(rb)) + '\t' + str(t=
ype(d)) + '\n')
>     h.write(str(d) + '\n')      <--- line 28 of the application
>
>     h.close()
>     g.close()
>     f.close()
>
>
> Tail of Apache2 error.log
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D
>
> [Tue Aug 25 20:24:04.657933 2015] [wsgi:error] [pid 3677:tid 3029764928] =
[remote 192.168.1.5:27575]   File "reply.py", line 28, in application
> [Tue Aug 25 20:24:04.658001 2015] [wsgi:error] [pid 3677:tid 3029764928] =
[remote 192.168.1.5:27575]     h.write(str(d) + '\\n')
> [Tue Aug 25 20:24:04.658201 2015] [wsgi:error] [pid 3677:tid 3029764928] =
[remote 192.168.1.5:27575] UnicodeEncodeError: 'ascii' codec can't encode c=
haracter '\\xc7' in position 15: ordinal not in range(128)
>

What version of Python is Apache2 using? From the looks of the error,
it is probably using some version of Python2, in which case you'll
need to manually encode the string and pick an encoding for the file
(via an encoding argument to the open function). I'd recommend using
UTF-8.

You can log out the value of sys.version to find out the version number.


> Trying similar code within the Python interpreter
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
> rse@Alibaba:~/test$ python
> Python 3.4.0 (default, Jun 19 2015, 14:18:46)
> [GCC 4.8.2] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> di =3D {'userName': ['=C3=87a va !']}    <--- A dictionary
>>>> str(di)
> "{'userName': ['=C3=87a va !']}"           <--- and its string representa=
tion
>>>> type(str(di))
> <class 'str'>                   <--- Is a string indeed
>>>> fi =3D open('essai', 'a')
>>>> fi.write(str(di) + '\n')
> 26                              <--- It works well
>>>> fi.close()
>>>>

In this run, you are using Python 3.4, which defaults to UTF-8.