Path: csiph.com!usenet.pasdenom.info!news.redatomik.org!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <877fusybkb.fsf@elektro.pacujo.net>
References: <aae131a7-29a1-4f79-ac16-d1e223616c51@googlegroups.com> <CAPTjJmreaPu7MZQgmbFNnhhg9R6w9dHPPo=yBbMoG85HxK+H_Q@mail.gmail.com> <mailman.19274.1424970167.18130.python-list@python.org> <9169f3b1-2ac7-42a3-8033-584f84b88a1f@googlegroups.com> <7a75a23c-4678-4d7a-a2ec-9e8fff4c07f8@googlegroups.com> <132d5ce6-f672-4eec-99f9-1cc9e88b94f3@googlegroups.com> <mailman.33.1425444900.21433.python-list@python.org> <619e4cb5-1c4c-449b-a5d7-951101b32b45@googlegroups.com> <54f862ca$0$13014$c3e8da3$5496439d@news.astraweb.com> <c6caaa76-f448-4c2f-8874-c1f2716da744@googlegroups.com> <54fadc70$0$13004$c3e8da3$5496439d@news.astraweb.com> <87twxxxbvd.fsf@elektro.pacujo.net> <54fb1bf4$0$12993$c3e8da3$5496439d@news.astraweb.com> <87twxw4xlz.fsf@elektro.pacujo.net> <mailman.145.1425745085.21433.python-list@python.org> <87k2ysydtk.fsf@elektro.pacujo.net> <mailman.148.1425746496.21433.python-list@python.org> <87bnk4yci1.fsf@elektro.pacujo.net> <mailman.152.1425747654.21433.python-list@python.org> <877fusybkb.fsf@elektro.pacujo.net>
Date: Sun, 8 Mar 2015 04:26:58 +1100
Subject: Re: Newbie question about text encoding
From: Chris Angelico <rosuav@gmail.com>
Cc: "python-list@python.org" <python-list@python.org>
Content-Type: text/plain; charset=UTF-8
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.154.1425749228.21433.python-list@python.org>
Lines: 52
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:87114

On Sun, Mar 8, 2015 at 4:14 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> See:
>
>    $ mkdir /tmp/xyz
>    $ touch /tmp/xyz/
> \x80'
>    $ python3
>    Python 3.3.2 (default, Dec  4 2014, 12:49:00)
>    [GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux
>    Type "help", "copyright", "credits" or "license" for more information.
>    >>> import os
>    >>> os.listdir('/tmp/xyz')
>    ['\udc80']
>    >>> open(os.listdir('/tmp/xyz')[0])
>    Traceback (most recent call last):
>      File "<stdin>", line 1, in <module>
>    FileNotFoundError: [Errno 2] No such file or directory: '\udc80'
>
> File names encoded with Latin-X are quite commonplace even in UTF-8
> locales.

That is not a problem with UTF-8, though. I don't understand how
you're blaming UTF-8 for that. There are two things happening here:

1) The underlying file system is not UTF-8, and you can't depend on
that, ergo the decode to Unicode has to have some special handling of
failing bytes.
2) You forgot to put the path on that, so it failed to find the file.
Here's my version of your demo:

>>> open("/tmp/xyz/"+os.listdir('/tmp/xyz')[0])
<_io.TextIOWrapper name='/tmp/xyz/\udc80' mode='r' encoding='UTF-8'>

Looks fine to me.

Alternatively, if you pass a byte string to os.listdir, you get back a
list of byte string file names:

>>> os.listdir(b"/tmp/xyz")
[b'\x80']
>>> open(b"/tmp/xyz/"+os.listdir(b'/tmp/xyz')[0])
<_io.TextIOWrapper name=b'/tmp/xyz/\x80' mode='r' encoding='UTF-8'>

Either way works. You can use bytes or text, and if you use text,
there is a way to smuggle bytes through it. None of this has anything
to do with UTF-8 as an encoding. (Note that the "encoding='UTF-8'"
note in the response has to do with the presumed encoding of the file
contents, not of the file name. As an empty file, it can be considered
to be a stream of zero Unicode characters, encoded UTF-8, so that's
valid.)

ChrisA