Path: csiph.com!usenet.pasdenom.info!news.redatomik.org!newsfeed.xs4all.nl!newsfeed2a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <54f9bea1$0$12994$c3e8da3$5496439d@news.astraweb.com>
References: <aae131a7-29a1-4f79-ac16-d1e223616c51@googlegroups.com> <201502241620.t1OGKf4n002146@fido.openend.se> <54ECB134.5090304@davea.name> <201502241945.t1OJjshO013092@fido.openend.se> <201502241957.t1OJvrJS015604@fido.openend.se> <mailman.19148.1424810518.18130.python-list@python.org> <ef520397-b1f0-47bf-8d24-585a9ba230e2@googlegroups.com> <CAPTjJmreaPu7MZQgmbFNnhhg9R6w9dHPPo=yBbMoG85HxK+H_Q@mail.gmail.com> <mailman.19274.1424970167.18130.python-list@python.org> <9169f3b1-2ac7-42a3-8033-584f84b88a1f@googlegroups.com> <7a75a23c-4678-4d7a-a2ec-9e8fff4c07f8@googlegroups.com> <132d5ce6-f672-4eec-99f9-1cc9e88b94f3@googlegroups.com> <mailman.33.1425444900.21433.python-list@python.org> <619e4cb5-1c4c-449b-a5d7-951101b32b45@googlegroups.com> <54f862ca$0$13014$c3e8da3$5496439d@news.astraweb.com> <c6caaa76-f448-4c2f-8874-c1f2716da744@googlegroups.com> <mailman.88.1425619223.21433.python-list@python.org> <01dd9b83-db3e-4e7d-9022-dc6af75eb570@googlegroups.com> <54f9bea1$0$12994$c3e8da3$5496439d@news.astraweb.com>
Date: Sat, 7 Mar 2015 02:27:22 +1100
Subject: Re: Newbie question about text encoding
From: Chris Angelico <rosuav@gmail.com>
Cc: "python-list@python.org" <python-list@python.org>
Content-Type: text/plain; charset=UTF-8
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.114.1425655645.21433.python-list@python.org>
Lines: 55
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:87027

On Sat, Mar 7, 2015 at 1:50 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Rustom Mody wrote:
>
>> On Friday, March 6, 2015 at 10:50:35 AM UTC+5:30, Chris Angelico wrote:
>
> [snip example of an analogous situation with NULs]
>
>> Strawman.
>
> Sigh. If I had a dollar for every time somebody cried "Strawman!" when what
> they really should say is "Yes, that's a good argument, I'm afraid I can't
> argue against it, at least not without considerable thought", I'd be a
> wealthy man...

If I had a dollar for every time anyone said "If I had <insert
currency unit here> for every time...", I'd go meta all day long and
profit from it... :)

> - If you are writing your own file system layer, it's 2015 fer fecks sake,
> file names should be Unicode strings, not bytes! (That's one part of the
> Unix model that needs to die.) You can use UTF-8 or UTF-16 in the file
> system, whichever you please, but again remember that both are
> variable-width formats.

I agree that that part of the Unix model needs to change, but there
are two viable ways to move forward:

1) Keep file names as bytes, but mandate that they be valid UTF-8
streams, and recommend that they be decoded UTF-8 for display to a
human
2) Change the entire protocol stack from the file system upwards so
that file names become Unicode strings.

Trouble with #2 is that file names need to be passed around somehow,
which means bytes in memory. So ultimately, #2 really means "keep file
names as bytes, and mandate an encoding all the way up the stack"...
so it's a massive documentation change that really comes down to the
same thing as #1.

This is one area where, as I understand it, Mac OS got it right. It's
time for other Unix variants to adopt the same policy. The bulk of
file names will be ASCII-only anyway, so requiring UTF-8 won't affect
them; a lot of others are already UTF-8; so all we need is a
transition scheme for the remaining ones. If there's a known FS
encoding, it ought to be possible to have a file system conversion
tool that goes through everything, decodes, re-encodes UTF-8, and then
flags the file system as UTF-8 compliant. All that'd be left would be
the file names that are broken already - ones that don't decode in the
FS encoding - and there's nothing to be done with them but wrap them
up into something probably-meaningless-but reversible.

When can we start doing this? ext5?

ChrisA