Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!rt.uk.eu.org!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Date: Tue, 07 May 2013 21:13:11 -0400
From: Dave Angel <davea@davea.name>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130329 Thunderbird/17.0.5
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Making safe file names
References: <51895D03.4000300@gmail.com> <518998FE.6030805@davea.name> <5189A18C.5060109@gmail.com>
In-Reply-To: <5189A18C.5060109@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.1431.1367975609.3114.python-list@python.org>
Lines: 42
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:44925

On 05/07/2013 08:51 PM, Andrew Berg wrote:
> On 2013.05.07 19:14, Dave Angel wrote:
>> You also need to decide how to handle Unicode characters, since they're
>> different for different OS.  In Windows on NTFS, filenames are in
>> Unicode, while on Unix, filenames are bytes.  So on one of those, you
>> will be encoding/decoding if your code is to be mostly portable.
> Characters outside whatever sys.getfilesystemencoding() returns won't be allowed. If the user's locale settings don't support Unicode, my
> program will be far from the only one to have issues with it. Any problem reports that arise from a user moving between legacy encodings
> will generally be ignored. I haven't yet decided how I will handle artist names with characters outside UTF-8,

There aren't any characters "outside UTF-8".  But a character is not "in 
utf-8", it can be encoded by utf-8.

  but inside UTF-16/32 (UTF-16

Nor outside UTF-16 or 32.

> is just fine on Windows/NTFS, but on Unix(-ish) systems, many use UTF-8 in their locale settings).
>> Don't forget that ls and rm may not use the same encoding you're using.
>> So you may not consider it adequate to make the names legal, but you
>> may also want they easily typeable in the shell.
> I don't understand. I have no intention of changing Unicode characters.

So you're comfortable typing arbitrary characters?  what about all the 
characters that have identical displays in your font? What about viewing 
0x07 in the terminal window?  Or 0x04?

>
>
> This is not a Unicode issue since (modern) file systems will happily accept it. The issue is that certain characters (which are ASCII) are
> not allowed on some file systems:
>   \ / : * ? " < > | @ and the NUL character
> The first 9 are not allowed on NTFS, the @ is not allowed on ext3cow, and NUL and / are not allowed on pretty much any file system. Locale
> settings and encodings aside, these 11 characters will need to be escaped.
>

As soon as you have a small, finite list of invalid characters, writing 
an escape system is pretty easy.


-- 
DaveA