Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!xlned.com!feeder5.xlned.com!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Sun, 9 Jun 2013 19:16:06 +1000
From: Cameron Simpson <cs@zip.com.au>
To: "Steven D'Aprano" <steve+comp.lang.python@pearwood.info>
Subject: Re: Changing filenames from Greeklish => Greek (subprocess complain)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <51b4398a$0$30001$c3e8da3$5496439d@news.astraweb.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
References: <51b4398a$0$30001$c3e8da3$5496439d@news.astraweb.com>
Cc: python-list@python.org
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.2912.1370769399.3114.python-list@python.org>
Lines: 51
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:47439

On 09Jun2013 08:15, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
| On Sun, 09 Jun 2013 00:00:53 -0700, nagia.retsina wrote:
| > path = b'/home/nikos/public_html/data/apps/'
| > files = os.listdir( path )
| > 
| > for filename in files:
| > 	# Compute 'path/to/filename'
| > 	filepath_bytes = path + filename
| > 	for encoding in ('utf-8', 'iso-8859-7', 'latin-1'):
| > 		try:
| > 			filepath = filepath_bytes.decode( encoding )
| > 		except UnicodeDecodeError:
| > 			continue
| >         
| > 		# Rename to something valid in UTF-8
| > 		if encoding != 'utf-8':
| > 			os.rename( filepath_bytes, 
| >                                  filepath.encode('utf-8') )
| > 		assert os.path.exists( filepath )
| > 		break
| > 	else:
| > 		# This only runs if we never reached the break 
| >               raise ValueError(
| >                     'unable to clean filename %r' % filepath_bytes )
| 
| Editing the traceback to get rid of unnecessary noise from the logging:
| 
| Traceback (most recent call last):
|   File "/home/nikos/public_html/cgi-bin/files.py", line 83, in <module>
|   assert os.path.exists( filepath )
|   File "/usr/local/lib/python3.3/genericpath.py", line 18, in exists
|   os.stat(path)
| UnicodeEncodeError: 'ascii' codec can't encode characters in position 
| 34-37: ordinal not in range(128)
| 
| > Why am i still receing unicode decore errors? With the help of you guys
| > we have writen a prodecure just to avoid this kind of decoding issues
| > and rename all greek_byted_filenames to utf-8_byted.
| 
| That's a very good question. It works for me when I test it, so I cannot 
| explain why it fails for you.

If he's lucky the UnicodeEncodeError occurred while trying to print
an error message, printing a greek Unicode string in the error with
ASCII as the output encoding (default when not a tty IIRC).

Cheers,
-- 
Cameron Simpson <cs@zip.com.au>

I generally avoid temptation unless I can't resist it.  - Mae West