Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!xlned.com!feeder5.xlned.com!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <j.bagg@kent.ac.uk>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.111
X-Spam-Level: *
X-Spam-Evidence: '*H*': 0.78; '*S*': 0.00; 'python': 0.11; "'utf-8',": 0.16; 'header:User-Agent:1': 0.23; 'unicode': 0.24; 'script': 0.25; 'characters': 0.30; "i'm": 0.30; 'too.': 0.31; 'obscure': 0.31; 'probably': 0.32; 'languages': 0.32; 'run': 0.32; 'ago': 0.33; 'raw': 0.33; 'problem': 0.35; 'done': 0.36; "i'll": 0.36; 'subject:?': 0.36; 'to:addr:python-list': 0.38; 'files': 0.38; 'to:addr:python.org': 0.39; 'received:org': 0.40; 'remove': 0.60; 'more': 0.64
Date: Tue, 24 Sep 2013 14:01:04 +0100
From: "J. Bagg" <j.bagg@kent.ac.uk>
Organization: Dept of Anthropology, University of Kent
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.1.16) Gecko/20101125 Thunderbird/3.0.11
MIME-Version: 1.0
To: python-list@python.org
Subject: removing BOM prepended by codecs?
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Mailman-Approved-At: Tue, 24 Sep 2013 15:04:00 +0200
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.294.1380027842.18130.python-list@python.org>
Lines: 18
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1380027842 news.xs4all.nl 15967 [2001:888:2000:d::a6]:60600
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:54695

I'm using:

outputfile = codecs.open (fn, 'w+', 'utf-8', errors='strict')

to write as I know that the files are unicode compliant. I run the raw 
files that are delivered through a Python script to check the unicode 
and report problem characters which are then edited. The files use a 
whole variety of languages from Sanskrit to Cyrillic and more obscure 
ones too.

I'll probably have to remove it in the servlet as we have standardised 
on utf-8. This was done some years ago when utf-16 was rare (apart from 
Macs).

J