Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.mixmin.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path: <rosuav@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.001
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'encoding': 0.05; 'subject:Python': 0.06; 'correct.': 0.07; 'utf-8': 0.07; 'encode': 0.09; 'subject:into': 0.09; 'cc:addr:python-list': 0.11; 'python': 0.11; 'windows': 0.15; 'apostrophes': 0.16; 'ascii,': 0.16; 'csv': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'grep': 0.16; 'hex': 0.16; 'locating': 0.16; 'reasonably': 0.16; 'sure.': 0.16; 'think.': 0.16; 'thanks,': 0.17; 'wrote:': 0.18; 'any,': 0.19; 'file,': 0.19; 'help.': 0.21; 'feb': 0.22; 'input': 0.22; 'otherwise,': 0.22; 'cc:addr:python.org': 0.22; 'error': 0.23; 'byte': 0.24; 'bytes': 0.24; 'certainly': 0.24; 'skip': 0.24; "haven't": 0.24; 'cc:2**0': 0.24; 'defined': 0.27; 'header :In-Reply-To:1': 0.27; 'chris': 0.29; 'am,': 0.29; "doesn't": 0.30; 'characters': 0.30; 'message-id:@mail.gmail.com': 0.30; "i'm": 0.30; 'that.': 0.31; '(unless': 0.31; 'assumes': 0.31; 'file': 0.32; 'this.': 0.32; 'critical': 0.32; 'up.': 0.33; 'entirely': 0.33; 'actual': 0.34; "i'd": 0.34; 'could': 0.34; 'something': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'crazy': 0.36; 'representing': 0.36; 'should': 0.36; 'easiest': 0.38; 'pm,': 0.38; 'sure': 0.39; 'system.': 0.39; 'helps': 0.61; 'course': 0.61; "you'll": 0.62; 'show': 0.63; 'telling': 0.64; 'worth': 0.66; 'between': 0.67; 'export': 0.74; '2015': 0.84; 'characters,': 0.84; 'to:none': 0.92
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type; bh=mb7+GNReMfMRwRBoNyT5W/zMgHWKCE7WHntkCTiyJ0c=; b=KctZFARWaD8SyrlKMoLPe0LtmnOUKFdODwWK7QWxZZdOVTkG4+45yHiFLqGUyjrIeG 3+CUJqt0YP5VlNd6K2mFunnpkIac+J3ijfjS9MLLSdPdXpLnmwGqBSR1k7JGNWf824dp xaAYNYMh7bnZ6VZzdnFgSp9F0J2KyEaeUFd9cZWw5Zkpsc8Zq6vf0brSWSn8NB209QSZ XBgC/z7FuuiWLfM9jJHKZsWiwHb9dC/EkkUtB+F5MK5CruSfsCpdQgjth9E8GiUdSz4w rFhAPlryvlf2jum7QtzxAohrl+t4WSSCClOas4TSOa3IALCa57uIRkzmUW+ZjAnYl4JW +zIg==
MIME-Version: 1.0
X-Received: by 10.42.52.200 with SMTP id k8mr26437252icg.26.1423503614777; Mon, 09 Feb 2015 09:40:14 -0800 (PST)
In-Reply-To: <CANc-5Uwb51DKan1xgqwWDT2WFAZk-_NcqgLcnQfzYHzQzps3xA@mail.gmail.com>
References: <CANc-5UwYdoPLKuqRbrfbOrQTVafOaZmWZ+8SnDc-TZp1NEXcQg@mail.gmail.com> <CAPTjJmpJV4KYcL50_=0hv+BLKaGX_X9wqQgMJeszMrHgAK+drA@mail.gmail.com> <CANc-5Uwb51DKan1xgqwWDT2WFAZk-_NcqgLcnQfzYHzQzps3xA@mail.gmail.com>
Date: Tue, 10 Feb 2015 04:40:14 +1100
Subject: Re: Python 3.x stuffing utf-8 into SQLite db
From: Chris Angelico <rosuav@gmail.com>
Cc: Python <python-list@python.org>
Content-Type: text/plain; charset=UTF-8
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.18568.1423503624.18130.python-list@python.org>
Lines: 34
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1423503624 news.xs4all.nl 2913 [2001:888:2000:d::a6]:44266
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:85384

On Tue, Feb 10, 2015 at 4:30 AM, Skip Montanaro
<skip.montanaro@gmail.com> wrote:
> On Sun, Feb 8, 2015 at 9:58 PM, Chris Angelico <rosuav@gmail.com> wrote:
>> Those three characters are the CP-1252 decode of the bytes for U+2019
>> in UTF-8 (E2 80 99). Not sure if that helps any, but given that it was
>> an XLSX file, Windows codepages are reasonably likely to show up.
>
> Thanks, Chris. Are you telling me I should have defined the input file
> encoding for my CSV file as CP-1252, or that something got hosed on
> the export from XLSX to CSV? Or something else?
>
> Skip

Well, I'm not entirely sure. If your input file is actually CP-1252
and you try to decode it as UTF-8, you'll almost certainly get an
error (unless of course it's all ASCII, but you know it isn't in this
case). Also, I'd say chardet will be correct. But it might be worth
locating one of those apostrophes in the file and looking at the
actual bytes representing it... because what you may have is a crazy
double-encoded system. If you take a document with U+2019 in it and
encode it UTF-8, then decode it as CP-1252, then re-encode as UTF-8,
you could get that. (I think. Haven't actually checked.) If someone
gave UTF-8 bytes to a program that doesn't know the difference between
bytes and characters, and assumes CP-1252, then you might well get
something like this. Hence, having a look at the exact bytes in the
.CSV file may help.

Easiest might be to pull it up in a hex viewer (I use 'hd' on my
Debian systems), and grep for the critical line. Otherwise, use Python
and try to pull out a line from the byte stream.

Good luck. You may need it.

ChrisA