Path: csiph.com!usenet.pasdenom.info!news.etla.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
From: Matthew Ruffalo <mmr15@case.edu>
Date: Mon, 09 Feb 2015 12:54:00 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0
MIME-Version: 1.0
To: Skip Montanaro <skip.montanaro@gmail.com>, Python <python-list@python.org>
Subject: Re: Python 3.x stuffing utf-8 into SQLite db
References: <CANc-5UwYdoPLKuqRbrfbOrQTVafOaZmWZ+8SnDc-TZp1NEXcQg@mail.gmail.com> <CAPTjJmpJV4KYcL50_=0hv+BLKaGX_X9wqQgMJeszMrHgAK+drA@mail.gmail.com> <CANc-5Uwb51DKan1xgqwWDT2WFAZk-_NcqgLcnQfzYHzQzps3xA@mail.gmail.com>
In-Reply-To: <CANc-5Uwb51DKan1xgqwWDT2WFAZk-_NcqgLcnQfzYHzQzps3xA@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.18571.1423504827.18130.python-list@python.org>
Lines: 57
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:85387

On 02/09/2015 12:30 PM, Skip Montanaro wrote:
> Thanks, Chris. Are you telling me I should have defined the input file
> encoding for my CSV file as CP-1252, or that something got hosed on
> the export from XLSX to CSV? Or something else?
>
> Skip

Hi Skip-

I think it's most likely that the encoding issues happened in the export
from XLSX to CSV (unless the data is malformed in the original XLSX
file, of course). The file you're reading *is* valid UTF-8, and you're
reading it in text mode, so seeing 'â€™' in certain lines implies that
the files might contain 'â€™'.encode('utf-8') ==
b'\xc3\xa2\xe2\x82\xac\xe2\x84\xa2'. You could verify this with a hex
editor, or use something like the following:

"""
#!/usr/bin/env python3
from argparse import ArgumentParser
                                                                                                                       

def
dump_lines_with_high_bytes(filename):                                                                              

    with open(filename, 'rb') as
f:                                                                                    

        for line in
f:                                                                                                 

            if any(byte >= 0x80 for byte in
line):                                                                     
               
print(line)                                                                                            


if __name__ == '__main__':
    p = ArgumentParser()
    p.add_argument('filename')
    args = p.parse_args()
    dump_lines_with_high_bytes(args.filename)
"""

I'm a bit surprised that LibreOffice would mangle the encoding like this
-- the entire point of using a format like XLSX is to avoid encoding
issues. I just created a blank spreadsheet with Excel 2013 on Windows 7,
pasted a U+2019 RIGHT SINGLE QUOTATION MARK into the top-left cell,
saved it to a .xlsx file, and opened it with LibreOffice Calc 4.2.7.2 on
Linux. The quot character displayed correctly, and I was able to save it
to valid UTF-8 CSV that I could read with Python 3.4 without any mojibake.

I wonder whether the encoding problems happened with whatever data was
used to create your .xlsx file -- as Chris mentioned, this would occur
if UTF-8 bytes are incorrectly decoded as Windows cp1252.

MMR...