Path: csiph.com!goblin3!goblin1!goblin.stu.neva.ru!uio.no!news.tele.dk!news.tele.dk!small.news.tele.dk!newsgate.cistron.nl!newsgate.news.xs4all.nl!nzpost1.xs4all.net!not-for-mail
Return-Path: <python@mrabarnett.plus.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.000
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; '(except': 0.05; 'mrab': 0.05; 'indicates': 0.09; 'literal': 0.09; 'posted.': 0.09; 'python.': 0.11; 'wed,': 0.15; 'encoding': 0.15; 'file,': 0.15; '23,': 0.16; '255': 0.16; 'from:addr:mrabarnett.plus.com': 0.16; 'from:addr:python': 0.16; 'from:name:mrab': 0.16; 'hex': 0.16; 'mangled': 0.16; 'message-id:@mrabarnett.plus.com': 0.16; 'received:192.168.1.4': 0.16; 'subject:non': 0.16; "would've": 0.16; 'wrote:': 0.16; "wouldn't": 0.16; 'bytes': 0.18; 'say,': 0.18; '>>>': 0.20; '2015': 0.20; 'assuming': 0.22; 'occurs': 0.22; 'sep': 0.22; 'appears': 0.23; 'header:In-Reply-To:1': 0.24; 'header:User-Agent:1': 0.26; "doesn't": 0.26; 'starts': 0.29; "i'm": 0.30; 'certain': 0.31; 'surprised': 0.33; 'windows.': 0.33; 'throughout': 0.34; 'file': 0.34; 'could': 0.35; 'but': 0.36; 'beginning': 0.36; 'to:addr:python-list': 0.36; 'pm,': 0.36; 'subject:: ': 0.37; 'being': 0.37; 'signature': 0.37; 'mean': 0.38; 'does': 0.39; 'received:192': 0.39; 'to:addr:python.org': 0.40; 'called': 0.40; 'email addr:gmail.com': 0.62; 'better.': 0.66; "they're": 0.66; 'case?': 0.84; "op's": 0.84
X-CM-Score: 0.00
X-CNFS-Analysis: v=2.1 cv=CvRCCSMD c=1 sm=1 tr=0 a=0nF1XD0wxitMEM03M9B4ZQ==:117 a=0nF1XD0wxitMEM03M9B4ZQ==:17 a=0Bzu9jTXAAAA:8 a=EBOSESyhAAAA:8 a=IkcTkHD0fZMA:10 a=pGLkceISAAAA:8 a=PIQvzTxkMM9OA8DrQT8A:9 a=QEXdDO2ut3YA:10
X-AUTH: mrabarnett@:2500
Subject: Re: Readlines returns non ASCII character
To: python-list@python.org
References: <CAFHq_S4dQxkQoHhP0hQfNvZ0KVtz2F-PFqbePOyYUGFXCLqipg@mail.gmail.com> <CAJ4+4aoMn+93oQsVXnDF2wmnpDfUbg9zna==v2RVh2DP_o4otg@mail.gmail.com> <CAFHq_S6wVanCc6GKRRRWdYwFcqzq2Cv-Cdz3KPz=hX0w9jZdPw@mail.gmail.com> <CAJ4+4aqoYLeeShM+PSR5hxdHa=v_d=XGCY_MifxUPQUwVKuuMQ@mail.gmail.com> <56033F56.1020308@mrabarnett.plus.com> <CALwzidnuUP8bL7cTP05VTCtC_1ok8i4VMXEg-exGc-_egu=beA@mail.gmail.com>
From: MRAB <python@mrabarnett.plus.com>
Date: Thu, 24 Sep 2015 03:02:21 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0
MIME-Version: 1.0
In-Reply-To: <CALwzidnuUP8bL7cTP05VTCtC_1ok8i4VMXEg-exGc-_egu=beA@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.20+
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.114.1443060146.28679.python-list@python.org>
Lines: 31
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1443060146 news.xs4all.nl 23747 [2001:888:2000:d::a6]:41619
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:97056

On 2015-09-24 02:37, Ian Kelly wrote:
> On Wed, Sep 23, 2015 at 6:09 PM, MRAB <python@mrabarnett.plus.com> wrote:
>> On 2015-09-24 00:51, paul.hermeneutic@gmail.com wrote:
>>>
>>>   If this starts at the beginning of the file, then it indicates that
>>> the file is UTF-16 (LE).
>>>
>>> UTF-8[t 1]     EF BB BF       239 187 191
>>> UTF-16 (BE)    FE FF          254 255
>>> UTF-16 (LE)    FF FE          255 254
>>> UTF-32 (BE)    00 00 FE FF    0 0 254 255
>>> UTF-32 (LE)    FF FE 00 00    255 254 0 0
>>>
>> The "signature" EF BB BF indicates the encoding called "utf-8-sig" by
>> Python. It occurs on Windows.
>>
>> If the file doesn't start with any of these, then it could be using any
>> encoding (except UTF-16 or UTF-32).
>
> Yes, but what does it mean when the signature is 00 FF 00 FE 00 FF and
> occurs not at the beginning but repeatedly throughout the file, as
> appears in the OP's case?
>
> At least, I'm assuming that the high-order bytes are 00 based on what
> the OP posted. I wouldn't be surprised though if they're just being
> mangled by the terminal, if it happens to be a certain one that will
> not be named but uses CP 1252.
>
Yes, a byte-string literal or a hex dump of, say, the first 256 bytes
would've been better.