Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!feeder2.ecngs.de!ecngs!feeder.ecngs.de!xlned.com!feeder1.xlned.com!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'subject:Python': 0.05; '-be': 0.05; 'encoded': 0.05; '(unicode': 0.07; 'character,': 0.07; 'correspond': 0.07; 'present,': 0.07; 'width': 0.07; "'rb')": 0.09; 'codecs': 0.09; 'correct,': 0.09; 'subject:()': 0.09; 'to:addr:comp.lang.python': 0.09; 'cc:addr:python-list': 0.10; '2.7': 0.13; 'encoding': 0.15; "skip:' 30": 0.15; '>>': 0.16; "'r',": 0.16; '-le': 0.16; 'eckhardt': 0.16; 'mark,': 0.16; 'present?': 0.16; 'wrote:': 0.17; 'byte': 0.17; '>>>': 0.18; 'module': 0.19; 'bit': 0.21; '3.2': 0.22; 'assuming': 0.22; 'fine,': 0.22; 'wednesday,': 0.22; '>': 0.23; 'cc:no real name:2**0': 0.24; 'cc:2**1': 0.24; 'cc:addr:python.org': 0.25; 'header:In-Reply-To:1': 0.25; 'header:User-Agent:1': 0.26; '---': 0.26; 'lines': 0.28; 'behaviour': 0.29; 'probably': 0.29; 'file': 0.32; 'correctly.': 0.33; 'equal': 0.33; 'skip:& 20': 0.33; 'received:google.com': 0.34; 'platforms,': 0.35; 'received:209.85': 0.35; 'except': 0.36; 'but': 0.36; 'should': 0.36; 'skip:p 20': 0.36; 'possible': 0.37; 'does': 0.37; 'why': 0.37; 'received:209': 0.37; 'far': 0.37; 'subject:: ': 0.38; 'fact': 0.38; 'skip:o 20': 0.38; 'nothing': 0.38; 'space': 0.39; 'where': 0.40; 'july': 0.60; 'from:no real name:2**0': 0.60; 'skip:u 10': 0.60; 'first': 0.61; 'provide': 0.62; 'charset:windows-1252': 0.65; 'zeile': 0.84; 'abc': 0.91; 'subject:skip:S 20': 0.91; 'technically': 0.91 Newsgroups: comp.lang.python Date: Wed, 25 Jul 2012 03:26:42 -0700 (PDT) In-Reply-To: Complaints-To: groups-abuse@google.com Injection-Info: glegroupsg2000goo.googlegroups.com; posting-host=81.62.158.207; posting-account=ung4FAoAAAC46zhHJ0Nsnuox7M5gDvs_ References: <65c7dc3b-3dce-45f2-981b-9c8171418f09@googlegroups.com> User-Agent: G2/1.0 X-Google-Web-Client: true X-Google-IP: 81.62.158.207 MIME-Version: 1.0 Subject: Re: Python 2.6 StreamReader.readline() From: wxjmfauth@gmail.com To: comp.lang.python@googlegroups.com Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: python-list@python.org, Ulrich Eckhardt X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Message-ID: Lines: 71 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1343212004 news.xs4all.nl 6921 [2001:888:2000:d::a6]:38521 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:26030 On Wednesday, July 25, 2012 11:02:01 AM UTC+2, Walter D=F6rwald wrote: > On 25.07.12 08:09, Ulrich Eckhardt wrote: >=20 > > Am 24.07.2012 17:01, schrieb cpppwner@gmail.com: > >> reader =3D codecs.getreader(encoding) > >> lines =3D [] > >> with open(filename, 'rb') as f: > >> lines =3D reader(f, 'strict').readlines(keepen= ds=3DFalse) > >> > >> where encoding =3D=3D 'utf-16-be' > >> Everything works fine, except that lines[0] is equal to > >> codecs.BOM_UTF16_BE > >> Is this behaviour correct, that the BOM is still present? > > > > Yes, assuming the first line only contains that BOM. Technically it&= #39;s a > > space character, and why should those be removed? >=20 > If the first "character" in the file is a BOM the file encoding= is=20 > probably not utf-16-be but utf-16. >=20 > Servus, > Walter The byte order mark, if present, is nothing else than an encoded=20 >>> ud.name('\ufeff') 'ZERO WIDTH NO-BREAK SPACE' *code point*. Five "BOM" are possible (Unicode consortium). utf-8-sig, utf-16-be, utf-16-le, utf-32-be, utf-32-le. The codecs module provide many aliases. The fact that utf-16/32 does correspond to -le or to -be may vary according to the platforms, the compilers, ... >>> sys.version '3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit (Intel)]' >>> codecs.BOM_UTF16_BE b'\xfe\xff' >>> codecs.BOM_UTF16_LE b'\xff\xfe' >>> codecs.BOM_UTF16 b'\xff\xfe' >>> --- As far as I know, Py 2.7 or Py 3.2 never return a "BOM" when a file is read correctly. >>> with open('a-utf-16-be.txt', 'r', encoding=3D'utf-16-be') as f: ... r =3D f.readlines() ... for zeile in r: ... print(zeile.rstrip()) ... =20 abc =E9l=E8ve c=9Cur =80uro >>>=20 jmf