Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #26030

Re: Python 2.6 StreamReader.readline()

Show key headers only | View raw

On Wednesday, July 25, 2012 11:02:01 AM UTC+2, Walter Dörwald wrote:
> On 25.07.12 08:09, Ulrich Eckhardt wrote:
> 
> > Am 24.07.2012 17:01, schrieb cpppwner@gmail.com:
> >>      reader = codecs.getreader(encoding)
> >>      lines  =  []
> >>      with open(filename, 'rb') as f:
> >>          lines  = reader(f, 'strict').readlines(keepends=False)
> >>
> >> where encoding == 'utf-16-be'
> >> Everything works fine, except that lines[0] is equal to
> >> codecs.BOM_UTF16_BE
> >> Is this behaviour correct, that the BOM is still present?
> >
> > Yes, assuming the first line only contains that BOM. Technically it's a
> > space character, and why should those be removed?
> 
> If the first "character" in the file is a BOM the file encoding is 
> probably not utf-16-be but utf-16.
> 
> Servus,
>     Walter

The byte order mark, if present, is nothing else than
an encoded 

>>> ud.name('\ufeff')
'ZERO WIDTH NO-BREAK SPACE'

*code point*.

Five "BOM" are possible (Unicode consortium). utf-8-sig, utf-16-be,
utf-16-le, utf-32-be, utf-32-le. The codecs module provide many
aliases.

The fact that utf-16/32 does correspond to -le or to -be may
vary according to the platforms, the compilers, ...

>>> sys.version
'3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit
(Intel)]'
>>> codecs.BOM_UTF16_BE
b'\xfe\xff'
>>> codecs.BOM_UTF16_LE
b'\xff\xfe'
>>> codecs.BOM_UTF16
b'\xff\xfe'
>>>

---

As far as I know, Py 2.7 or Py 3.2 never return a "BOM" when
a file is read correctly.

>>> with open('a-utf-16-be.txt', 'r', encoding='utf-16-be') as f:
...     r = f.readlines()
...     for zeile in r:
...         print(zeile.rstrip())
...         
abc
élève
cœur
€uro
>>> 


jmf

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Python 2.6 StreamReader.readline() cpppwner@gmail.com - 2012-07-24 08:01 -0700
  Re: Python 2.6 StreamReader.readline() Ulrich Eckhardt <ulrich.eckhardt@dominolaser.com> - 2012-07-25 08:09 +0200
    Re: Python 2.6 StreamReader.readline() Walter Dörwald <walter@livinglogic.de> - 2012-07-25 11:02 +0200
      Re: Python 2.6 StreamReader.readline() wxjmfauth@gmail.com - 2012-07-25 03:26 -0700
      Re: Python 2.6 StreamReader.readline() wxjmfauth@gmail.com - 2012-07-25 03:26 -0700

csiph-web