Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #25995 > unrolled thread

Python 2.6 StreamReader.readline()

Back to article view | Back to comp.lang.python

Contents

  Python 2.6 StreamReader.readline() cpppwner@gmail.com - 2012-07-24 08:01 -0700
    Re: Python 2.6 StreamReader.readline() Ulrich Eckhardt <ulrich.eckhardt@dominolaser.com> - 2012-07-25 08:09 +0200
      Re: Python 2.6 StreamReader.readline() Walter Dörwald <walter@livinglogic.de> - 2012-07-25 11:02 +0200
        Re: Python 2.6 StreamReader.readline() wxjmfauth@gmail.com - 2012-07-25 03:26 -0700
        Re: Python 2.6 StreamReader.readline() wxjmfauth@gmail.com - 2012-07-25 03:26 -0700

#25995 — Python 2.6 StreamReader.readline()

Hi,

I have a simple question, I'm using something like the following lines in python 2.6.2

    reader = codecs.getreader(encoding)
    lines  =  []
    with open(filename, 'rb') as f:
        lines  = reader(f, 'strict').readlines(keepends=False)

where encoding == 'utf-16-be'
Everything works fine, except that lines[0] is equal to codecs.BOM_UTF16_BE
Is this behaviour correct, that the BOM is still present?

Thanks in advance for your help.

Best,
Stefan

[toc] | [next] | [standalone]

#26023

Am 24.07.2012 17:01, schrieb cpppwner@gmail.com:
>      reader = codecs.getreader(encoding)
>      lines  =  []
>      with open(filename, 'rb') as f:
>          lines  = reader(f, 'strict').readlines(keepends=False)
>
> where encoding == 'utf-16-be'
> Everything works fine, except that lines[0] is equal to codecs.BOM_UTF16_BE
> Is this behaviour correct, that the BOM is still present?

Yes, assuming the first line only contains that BOM. Technically it's a 
space character, and why should those be removed?

Uli

[toc] | [prev] | [next] | [standalone]

#26028

On 25.07.12 08:09, Ulrich Eckhardt wrote:

> Am 24.07.2012 17:01, schrieb cpppwner@gmail.com:
>>      reader = codecs.getreader(encoding)
>>      lines  =  []
>>      with open(filename, 'rb') as f:
>>          lines  = reader(f, 'strict').readlines(keepends=False)
>>
>> where encoding == 'utf-16-be'
>> Everything works fine, except that lines[0] is equal to
>> codecs.BOM_UTF16_BE
>> Is this behaviour correct, that the BOM is still present?
>
> Yes, assuming the first line only contains that BOM. Technically it's a
> space character, and why should those be removed?

If the first "character" in the file is a BOM the file encoding is 
probably not utf-16-be but utf-16.

Servus,
    Walter

[toc] | [prev] | [next] | [standalone]

#26029

On Wednesday, July 25, 2012 11:02:01 AM UTC+2, Walter Dörwald wrote:
> On 25.07.12 08:09, Ulrich Eckhardt wrote:
> 
> > Am 24.07.2012 17:01, schrieb cpppwner@gmail.com:
> >>      reader = codecs.getreader(encoding)
> >>      lines  =  []
> >>      with open(filename, 'rb') as f:
> >>          lines  = reader(f, 'strict').readlines(keepends=False)
> >>
> >> where encoding == 'utf-16-be'
> >> Everything works fine, except that lines[0] is equal to
> >> codecs.BOM_UTF16_BE
> >> Is this behaviour correct, that the BOM is still present?
> >
> > Yes, assuming the first line only contains that BOM. Technically it's a
> > space character, and why should those be removed?
> 
> If the first "character" in the file is a BOM the file encoding is 
> probably not utf-16-be but utf-16.
> 
> Servus,
>     Walter

The byte order mark, if present, is nothing else than
an encoded 

>>> ud.name('\ufeff')
'ZERO WIDTH NO-BREAK SPACE'

*code point*.

Five "BOM" are possible (Unicode consortium). utf-8-sig, utf-16-be,
utf-16-le, utf-32-be, utf-32-le. The codecs module provide many
aliases.

The fact that utf-16/32 does correspond to -le or to -be may
vary according to the platforms, the compilers, ...

>>> sys.version
'3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit
(Intel)]'
>>> codecs.BOM_UTF16_BE
b'\xfe\xff'
>>> codecs.BOM_UTF16_LE
b'\xff\xfe'
>>> codecs.BOM_UTF16
b'\xff\xfe'
>>>

---

As far as I know, Py 2.7 or Py 3.2 never return a "BOM" when
a file is read correctly.

>>> with open('a-utf-16-be.txt', 'r', encoding='utf-16-be') as f:
...     r = f.readlines()
...     for zeile in r:
...         print(zeile.rstrip())
...         
abc
élève
cœur
€uro
>>> 


jmf

[toc] | [prev] | [next] | [standalone]

#26030

On Wednesday, July 25, 2012 11:02:01 AM UTC+2, Walter Dörwald wrote:
> On 25.07.12 08:09, Ulrich Eckhardt wrote:
> 
> > Am 24.07.2012 17:01, schrieb cpppwner@gmail.com:
> >>      reader = codecs.getreader(encoding)
> >>      lines  =  []
> >>      with open(filename, 'rb') as f:
> >>          lines  = reader(f, 'strict').readlines(keepends=False)
> >>
> >> where encoding == 'utf-16-be'
> >> Everything works fine, except that lines[0] is equal to
> >> codecs.BOM_UTF16_BE
> >> Is this behaviour correct, that the BOM is still present?
> >
> > Yes, assuming the first line only contains that BOM. Technically it's a
> > space character, and why should those be removed?
> 
> If the first "character" in the file is a BOM the file encoding is 
> probably not utf-16-be but utf-16.
> 
> Servus,
>     Walter

The byte order mark, if present, is nothing else than
an encoded 

>>> ud.name('\ufeff')
'ZERO WIDTH NO-BREAK SPACE'

*code point*.

Five "BOM" are possible (Unicode consortium). utf-8-sig, utf-16-be,
utf-16-le, utf-32-be, utf-32-le. The codecs module provide many
aliases.

The fact that utf-16/32 does correspond to -le or to -be may
vary according to the platforms, the compilers, ...

>>> sys.version
'3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit
(Intel)]'
>>> codecs.BOM_UTF16_BE
b'\xfe\xff'
>>> codecs.BOM_UTF16_LE
b'\xff\xfe'
>>> codecs.BOM_UTF16
b'\xff\xfe'
>>>

---

As far as I know, Py 2.7 or Py 3.2 never return a "BOM" when
a file is read correctly.

>>> with open('a-utf-16-be.txt', 'r', encoding='utf-16-be') as f:
...     r = f.readlines()
...     for zeile in r:
...         print(zeile.rstrip())
...         
abc
élève
cœur
€uro
>>> 


jmf

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web