Path: csiph.com!usenet.pasdenom.info!gegeweb.org!usenet-fr.net!nerim.net!novso.com!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Robert Kern <robert.kern@gmail.com>
Subject: Re: Frustrating circular bytes issue
Date: Tue, 26 Jun 2012 17:53:24 +0100
References: <CAFB6qZuUYE5Mp_Td-=LiMagP4UM65WEPLLJ5t33__Sa_bOq2hg@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:13.0) Gecko/20120614 Thunderbird/13.0.1
In-Reply-To: <CAFB6qZuUYE5Mp_Td-=LiMagP4UM65WEPLLJ5t33__Sa_bOq2hg@mail.gmail.com>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.1519.1340729618.4697.python-list@python.org>
Lines: 82
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:24468

On 6/26/12 5:30 PM, J wrote:
> This is driving me batty... more enjoyment with the Python3
> "Everything must be bytes" thing... sigh...
> I have a file that contains a class used by other scripts.  The class
> is fed either a file, or a stream of output from another command, then
> interprets that output and returns a set that the main program can
> use...  confusing, perhaps, but not necessarily important.
>
> The class is created and then called with the load_filename method:
>
>
>   def load_filename(self, filename):
>          logging.info("Loading elements from filename: %s", filename)
>
>          file = open(filename, "rb", encoding="utf-8")
>          return self.load_file(file, filename)

I get this with Python 3.2:

Traceback (most recent call last):
   File "bytes_unicode.py", line 32, in <module>
     d.load_filename(__file__)
   File "bytes_unicode.py", line 6, in load_filename
     file = open(filename, "rb", encoding="utf-8")
ValueError: binary mode doesn't take an encoding argument

Are you sure you are copy-pasting the code that is actually running? Can you 
reduce your test case down to a small self-contained example that runs and 
demonstrates the problem? I suspect that there is some other detail that is 
causing things to fail. The following code works fine for me:

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

import re

class Dummy(object):
     """ This is a dummy file.

     éµ∫é∂∂é∂ üñîçø∂é
     """

     def load_filename(self, filename):
         file = open(filename, "r", encoding="utf-8")
         return self.load_file(file, filename)

     def load_file(self, file, filename="<stream>"):
         for string in self._reader(file):
             print(string)
             if not string:
                 break

     def _reader(self, file, size=4096, delimiter=r"\n{2,}"):
         buffer_old = ""
         while True:
             buffer_new = file.read()
             print(type(buffer_new))
             if not buffer_new:
                 break
             lines = re.split(delimiter, buffer_old + buffer_new)
             buffer_old = lines.pop(-1)

             for line in lines:
                 yield line

         yield buffer_old

if __name__ == '__main__':
     d = Dummy()
     d.load_filename(__file__)


-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco