Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.mixmin.net!feeder.erje.net!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Sun, 14 Oct 2012 14:59:24 +1100
From: Cameron Simpson <cs@zip.com.au>
To: python-list@python.org
Subject: trouble with nested closures: one of my variables is missing...
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.21 (2010-09-15)
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.2161.1350187169.27098.python-list@python.org>
Lines: 113
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:31236

I'm having some trouble with closures when defining a decorator.

TL;DR:
  I have a function that makes a decorator, and only some of the names
  from an outer scope appear in the inner closure's locals().
  And I do not understand why at all.

Let me explain...

Environment: python 2.7.3 on MacOSX Mountain Lion from MacPorts.

Background: I have a decorator called "file_property" that watches a
file for changes and reloads the file at need. Otherwise it returns a
cached value. It has a lock and some sanity checks and works quite well.

Example method:

  class myclass(object):
    @file_property
    def rules(self):
      with open(self._rules_path) as rfp:
        R = parse(rfp)
      return R
  C = myclass()

and using C.rules fetches parses the rule file as needed and caches the
value until the file next changes.

I naively wrote file_property with a bunch of default parameters:

  def file_property(func, attr_name=None, unset_object=None, poll_rate=1):

and because I never overrode these failed to realise they're useless. Fine.

So, I'm rewriting file_property like this:

  def file_property(func):
    return make_file_property()(func)

i.e. it makes a "vanilla" file_property using the default internals.
Now I have make_file_property with the default parameters:

  def make_file_property(attr_name=None, unset_object=None, poll_rate=1):

which I might use as:

  class myclass(object):
    @make_file_property(poll_rate=3)
    def rules(self):
      with open(self._rules_path) as rfp:
        R = parse(rfp)
      return R

The inner function is the same, but it won't reload the file more often
that once every 3 seconds.

However, I can't make my make_file_property function work. I've stripped
the code down and it does this:

  [hg/css-mailfiler]fleet*1> python foo.py
  make_file_property(attr_name=None, unset_object=None, poll_rate=1): locals()={'attr_name': None, 'poll_rate': 1, 'unset_object': None}
  made_file_property(func=<function f at 0x10408b0c8>): locals()={'func': <function f at 0x10408b0c8>, 'unset_object': None}
  Traceback (most recent call last):
    File "foo.py", line 21, in <module>
      def f(self, foo=1):
    File "foo.py", line 4, in file_property
      return make_file_property()(func)
    File "foo.py", line 10, in made_file_property
      if attr_name is None:
  UnboundLocalError: local variable 'attr_name' referenced before assignment

Observe above that 'unset_object' is in locals(), but not 'attr_name'.
This surprises me.

The stripped back code (missing the internals of the file property
watcher) looks like this:

  import sys

  def file_property(func):
    return make_file_property()(func)

  def make_file_property(attr_name=None, unset_object=None, poll_rate=1):
    print >>sys.stderr, "make_file_property(attr_name=%r, unset_object=%r, poll_rate=%r): locals()=%r" % (attr_name, unset_object, poll_rate,locals())
    def made_file_property(func):
      print >>sys.stderr, "made_file_property(func=%r): locals()=%r" % (func, locals())
      if attr_name is None:
        attr_name = '_' + func.__name__
      lock_name = attr_name + '_lock'
      def getprop(self):
        with getattr(self, lock_name):
          # innards removed here
          pass
        return getattr(self, attr_name, unset_object)
      return property(getprop)
    return made_file_property

  @file_property
  def f(self, foo=1):
    print "foo=%r" % (foo,)

  @make_file_property(attr_name="_blah")
  def f2(self, foo=2):
    print "foo=%r" % (foo,)

Can someone explain what I'm doing wrong, or tell me this is a python
bug?
-- 
Cameron Simpson <cs@zip.com.au>

Bolts get me through times of no courage better than courage gets me
through times of no bolts!
        - Eric Hirst <eric@u.washington.edu>