Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #100989 > unrolled thread

using __getitem()__ correctly

Back to article view | Back to comp.lang.python

Contents

  using __getitem()__ correctly "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-12-30 12:57 +0000
    Re: using __getitem()__ correctly Chris Angelico <rosuav@gmail.com> - 2015-12-31 00:11 +1100
      Re: using __getitem()__ correctly "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-12-30 14:40 +0000
        Re: using __getitem()__ correctly Ian Kelly <ian.g.kelly@gmail.com> - 2015-12-30 08:35 -0700
          Re: using __getitem()__ correctly "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-12-30 16:58 +0000
            Re: using __getitem()__ correctly Ian Kelly <ian.g.kelly@gmail.com> - 2015-12-30 13:40 -0700
              Re: using __getitem()__ correctly "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-12-30 22:54 +0000
                Re: using __getitem()__ correctly "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-12-30 22:58 +0000
                Re: using __getitem()__ correctly Ben Finney <ben+python@benfinney.id.au> - 2015-12-31 10:13 +1100
                  Re: using __getitem()__ correctly "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-12-30 23:18 +0000
                  Re: using __getitem()__ correctly Steven D'Aprano <steve@pearwood.info> - 2015-12-31 10:50 +1100
                    Re: using __getitem()__ correctly Ben Finney <ben+python@benfinney.id.au> - 2015-12-31 11:21 +1100
                      Re: using __getitem()__ correctly "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-12-31 11:30 +0000
                        Re: using __getitem()__ correctly Ben Finney <ben+python@benfinney.id.au> - 2015-12-31 22:51 +1100
                        Re: using __getitem()__ correctly Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2015-12-31 12:12 +0000
                          Re: using __getitem()__ correctly "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-12-31 13:39 +0000
                        Re: using __getitem()__ correctly Steven D'Aprano <steve@pearwood.info> - 2016-01-01 02:03 +1100
                    Re: using __getitem()__ correctly "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-12-31 11:17 +0000
                      Re: using __getitem()__ correctly Steven D'Aprano <steve@pearwood.info> - 2016-01-01 01:43 +1100
                Re: using __getitem()__ correctly Ian Kelly <ian.g.kelly@gmail.com> - 2015-12-30 17:31 -0700
                  Re: using __getitem()__ correctly "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-12-31 12:45 +0000

Page 1 of 2  [1] 2  Next page →

#100989 — using __getitem()__ correctly

Hello,

I thought __getitem__() was invoked when an object is postfixed with an 
expression in brackets:

  - abc[n]

and __getattr__() was invoked when an object is postfixed with an dot:

  - abc.member

but my __getitem__ is being invoked at this time, where there's no 
subscript (going into a spiral recursion death):

  self.mcc = self.attrs.mcc

Can anybody explain to me why __getitem__()  would be invoked here?

I'm using __getitem__() AND __getattr__() to handle an array of objects 
which don't exist yet (autovivification).  My __getattr__() could handle 
the line above, but I don't know how to handle that in __getitem__():

  attrdict:av:__getitem__: entered for  mcc

That's printed by this line:

  print "attrdict:av:__getitem__: entered for ", key


I expected that "key" would always be numeric or a slice

TIA
cts

[toc] | [next] | [standalone]

#100990

On Wed, Dec 30, 2015 at 11:57 PM, Charles T. Smith
<cts.private.yahoo@gmail.com> wrote:
> Hello,
>
> I thought __getitem__() was invoked when an object is postfixed with an
> expression in brackets:
>
>   - abc[n]
>
> and __getattr__() was invoked when an object is postfixed with an dot:
>
>   - abc.member

That would be normal (with the caveat that __getattr__ is called only
if the attribute isn't found; __getattribute__ is called
unconditionally).

> but my __getitem__ is being invoked at this time, where there's no
> subscript (going into a spiral recursion death):
>
>   self.mcc = self.attrs.mcc
>
> Can anybody explain to me why __getitem__()  would be invoked here?

Can you post your entire class, or at least __getattr__? Also check
superclasses, if there are any.

ChrisA

[toc] | [prev] | [next] | [standalone]

#100999

On Thu, 31 Dec 2015 00:11:24 +1100, Chris Angelico wrote:

> On Wed, Dec 30, 2015 at 11:57 PM, Charles T. Smith
> <cts.private.yahoo@gmail.com> wrote:
>> Hello,
>>
>> I thought __getitem__() was invoked when an object is postfixed with an
>> expression in brackets:
>>
>>   - abc[n]
>>
>> and __getattr__() was invoked when an object is postfixed with an dot:
>>
>>   - abc.member
> 
> That would be normal (with the caveat that __getattr__ is called only if
> the attribute isn't found; __getattribute__ is called unconditionally).
> 
>> but my __getitem__ is being invoked at this time, where there's no
>> subscript (going into a spiral recursion death):
>>
>>   self.mcc = self.attrs.mcc
>>
>> Can anybody explain to me why __getitem__()  would be invoked here?
> 
> Can you post your entire class, or at least __getattr__? Also check
> superclasses, if there are any.
> 
> ChrisA

(PDB)isinstance (self.attrs, attrdict)
True

As is so often the case, in composing my answer to your question, I discovered
a number of problems in my class (e.g. I was calling __getitem__() myself!), but
I'm puzzled now how to proceed.  I thought the way you avoid triggering __getattr__()
from within that was to use self.__dict__[name] but that doesn't work:

  (PDB)p self.attrs.keys()
  ['mcc', 'abc']
  (PDB)p self.attrs.__dict__['abc']
  *** KeyError: KeyError('abc',)



class attrdict(dict):
    def __init__ (self, name = None):
        if name:
            self.update (name)
        print "attrdict: instantiated: ", name
    
    # AutoVivification
    def __getattr__ (self, name):
        print "attrdict:av:__getattr__: entered for ", name      #, " in ", self
        #if not name in self.__dict__.keys():
        if not name in self.keys():
            print "attrdict:av:__getattr__: autovivifying ", name
            #self.__dict__.__setitem__ (name, self.__class__()) 
            #self.__setitem__ (name, self.__class__())
            self.__setattr__ (name, self.__class__())
        #return self.__getitem__(name)
        #return self.__dict__.__getitem__(name)
        return self.__getattribute__ (name)

    def __getitem__ (self, key):
        print "attrdict:av:__getitem__: entered for ", key      #, " in ", self
        return self.__getitem__(key)


    def __getattr__deprecated (self, name):
        return self[name]

    def __setattr__(self, name, value):
        self[name] = value

[toc] | [prev] | [next] | [standalone]

#101001

On Dec 30, 2015 7:46 AM, "Charles T. Smith" <cts.private.yahoo@gmail.com> wrote:
> As is so often the case, in composing my answer to your question, I discovered
> a number of problems in my class (e.g. I was calling __getitem__() myself!), but
> I'm puzzled now how to proceed.  I thought the way you avoid triggering __getattr__()
> from within that was to use self.__dict__[name] but that doesn't work:
>
>   (PDB)p self.attrs.keys()
>   ['mcc', 'abc']
>   (PDB)p self.attrs.__dict__['abc']
>   *** KeyError: KeyError('abc',)

What leads you to believe that this is triggering a call to
__getattr__? The KeyError probably just means that the key 'abc'
wasn't found in the dict.

> class attrdict(dict):
>     def __init__ (self, name = None):
>         if name:
>             self.update (name)
>         print "attrdict: instantiated: ", name
>
>     # AutoVivification
>     def __getattr__ (self, name):
>         print "attrdict:av:__getattr__: entered for ", name      #, " in ", self
>         #if not name in self.__dict__.keys():
>         if not name in self.keys():

Use the "not in" operator, e.g. "if name not in self.keys()".

>             print "attrdict:av:__getattr__: autovivifying ", name
>             #self.__dict__.__setitem__ (name, self.__class__())
>             #self.__setitem__ (name, self.__class__())
>             self.__setattr__ (name, self.__class__())

No reason to explicitly call __setitem__ or __setattr__ here. I'd
probably just do self[name] = self.__class__()

>         #return self.__getitem__(name)
>         #return self.__dict__.__getitem__(name)
>         return self.__getattribute__ (name)

You shouldn't call __getattribute__ from __getattr__, because
__getattr__ is called from __getattribute__, so this would cause an
infinite loop.

Based on the preceding, you probably want to return the value you just
set in the dict, correct? So just return self[name].

[toc] | [prev] | [next] | [standalone]

#101002

On Wed, 30 Dec 2015 08:35:57 -0700, Ian Kelly wrote:

> On Dec 30, 2015 7:46 AM, "Charles T. Smith"
> <cts.private.yahoo@gmail.com> wrote:
>> As is so often the case, in composing my answer to your question, I
>> discovered a number of problems in my class (e.g. I was calling
>> __getitem__() myself!), but I'm puzzled now how to proceed.  I thought
>> the way you avoid triggering __getattr__() from within that was to use
>> self.__dict__[name] but that doesn't work:
>>
>>   (PDB)p self.attrs.keys()
>>   ['mcc', 'abc']
>>   (PDB)p self.attrs.__dict__['abc']
>>   *** KeyError: KeyError('abc',)
> 
> What leads you to believe that this is triggering a call to __getattr__?
> The KeyError probably just means that the key 'abc' wasn't found in the
> dict.


I meant, it doesn't work because I'm not getting at the attribute  Although keys()
sees it, it's not in the __dict__ attribute of attrs.  If it's not there, where is it?


>             print "attrdict:av:__getattr__: autovivifying ", name
>             #self.__dict__.__setitem__ (name, self.__class__())
>             #self.__setitem__ (name, self.__class__()) self.__setattr__
>             (name, self.__class__())
> 
> No reason to explicitly call __setitem__ or __setattr__ here. I'd
> probably just do self[name] = self.__class__()


The reason I used this is to avoid trigging the __setitem__() method:

  self.__setattr__(name, self.__class__())

which is invoked if I use the "self[name]" syntax.  But that didn't work.

Is it just impossible to get at attributes without going through either
__getattr__() or __getitem__()?



> Based on the preceding, you probably want to return the value you just
> set in the dict, correct? So just return self[name].


The problem is that then triggers the __getitem__() method and I don't
know how to get to the attributes without triggering __getattr__().

It's the interplay of the two that's killing me.

In the example, if I have:

  self.mcc = self.attrs.mcc


The crux:

Then __getattr__() triggers for the mcc.  If I try to use self.attrs['mcc']
to get it, then that triggers __getitem__().  Okay, if the key is not an int,
I'll go and get it and return it... unfortunately that triggers __getattr__(),
an infinite loop.

I tried using:

    attrdict.__getattr__ (self, 'mcc')

but that didn't help, of course.  I also tried so, but I've got this wrong, somehow:

    super (attrdict, self).__getattr__ ('mcc')






class attrdict(dict):
    def __init__ (self, name = None):
        if name:
            self.update (name)
        print "attrdict: instantiated: ", name
        
    # AutoVivification
    def __getattr__ (self, name):
        print "attrdict:av:__getattr__: entered for ", name
        if name not in self.keys():
            print "attrdict:av:__getattr__: autovivifying ", name
            self[name] = self.__class__()
        return self[name]

    def __getitem__ (self, key):
        print "attrdict:av:__getitem__: entered for ", key
        if type (key) is int:          # TODO: support slices
            return self.__getitem__(key)
        return attrdict.__getattr__(self, key)

    def __setattr__(self, name, value):
        self[name] = value

[toc] | [prev] | [next] | [standalone]

#101010

On Wed, Dec 30, 2015 at 9:58 AM, Charles T. Smith
<cts.private.yahoo@gmail.com> wrote:
> On Wed, 30 Dec 2015 08:35:57 -0700, Ian Kelly wrote:
>
>> On Dec 30, 2015 7:46 AM, "Charles T. Smith"
>> <cts.private.yahoo@gmail.com> wrote:
>>> As is so often the case, in composing my answer to your question, I
>>> discovered a number of problems in my class (e.g. I was calling
>>> __getitem__() myself!), but I'm puzzled now how to proceed.  I thought
>>> the way you avoid triggering __getattr__() from within that was to use
>>> self.__dict__[name] but that doesn't work:
>>>
>>>   (PDB)p self.attrs.keys()
>>>   ['mcc', 'abc']
>>>   (PDB)p self.attrs.__dict__['abc']
>>>   *** KeyError: KeyError('abc',)
>>
>> What leads you to believe that this is triggering a call to __getattr__?
>> The KeyError probably just means that the key 'abc' wasn't found in the
>> dict.
>
>
> I meant, it doesn't work because I'm not getting at the attribute  Although keys()
> sees it, it's not in the __dict__ attribute of attrs.  If it's not there, where is it?

I think you're probably getting confused because there are three
different dicts at play here:

* Since your attrdict class inherits from dict, self.attrs is a dict.
* self.attrs.__dict__ is a *different* dict, used to store the
instance attributes of self.attrs.
* self.attrs.__class__.__dict__  is another different dict, used to
store the class attributes of attrdict.

The keys method that you're calling above is a method of the
self.attrs dict, which is where your attrdict's __setattr__ is setting
it. That's why you find it there but not in self.attrs.__dict__.

>>             print "attrdict:av:__getattr__: autovivifying ", name
>>             #self.__dict__.__setitem__ (name, self.__class__())
>>             #self.__setitem__ (name, self.__class__()) self.__setattr__
>>             (name, self.__class__())
>>
>> No reason to explicitly call __setitem__ or __setattr__ here. I'd
>> probably just do self[name] = self.__class__()
>
>
> The reason I used this is to avoid trigging the __setitem__() method:
>
>   self.__setattr__(name, self.__class__())
>
> which is invoked if I use the "self[name]" syntax.  But that didn't work.

But the body of your __setattr__ method is just "self[name] =
self.__class__()", which is the exact same code as what I suggested
and will still invoke __setitem__.

That said, I don't get why you're trying to avoid calling __setitem__.
If you're trying to store the attribute as a dict item, as you seem to
be doing, why shouldn't that dict's __setitem__ be involved?

> Is it just impossible to get at attributes without going through either
> __getattr__() or __getitem__()?

No.

>> Based on the preceding, you probably want to return the value you just
>> set in the dict, correct? So just return self[name].
>
>
> The problem is that then triggers the __getitem__() method and I don't
> know how to get to the attributes without triggering __getattr__().
>
> It's the interplay of the two that's killing me.

The only interplay of the two is what you have written into your class.

> In the example, if I have:
>
>   self.mcc = self.attrs.mcc
>
>
> The crux:
>
> Then __getattr__() triggers for the mcc.  If I try to use self.attrs['mcc']
> to get it, then that triggers __getitem__().  Okay, if the key is not an int,
> I'll go and get it and return it... unfortunately that triggers __getattr__(),
> an infinite loop.

How precisely are you trying to store these: as an attribute, or as a
dict item? If it's supposed to be in the dict, then why is your
__getitem__ trying to look up an attribute to begin with?

> class attrdict(dict):
>     def __init__ (self, name = None):
>         if name:
>             self.update (name)
>         print "attrdict: instantiated: ", name
>
>     # AutoVivification
>     def __getattr__ (self, name):
>         print "attrdict:av:__getattr__: entered for ", name
>         if name not in self.keys():
>             print "attrdict:av:__getattr__: autovivifying ", name
>             self[name] = self.__class__()
>         return self[name]
>
>     def __getitem__ (self, key):
>         print "attrdict:av:__getitem__: entered for ", key
>         if type (key) is int:          # TODO: support slices
>             return self.__getitem__(key)

Here the method as written is just going to end up calling itself. You
probably want super(attrdict, self).__getitem__(key)

>         return attrdict.__getattr__(self, key)

And here if you really want to access the instance attributes without
using __getattr__, just use self.__dict__[key].

I don't understand what it is that you're trying to accomplish here by
looking the key up in the instance attributes, though. It looks very
circular. I think you should clearly define where you expect the items
to be stored and then only check that location.

[toc] | [prev] | [next] | [standalone]

#101018

On Wed, 30 Dec 2015 13:40:44 -0700, Ian Kelly wrote:

> On Wed, Dec 30, 2015 at 9:58 AM, Charles T. Smith
>> The problem is that then triggers the __getitem__() method and I don't
>> know how to get to the attributes without triggering __getattr__().
>>
>> It's the interplay of the two that's killing me.
> 
> The only interplay of the two is what you have written into your class.
> 
>> In the example, if I have:
>>
>>   self.mcc = self.attrs.mcc
>>
>>
>> The crux:
>>
>> Then __getattr__() triggers for the mcc.  If I try to use
>> self.attrs['mcc'] to get it, then that triggers __getitem__().  Okay,
>> if the key is not an int, I'll go and get it and return it...
>> unfortunately that triggers __getattr__(), an infinite loop.
> 
> How precisely are you trying to store these: as an attribute, or as a
> dict item? If it's supposed to be in the dict, then why is your
> __getitem__ trying to look up an attribute to begin with?



I don't understand this distinction between an "attribute" and a "dict item".
attrdict is a stupid legacy class that inherits from a dictionary.  I think
the intent was to be faster than a normal class, but I doubt there's any value
to that.

In any case, I thought that class attributes were, in fact, items of __dict__?

The reason that my __getitem__() is trying to look up an attribute is, I
think, because this syntax triggers __getitem__ with a key of "mcc":

  return self[name]

Is that assumption wrong?  That was the reason I was looking to find
another way to get at the attributes, because
  return self.__getattr__(name)
does, too, and this doesn't even find them:
  return self.__getattribute__(name)

Just to establish the concept, this horrible thing is showing some promise:

  attriter = self.iteritems()
  for attr in attriter:
      if attr[0] == name:
          return attr[1]


(because the subscripts there are on a tuple type)

But I concede I must be doing something fundamentally wrong because this
assert is triggering:
    def __getattr__ (self, name):
        print "attrdict:av:__getattr__: entered for ", name
        assert name not in self.keys(), "attrdict:__getattr__: who lied?"
 
  

[toc] | [prev] | [next] | [standalone]

#101022

On Wed, 30 Dec 2015 22:54:44 +0000, Charles T. Smith wrote:
> But I concede I must be doing something fundamentally wrong because this
> assert is triggering:
>     def __getattr__ (self, name):
>         print "attrdict:av:__getattr__: entered for ", name

>         assert name not in self.keys(), "attrdict:__getattr__: who lied?"

Tt's really a hassle how pan(1) sometimes wraps and sometimes doesn't.
Is there a better way to do this?

[toc] | [prev] | [next] | [standalone]

#101023

"Charles T. Smith" <cts.private.yahoo@gmail.com> writes:

> I don't understand this distinction between an "attribute" and a "dict
> item".

When did you most recently work through the Python tutorial
<URL:https://docs.python.org/3/tutorial/>> You may want to work through
it again, from start to finish and exercising each example, to be sure
you have a solid understanding of basic concepts like these.

In brief: Objects have attributes; looking up an attribute on an object
has specific syntax. Dictionaries are collections of items; the items
are looked up by key, using a quite different syntax. Those two
different syntaxes translate to distinct special methods.

You may be familiar with other languages where the distinction between
“attribute of an object” is not distinct from “item in a dictionary”.
Python is not one of those languages; the distinction is real and
important. You'll need to do some remedial learning of Python, and I
recommend working through the Python tutorial.

-- 
 \            “But it is permissible to make a judgment after you have |
  `\    examined the evidence. In some circles it is even encouraged.” |
_o__)                    —Carl Sagan, _The Burden of Skepticism_, 1987 |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#101024

On Thu, 31 Dec 2015 10:13:53 +1100, Ben Finney wrote:

> "Charles T. Smith" <cts.private.yahoo@gmail.com> writes:
> 
>> I don't understand this distinction between an "attribute" and a "dict
>> item".
> 
> When did you most recently work through the Python tutorial
> <URL:https://docs.python.org/3/tutorial/>> You may want to work through
> it again, from start to finish and exercising each example, to be sure
> you have a solid understanding of basic concepts like these.
> 
> In brief: Objects have attributes; looking up an attribute on an object
> has specific syntax. Dictionaries are collections of items; the items
> are looked up by key, using a quite different syntax. Those two
> different syntaxes translate to distinct special methods.
> 
> You may be familiar with other languages where the distinction between
> “attribute of an object” is not distinct from “item in a dictionary”.
> Python is not one of those languages; the distinction is real and
> important. You'll need to do some remedial learning of Python, and I
> recommend working through the Python tutorial.


Thanks, Ben, for your advice.  Actually, I think that Ian and Chris and
I are making fine progress.  I think you misread my question.

[toc] | [prev] | [next] | [standalone]

#101025

On Thu, 31 Dec 2015 10:13 am, Ben Finney wrote:

> You may be familiar with other languages where the distinction between
> “attribute of an object” is not distinct from “item in a dictionary”.
> Python is not one of those languages; the distinction is real and
> important.


I'm not sure what distinction you're referring to, can you explain?

Obviously there is a syntax difference between x.attr and x['key'], but
attributes *are* items in a dictionary (ignoring __slots__ and __getattr__
for the time being). Either the instance __dict__, the class __dict__, or a
superclass __dict__.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#101029

Steven D'Aprano <steve@pearwood.info> writes:

> On Thu, 31 Dec 2015 10:13 am, Ben Finney wrote:
>
> > You may be familiar with other languages where the distinction
> > between “attribute of an object” is not distinct from “item in a
> > dictionary”. Python is not one of those languages; the distinction
> > is real and important.
>
> I'm not sure what distinction you're referring to, can you explain?

Tersely: the relationship between an object and its attributes, is not
the same as the relationship between a dictionary and its items.

> Obviously there is a syntax difference between x.attr and x['key']

Not merely syntax; the attributes of an object are not generally
available as items of the container.

> but attributes *are* items in a dictionary

That's like saying everything in Python is a number: it conflates the
implementation with the semantics.

The distinction between a Python integer and a Python boolean value is
real and important, despite the incidental fact of their both being
implemented as numbers.

> Either the instance __dict__, the class __dict__, or a superclass
> __dict__.

No, I'm not referring to the ‘__dict__’ attribute of an object; I'm
referring to the object itself.

To talk about the attributes of an object ‘foo’ is distinct from talking
about the items in a dictionary ‘foo’. That distinction is real, and
important.

-- 
 \               “… correct code is great, code that crashes could use |
  `\           improvement, but incorrect code that doesn’t crash is a |
_o__)                    horrible nightmare.” —Chris Smith, 2008-08-22 |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#101049

On Thu, 31 Dec 2015 11:21:59 +1100, Ben Finney wrote:

> Steven D'Aprano <steve@pearwood.info> writes:
> 
>> On Thu, 31 Dec 2015 10:13 am, Ben Finney wrote:
>>
>> > You may be familiar with other languages where the distinction
>> > between “attribute of an object” is not distinct from “item in a
>> > dictionary”. Python is not one of those languages; the distinction is
>> > real and important.
...
> 
> Tersely: the relationship between an object and its attributes, is not
> the same as the relationship between a dictionary and its items.


I understand this to mean that the relationship between a dictionary and
its items is less complex than the relationship between an object and
its attributes.


I'd like to catalog the different attribute types/implications:
- perhaps a hierarchy of meta-ism - or user-relevance
- those things in the __dict__
- those things not in the __dict__ but without the "__"
- ???


> 
>> Obviously there is a syntax difference between x.attr and x['key']
> 
> Not merely syntax; the attributes of an object are not generally
> available as items of the container.


What are the set of ways that an attribute is accessible?  Including
implementation implications?

 
> 
>> Either the instance __dict__, the class __dict__, or a superclass
>> __dict__.
> 
> No, I'm not referring to the ‘__dict__’ attribute of an object; I'm
> referring to the object itself.
> 
> To talk about the attributes of an object ‘foo’ is distinct from talking
> about the items in a dictionary ‘foo’. That distinction is real, and
> important.


But wanting to deal with the attributes of an object without considering
the way it's implemented - although possible - requires a complete virtual
model that covers all implications.  It's easier to simply understand how the
objects work under the covers.

[toc] | [prev] | [next] | [standalone]

#101051

"Charles T. Smith" <cts.private.yahoo@gmail.com> writes:

> On Thu, 31 Dec 2015 11:21:59 +1100, Ben Finney wrote:
>
> > Tersely: the relationship between an object and its attributes, is
> > not the same as the relationship between a dictionary and its items.
>
> I understand this to mean that the relationship between a dictionary
> and its items is less complex than the relationship between an object
> and its attributes.

That's not implied by the above, nor is it what I meant.

I meant that the relationship is not the same.

* The attributes of an object ‘foo’ are what you access via ‘foo.bar’
  syntax. You don't access those attributes via ‘foo[bar]’.

* The items of a dictionary ‘foo’ are what you access via ‘foo[bar]’
  syntax. You don't access those items via ‘foo.bar’.

That's an important, and real, difference. And it's much better learned
as part of a comprehensive course on Python, not in a deep dive into the
magic methods.

I really don't know why it's been so difficult to talk about this, but I
hope my meaning is clear now.

Have a wonderful end of year, folks.

-- 
 \              “There are no chaplains in foxholes.” —Sergeant Justin |
  `\                                              Griffith, 2011-07-27 |
_o__)                                                                  |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#101052

On 31 December 2015 at 11:30, Charles T. Smith
<cts.private.yahoo@gmail.com> wrote:
>>> Obviously there is a syntax difference between x.attr and x['key']
>>
>> Not merely syntax; the attributes of an object are not generally
>> available as items of the container.
>
>
> What are the set of ways that an attribute is accessible?  Including
> implementation implications?
>
>
>>
>>> Either the instance __dict__, the class __dict__, or a superclass
>>> __dict__.
>>
>> No, I'm not referring to the ‘__dict__’ attribute of an object; I'm
>> referring to the object itself.
>>
>> To talk about the attributes of an object ‘foo’ is distinct from talking
>> about the items in a dictionary ‘foo’. That distinction is real, and
>> important.
>
>
> But wanting to deal with the attributes of an object without considering
> the way it's implemented - although possible - requires a complete virtual
> model that covers all implications.  It's easier to simply understand how the
> objects work under the covers.

When you write x.attr the name 'attr' is looked up on the object x.
This calls x.__getattribute__('attr'). In turn this checks the dict
associated with the object x i.e. x.__dict__['attr']. This in turn
calls x.__dict__.__getitem__('attr'). The lookup of x.__dict__ is
special and doesn't use the normal __getattribute__ mechanism
(otherwise this would be an infinite recursion). Generally special
attributes (with double underscores) are looked up in a different way.
If x.__dict__ does not have the attribute then the dict associated
with the class/type of x is checked i.e. x.__class__.__dict__['attr'].
The lookup of x.__class__ is also special. Failing this the other
classes in x.__class__.__mro__ are checked i.e.
x.__class__.__mro__[1].__dict__['attr']. Once these are exhausted
x.__getattribute__('attr') falls back on calling
x.__getattr__('attr').

IIUC you're trying to create an attribute dict where the same
attributes can be looked up via x.attr or x['attr'] (i.e.
x.__getattribute__('attr') or x.__getitem__('attr')). One way to do
this is to subclass dict and then set each instances __dict__ to be
itself. This way __getattribute__ will search the instance (which is a
dict subclass) as its own attribute dict. You can then use __getattr__
as the fallback for attributes that are not found. A simple
implementation of this could look like:

class attrdict(dict):
    def __init__(self, name=None):
        self.__dict__ = self
        kwargs = {'name':name} if name is not None else {}
        super(attrdict, self).__init__(**kwargs)
    def __getattr__(self, attrname):
        ob = self[attrname] = type(self)(name=attrname)
        return ob

>>> d = attrdict('foo')
>>> d
{'name': 'foo'}
>>> d.bar
{'name': 'bar'}
>>> d
{'bar': {'name': 'bar'}, 'name': 'foo'}

The problem with this as with any dict subclass approach is that a
dict has a load of methods (.items() .keys() .pop() ...) that will now
become conflated with your dict keys when you use the attribute
access:

>>> d.items
<built-in method items of attrdict object at 0x7f2025eab868>

This means that you can't use any of the dict method names in whatever
you're doing. This is the reason that a dict uses a different
mechanism __getitem__ so that it can store arbitrary keys at the same
time as having named methods which must be attributes. Personally I
think that the best approach is to ditch the idea of conflating
attributes and keys and just use the subscript x['attr'] syntax.

--
Oscar

[toc] | [prev] | [next] | [standalone]

#101055

On Thu, 31 Dec 2015 12:12:43 +0000, Oscar Benjamin wrote:


> When you write x.attr the name 'attr' is looked up on the object x. This
> calls x.__getattribute__('attr'). In turn this checks the dict
> associated with the object x i.e. x.__dict__['attr']. This in turn calls
> x.__dict__.__getitem__('attr'). The lookup of x.__dict__ is special and
> doesn't use the normal __getattribute__ mechanism (otherwise this would
> be an infinite recursion). Generally special attributes (with double
> underscores) are looked up in a different way. If x.__dict__ does not
> have the attribute then the dict associated with the class/type of x is
> checked i.e. x.__class__.__dict__['attr']. The lookup of x.__class__ is
> also special. Failing this the other classes in x.__class__.__mro__ are
> checked i.e. x.__class__.__mro__[1].__dict__['attr']. Once these are
> exhausted x.__getattribute__('attr') falls back on calling
> x.__getattr__('attr').


Very good overview of the steps, thank you.


...> 
> The problem with this as with any dict subclass approach is that a dict
> has a load of methods (.items() .keys() .pop() ...) that will now become
> conflated with your dict keys when you use the attribute access:
> 
>>>> d.items
> <built-in method items of attrdict object at 0x7f2025eab868>


Yeah, this makes it confusing, too.  :)

[toc] | [prev] | [next] | [standalone]

#101057

On Thu, 31 Dec 2015 10:30 pm, Charles T. Smith wrote:

> On Thu, 31 Dec 2015 11:21:59 +1100, Ben Finney wrote:
> 
>> Steven D'Aprano <steve@pearwood.info> writes:
>> 
>>> On Thu, 31 Dec 2015 10:13 am, Ben Finney wrote:
>>>
>>> > You may be familiar with other languages where the distinction
>>> > between “attribute of an object” is not distinct from “item in a
>>> > dictionary”. Python is not one of those languages; the distinction is
>>> > real and important.
> ...
>> 
>> Tersely: the relationship between an object and its attributes, is not
>> the same as the relationship between a dictionary and its items.
> 
> 
> I understand this to mean that the relationship between a dictionary and
> its items is less complex than the relationship between an object and
> its attributes.

I think that is a fair comment, since attribute access involves MUCH more
complexity than dict item access. Attribute access involves one *or more*
dict access, *plus* a whole lot of extra complexity, while dict access by
definition involves only a single dict access.


> I'd like to catalog the different attribute types/implications:
> - perhaps a hierarchy of meta-ism - or user-relevance
> - those things in the __dict__
> - those things not in the __dict__ but without the "__"
> - ???

I don't discourage you from learning for the sake of learning, but how does
this get you closer to solving your actual problem?


>>> Obviously there is a syntax difference between x.attr and x['key']
>> 
>> Not merely syntax; the attributes of an object are not generally
>> available as items of the container.

I've lost track of who you are quoting here. I think Ben?


> What are the set of ways that an attribute is accessible?  Including
> implementation implications?

You can write:

x.attr

which is equivalent to calling the function:

getattr(x, "attr")


But really, the only limit is what you program x to do. Python gives you the
tools to make (say):

    x.what_is_my_name()[-1] + None

return x.attr, although why would you want to?

So let's put aside all the infinite number of weird and wacky ways that you
could, with sufficient work, access attributes, and consider only the
natural ways to do so. There are two:

x.attr
getattr(x, "attr")


>>> Either the instance __dict__, the class __dict__, or a superclass
>>> __dict__.
>> 
>> No, I'm not referring to the ‘__dict__’ attribute of an object; I'm
>> referring to the object itself.
>> 
>> To talk about the attributes of an object ‘foo’ is distinct from talking
>> about the items in a dictionary ‘foo’. That distinction is real, and
>> important.
> 
> 
> But wanting to deal with the attributes of an object without considering
> the way it's implemented - although possible - requires a complete virtual
> model that covers all implications.  It's easier to simply understand how
> the objects work under the covers.

You should understand how attributes are implemented, but it is not
necessary to understand it in all the gory detail just to use them. Now I'm
feeling some sympathy for the people on StackOverflow who told you that you
don't need to understand this stuff -- perhaps I was too harsh on them :-)

There's a lot of complicated implementation detail involved in attribute
access, but for 99% of uses that's all handled for you and all you need do
is write x.attr.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#101048

On Thu, 31 Dec 2015 10:50:53 +1100, Steven D'Aprano wrote:

> I'm not sure what distinction you're referring to, can you explain?

Ian Kelly had said:

>> How precisely are you trying to store these: as an attribute, or as a
>> dict item? If it's supposed to be in the dict, then why is your
>> __getitem__ trying to look up an attribute to begin with?

to which I answered:

>> In any case, I thought that class attributes were, in fact, items of
>>__dict__?

> 
> Obviously there is a syntax difference between x.attr and x['key'], but
> attributes *are* items in a dictionary (ignoring __slots__ and
> __getattr__ for the time being). Either the instance __dict__, the class
> __dict__, or a superclass __dict__.


That was my understanding but I wasn't sure what Ian meant when he went on
to say:

>> If it's supposed to be in the dict, then why is your __getitem__
>> trying to look up an attribute to begin with?

Which raises the question of where they would be if not in a dictionary.

This brings up a fundamental unclarity of mine: an object has attributes, one
of which is __dict__, which has attributes.

- how does one access the attributes in e.g. self
- where do I find the attribute 'mcc' by cruising around in pdb, given
  the objs below?


(PDB)pp dir (self)
['__class__',
 '__cmp__',
 ...
 '__dict__',
  ...
 '__weakref__',
 'clear',
 'copy',
 'fromkeys',
 'get',
 'has_key',
 'items',
 'iteritems',
 'iterkeys',
 'itervalues',
 'keys',
 'pop',
 'popitem',
 'setdefault',
 'update',
 'values']

(PDB)pp dir (self.__dict__)
['__class__',
 '__cmp__',
 ...
 '__delitem__',
 '__doc__',
 ...
 '__str__',
 '__subclasshook__',
 'clear',
 'copy',
 'fromkeys',
 'get',
 'has_key',
 'items',
 'iteritems',
 'iterkeys',
 'itervalues',
 'keys',
 'pop',
 'popitem',
 'setdefault',
 'update',
 'values']

(PDB)pp (self.keys())
['mcc']


It was recommended that I use the obj[name] syntax in __getattr__()
but that then invoked __getitem__(), complicating the matter.


To be specific, if an attribute is not available, I want to assume
it's a graph node and create it.  If the attribute has an index,
I want to create and access an array of those nodes.

It seems to fall perfectly within the definition of __getattr__()
and __getitem__(), but I suspect that slight programming errors
(i.e. mine) are hiding the proper functionality of these methods.


Thanks everybody, for your help so far, and hopefully you can me
pointed in the right direction.

Happy New Year!
cts

[toc] | [prev] | [next] | [standalone]

#101056

Hmmm, you seem to be pasting in text from multiple messages, and jumping
around in time ("Ian had said, to which I answered") which may get a bit
confusing. Hopefully I can answer without getting lost :-)


On Thu, 31 Dec 2015 10:17 pm, Charles T. Smith wrote:

> On Thu, 31 Dec 2015 10:50:53 +1100, Steven D'Aprano wrote:
> 
>> I'm not sure what distinction you're referring to, can you explain?

I'm pretty sure I was directing that question to Ben, who has explained his
position.


> Ian Kelly had said:
> 
>>> How precisely are you trying to store these: as an attribute, or as a
>>> dict item? If it's supposed to be in the dict, then why is your
>>> __getitem__ trying to look up an attribute to begin with?
> 
> to which I answered:
> 
>>> In any case, I thought that class attributes were, in fact, items of
>>>__dict__?

Ah, but *which* __dict__? There could be many.

Let's put aside some of the technical details which add complexity, and
consider just a simple case: we have an instance "x" of some class X, and
we write "x.spam" to look up the attribute "spam". What happens?

This is equivalent to the function call getattr(x, "spam"), which looks for
an instance attribute, then a class attribute, then in each of the
superclasses (if any). What does getattr do?

We can imagine that it looks something like this simplified version:

def getattr(obj, name):
    try:
        return obj.__dict__[name]
    except KeyError:
        for cls in type(obj).__mro__:
            try:
                return cls.__dict__[name]
            except KeyError:
                pass
        # Still here?
        raise AttributeError('not found')


I've skipped a lot of detail -- calling of __getattribute__ and __getattr__
methods, the possibility of __slots__, the possibility that the instance
doesn't have a __dict__, descriptors, the influence of the metaclass,
differences between classic and new-style classes -- but the above shows
the essentials: an attribute lookup may look in multiple dicts to find the
attribute.

The astute reader will notice that inside getattr() I'm doing an attribute
lookup "obj.__dict__". How does that not trigger an endless series of
recursive calls to getattr?

The answer is that a handful of special attributes, including __dict__, are
built into the structure of the object itself, not part of the __dict__,
and so the Python interpreter can find them without looking in the dict.
Don't worry about it -- it's not relevant except in the senses:

(1) the problem of endless recursive calls to getattr() is solved; and
(2) objects can have attributes which aren't actually stored in a dict, 
    such as __dict__ itself, and __slots__, and probably others.

What those attributes are is, I think, an implementation detail and
irrelevant. So long as you use the existing mechanisms for doing lookups:

x.spam
getattr(x, "spam")

it will just work.


I wrote to Ben:
>> Obviously there is a syntax difference between x.attr and x['key'], but
>> attributes *are* items in a dictionary (ignoring __slots__ and
>> __getattr__ for the time being). Either the instance __dict__, the class
>> __dict__, or a superclass __dict__.
> 
> 
> That was my understanding but I wasn't sure what Ian meant when he went on
> to say:
> 
>>> If it's supposed to be in the dict, then why is your __getitem__
>>> trying to look up an attribute to begin with?
> 
> Which raises the question of where they would be if not in a dictionary.

They could be created on the fly by __getattr__ or __getattribute__, they
could be in a __slot__, or they could be a special attribute built into the
object structure. But I don't think that is relevant to Ian's question.


> This brings up a fundamental unclarity of mine: an object has attributes,
> one of which is __dict__, which has attributes.
> 
> - how does one access the attributes in e.g. self

self.attribute_name
getattr(self, "attribute_name")


> - where do I find the attribute 'mcc' by cruising around in pdb, given
>   the objs below?

What is the object "self" below? 

What makes you think that 'mcc' is an attribute? The evidence suggests that
it is not. You write:

dir(self)

and the result does *not* include 'mcc', which is very strong evidence
that 'mcc' is not an attribute.



> (PDB)pp dir (self)
[...]
>  'clear',
>  'copy',
>  'fromkeys',
>  'get',
>  'has_key',

etc. This suggests that "self" is a dict, or specifically a subclass of
dict. So you would have:

class MyDict(dict):
    ...


x = MyDict()
x.some_method()

and then dropped into the debugger. Now you're looking at the instance x
from inside one of the methods, where it is known as "self". Being a
(subclass of a) dict, it has all the usual dict attributes, like methods
clear, copy, etc., plus whatever extra attributes you give it. There is no
evidence that you have given it any extra attributes.


> (PDB)pp dir (self.__dict__)
[...]
>  'clear',
>  'copy',
>  'fromkeys',
>  'get',
>  'has_key',
etc.

Being an instance, x will usually have an instance dict. (There are
exceptions, but this is not one of those cases.) You're looking at the
attributes of that dict. Being a dict, it has all the usual dict
attributes, like methods clear, copy, etc. 


> (PDB)pp (self.keys())
> ['mcc']

self is a dict, and dicts have keys:values. This particular dict has a
single key, 'mcc'. That's not an attribute. What makes you think it is an
attribute?


> It was recommended that I use the obj[name] syntax in __getattr__()
> but that then invoked __getitem__(), complicating the matter.

It was recommended by whom? For what purpose? There's something which either
you haven't told us, or I've missed. What are you trying to accomplish?


> To be specific, if an attribute is not available, I want to assume
> it's a graph node and create it.  If the attribute has an index,
> I want to create and access an array of those nodes.

What's a graph node? Create it where? What do you mean by "attribute has an
index"? Where are you creating "an array of those nodes"?

What does any of this business got to do with self.__dict__?


> It seems to fall perfectly within the definition of __getattr__()
> and __getitem__(), but I suspect that slight programming errors
> (i.e. mine) are hiding the proper functionality of these methods.

I think you have gotten lost in the details and are over-complicating
matters greatly. Can you start with a HIGH LEVEL overview of what this
class is supposed to represent? It seems to be a subclass of dict. Why?




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#101031

On Wed, Dec 30, 2015 at 3:54 PM, Charles T. Smith
<cts.private.yahoo@gmail.com> wrote:
> On Wed, 30 Dec 2015 13:40:44 -0700, Ian Kelly wrote:
>
>> On Wed, Dec 30, 2015 at 9:58 AM, Charles T. Smith
>>> The problem is that then triggers the __getitem__() method and I don't
>>> know how to get to the attributes without triggering __getattr__().
>>>
>>> It's the interplay of the two that's killing me.
>>
>> The only interplay of the two is what you have written into your class.
>>
>>> In the example, if I have:
>>>
>>>   self.mcc = self.attrs.mcc
>>>
>>>
>>> The crux:
>>>
>>> Then __getattr__() triggers for the mcc.  If I try to use
>>> self.attrs['mcc'] to get it, then that triggers __getitem__().  Okay,
>>> if the key is not an int, I'll go and get it and return it...
>>> unfortunately that triggers __getattr__(), an infinite loop.
>>
>> How precisely are you trying to store these: as an attribute, or as a
>> dict item? If it's supposed to be in the dict, then why is your
>> __getitem__ trying to look up an attribute to begin with?
>
>
>
> I don't understand this distinction between an "attribute" and a "dict item".
> attrdict is a stupid legacy class that inherits from a dictionary.  I think
> the intent was to be faster than a normal class, but I doubt there's any value
> to that.
>
> In any case, I thought that class attributes were, in fact, items of __dict__?

That's correct, but as I said in my previous message, self.attrs and
self.attrs.__dict__ are two different dicts, and you're confusing one
for the other. Maybe this will be illuminating:

>>> class mydict(dict): pass
...
>>> md = mydict()
>>> md['foo'] = 42  # Set an item in md
>>> md['foo']  # 'foo' exists as an item
42
>>> md.foo  # but not as an attribute
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'mydict' object has no attribute 'foo'
>>> md.__dict__['foo']  # and it's not in md.__dict__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'foo'
>>> md.bar = 43  # Set an attribute on md
>>> md.bar  # 'bar' exists as an attribute
43
>>> md.__dict__['bar']  # and it's in md.__dict__
43
>>> md['bar']  # but it's not in md itself
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'bar'

And to hopefully drive the point home:

>>> md.items()
[('foo', 42)]
>>> md.__dict__.items()
[('bar', 43)]



> The reason that my __getitem__() is trying to look up an attribute is, I
> think, because this syntax triggers __getitem__ with a key of "mcc":
>
>   return self[name]
>
> Is that assumption wrong?

That assumption is correct, but this still doesn't explain to me why
you want __getitem__ to be looking up attributes, i.e. looking in
self.__dict__ when the expectation is that it would just look in self.

Is the goal here that self.foo and self['foo'] are the same thing? If
so, then you shouldn't need to worry about __getitem__ and __setitem__
at all. Just override your __getattr__ and __setattr__ to store
attributes in self instead of self.__dict__.

> Just to establish the concept, this horrible thing is showing some promise:
>
>   attriter = self.iteritems()
>   for attr in attriter:
>       if attr[0] == name:
>           return attr[1]

This is equivalent to but slower than "return self.get(name)". What
method is this in and what's the rest of the code? It's hard to
analyze your code in any helpful way when you keep changing it.

> (because the subscripts there are on a tuple type)

I don't know what this has to do with anything.

> But I concede I must be doing something fundamentally wrong because this
> assert is triggering:
>     def __getattr__ (self, name):
>         print "attrdict:av:__getattr__: entered for ", name
>         assert name not in self.keys(), "attrdict:__getattr__: who lied?"

When does this trigger? What's calling it? What name is passed in?

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web