Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #40641 > unrolled thread

Creating an object that can track when its attributes are modified

Back to article view | Back to comp.lang.python

Contents

  Creating an object that can track when its attributes are modified Ben Sizer <kylotan@gmail.com> - 2013-03-06 08:07 -0800
    Re: Creating an object that can track when its attributes are modified Chris Angelico <rosuav@gmail.com> - 2013-03-07 03:22 +1100
      Re: Creating an object that can track when its attributes are modified Ben Sizer <kylotan@gmail.com> - 2013-03-06 08:56 -0800
        Re: Creating an object that can track when its attributes are modified Chris Angelico <rosuav@gmail.com> - 2013-03-07 04:03 +1100
        Re: Creating an object that can track when its attributes are modified 88888 Dihedral <dihedral88888@googlemail.com> - 2013-03-06 13:07 -0800
        Re: Creating an object that can track when its attributes are modified 88888 Dihedral <dihedral88888@googlemail.com> - 2013-03-06 13:07 -0800
      Re: Creating an object that can track when its attributes are modified Ben Sizer <kylotan@gmail.com> - 2013-03-06 08:56 -0800
        Re: Creating an object that can track when its attributes are modified Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-03-07 00:07 +0000
          Re: Creating an object that can track when its attributes are modified Ben Sizer <kylotan@gmail.com> - 2013-03-06 16:26 -0800
            Re: Creating an object that can track when its attributes are modified Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-03-08 18:50 +0000
    Re: Creating an object that can track when its attributes are modified Lele Gaifax <lele@metapensiero.it> - 2013-03-06 17:56 +0100
    Re: Creating an object that can track when its attributes are modified Schneider <js@globe.de> - 2013-03-07 13:21 +0100

#40641 — Creating an object that can track when its attributes are modified

I am trying to make an object that can track when its attributes have been assigned new values, and which can rollback to previous values where necessary. I have the following code which I believe works, but would like to know if there are simpler ways to achieve this goal, or if there are any bugs I haven't seen yet.


class ChangeTrackingObject(object):
    def __init__(self):
        self.clean()

    def clean(self):
        """Mark all attributes as unmodified."""
        object.__setattr__(self, '_dirty_attributes', dict())

    def dirty_vals(self):
        """Returns all dirty values."""
        return dict( [ (k,v) for k,v in self.__dict__.iteritems() if k in self._dirty_attributes]  )

    def get_changes_and_clean(self):
        """Helper that collects all the changes and returns them, cleaning the dirty flags at the same time."""
        changes = self.dirty_vals()
        self.clean()
        return changes

    def rollback(self):
        """Reset attributes to their previous values."""
        for k,v in self._dirty_attributes.iteritems():
            object.__setattr__(self, k, v)
        self.clean()

    def __setattr__(self, key, value):
        # If the first modification to this attribute, store the old value
        if key not in self._dirty_attributes:
            if key in self.__dict__:
                self._dirty_attributes[key] = object.__getattribute__(self, key)
            else:
                self._dirty_attributes[key] = None
        # Set the new value
        object.__setattr__(self, key, value)


I am aware that adding a new attribute and then calling rollback() leaves the new attribute in place with a None value - maybe I can use a special DeleteMe marker object in the _dirty_attributes dict along with a loop that calls delattr on any attribute that has that value after a rollback.

I also believe that this won't catch modification to existing attributes as opposed to assignments: eg. if one of the attributes is a list and I append to it, this system won't notice. Is that something I can rectify easily?

Any other comments or suggestions?

Thanks,
-- 
Ben Sizer

[toc] | [next] | [standalone]

#40643

On Thu, Mar 7, 2013 at 3:07 AM, Ben Sizer <kylotan@gmail.com> wrote:
> I also believe that this won't catch modification to existing attributes as opposed to assignments: eg. if one of the attributes is a list and I append to it, this system won't notice. Is that something I can rectify easily?

The only way you could detect mutation of one of its attributes is
with that object's assistance. Effectively, you would need to have a
subclass of list/dict/tuple/whatever that can respond to the change.
Alternatively, you could retain a deep copy and do a comparison at
time of rollback; this, however, would have annoying consequences wrt
performance and other references and such.

What's the goal of this class? Can you achieve the same thing by
using, perhaps, a before-and-after snapshot of a JSON-encoded form of
the object?

ChrisA

[toc] | [prev] | [next] | [standalone]

#40645

On Wednesday, 6 March 2013 16:22:56 UTC, Chris Angelico  wrote:
> 
> Effectively, you would need to have a
> subclass of list/dict/tuple/whatever that can respond to the change. 

This is certainly something I'd be interested in having, but I guess that would be fragile since the user would have the burden of having to remember to use those types.

> What's the goal of this class? Can you achieve the same thing by
> using, perhaps, a before-and-after snapshot of a JSON-encoded form of
> the object?
> 

I need to be able to perform complex operations on the object that may modify several properties, and then gather the properties at the end as an efficient way to see what has changed and to store those changes. Any comparison of before-and-after snapshots could work in theory, but in practice it could be expensive to produce the snapshots on larger objects and probably expensive to calculate the differences that way too. Performance is important so I would probably just go for an explicit function call to mark an attribute as having been modified rather than trying to do a diff like that. (It wouldn't work for rollbacks, but I can accept that.)

-- 
Ben Sizer

[toc] | [prev] | [next] | [standalone]

#40651

On Thu, Mar 7, 2013 at 3:56 AM, Ben Sizer <kylotan@gmail.com> wrote:
> On Wednesday, 6 March 2013 16:22:56 UTC, Chris Angelico  wrote:
>>
>> Effectively, you would need to have a
>> subclass of list/dict/tuple/whatever that can respond to the change.
>
> This is certainly something I'd be interested in having, but I guess that would be fragile since the user would have the burden of having to remember to use those types.

Since you're already overriding setattr, you could simply force all
non-string sequences to your special subclass of list. That reduces
that burden, though it'd break if there are any other references to
the object.

>> What's the goal of this class? Can you achieve the same thing by
>> using, perhaps, a before-and-after snapshot of a JSON-encoded form of
>> the object?
>>
>
> I need to be able to perform complex operations on the object that may modify several properties, and then gather the properties at the end as an efficient way to see what has changed and to store those changes. Any comparison of before-and-after snapshots could work in theory, but in practice it could be expensive to produce the snapshots on larger objects and probably expensive to calculate the differences that way too. Performance is important so I would probably just go for an explicit function call to mark an attribute as having been modified rather than trying to do a diff like that. (It wouldn't work for rollbacks, but I can accept that.)

Hmm. Interesting. The perfect solution probably is too messy, yeah.
But if you have your subclassing done, you could possibly
snapshot-on-write, which would allow the rollback. Not sure if it'd
help though.

ChrisA

[toc] | [prev] | [next] | [standalone]

#40663

Ben Sizer於 2013年3月7日星期四UTC+8上午12時56分09秒寫道：
> On Wednesday, 6 March 2013 16:22:56 UTC, Chris Angelico  wrote:
> 
> > 
> 
> > Effectively, you would need to have a
> 
> > subclass of list/dict/tuple/whatever that can respond to the change. 
> 
> 
> 
> This is certainly something I'd be interested in having, but I guess that would be fragile since the user would have the burden of having to remember to use those types.
> 
> 
> 
> > What's the goal of this class? Can you achieve the same thing by
> 
> > using, perhaps, a before-and-after snapshot of a JSON-encoded form of
> 
> > the object?
> 
> > 
> 
> 
> 
> I need to be able to perform complex operations on the object that may modify several properties, and then gather the properties at the end as an efficient way to see what has changed and to store those changes. Any comparison of before-and-after snapshots could work in theory, but in practice it could be expensive to produce the snapshots on larger objects and probably expensive to calculate the differences that way too. Performance is important so I would probably just go for an explicit function call to mark an attribute as having been modified rather than trying to do a diff like that. (It wouldn't work for rollbacks, but I can accept that.)
> 
> 
> 
> -- 
> 
> Ben Sizer
Please hook a stack implemented as a list in python  to every property 
of the object  that you want to track down.

[toc] | [prev] | [next] | [standalone]

#40664

Ben Sizer於 2013年3月7日星期四UTC+8上午12時56分09秒寫道：
> On Wednesday, 6 March 2013 16:22:56 UTC, Chris Angelico  wrote:
> 
> > 
> 
> > Effectively, you would need to have a
> 
> > subclass of list/dict/tuple/whatever that can respond to the change. 
> 
> 
> 
> This is certainly something I'd be interested in having, but I guess that would be fragile since the user would have the burden of having to remember to use those types.
> 
> 
> 
> > What's the goal of this class? Can you achieve the same thing by
> 
> > using, perhaps, a before-and-after snapshot of a JSON-encoded form of
> 
> > the object?
> 
> > 
> 
> 
> 
> I need to be able to perform complex operations on the object that may modify several properties, and then gather the properties at the end as an efficient way to see what has changed and to store those changes. Any comparison of before-and-after snapshots could work in theory, but in practice it could be expensive to produce the snapshots on larger objects and probably expensive to calculate the differences that way too. Performance is important so I would probably just go for an explicit function call to mark an attribute as having been modified rather than trying to do a diff like that. (It wouldn't work for rollbacks, but I can accept that.)
> 
> 
> 
> -- 
> 
> Ben Sizer
Please hook a stack implemented as a list in python  to every property 
of the object  that you want to track down.

[toc] | [prev] | [next] | [standalone]

#40647

On Wednesday, 6 March 2013 16:22:56 UTC, Chris Angelico  wrote:
> 
> Effectively, you would need to have a
> subclass of list/dict/tuple/whatever that can respond to the change. 

This is certainly something I'd be interested in having, but I guess that would be fragile since the user would have the burden of having to remember to use those types.

> What's the goal of this class? Can you achieve the same thing by
> using, perhaps, a before-and-after snapshot of a JSON-encoded form of
> the object?
> 

I need to be able to perform complex operations on the object that may modify several properties, and then gather the properties at the end as an efficient way to see what has changed and to store those changes. Any comparison of before-and-after snapshots could work in theory, but in practice it could be expensive to produce the snapshots on larger objects and probably expensive to calculate the differences that way too. Performance is important so I would probably just go for an explicit function call to mark an attribute as having been modified rather than trying to do a diff like that. (It wouldn't work for rollbacks, but I can accept that.)

-- 
Ben Sizer

[toc] | [prev] | [next] | [standalone]

#40674

On Wed, 06 Mar 2013 08:56:09 -0800, Ben Sizer wrote:

> I need to be able to perform complex operations on the object that may
> modify several properties, and then gather the properties at the end as
> an efficient way to see what has changed and to store those changes. Any
> comparison of before-and-after snapshots could work in theory, but in
> practice it could be expensive to produce the snapshots on larger
> objects and probably expensive to calculate the differences that way
> too. Performance is important so I would probably just go for an
> explicit function call to mark an attribute as having been modified
> rather than trying to do a diff like that. (It wouldn't work for
> rollbacks, but I can accept that.)


Premature optimization.

Unless you have been eating and breathing Python code for 15+ years, your 
intuition of what is expensive and what isn't will probably be *way* off. 
I've been using Python for ~15 years, and I wouldn't want to try to guess 
what the most efficient way to do this will be.

Actually I lie. I would guess that the simple, most obvious way is 
faster: don't worry about storing what changed, just store *everything*. 
But I could be wrong.

Fortunately, Python development is rapid enough that you can afford to 
develop this object the straightforward way, profile your application to 
see where the bottlenecks are, and if it turns out that the simple 
approach is too expensive, then try something more complicated.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#40678

On Thursday, 7 March 2013 00:07:02 UTC, Steven D'Aprano  wrote:
> On Wed, 06 Mar 2013 08:56:09 -0800, Ben Sizer wrote:
> 
> > I need to be able to perform complex operations on the object that may
> > modify several properties, and then gather the properties at the end as
> > an efficient way to see what has changed and to store those changes. Any
> > comparison of before-and-after snapshots could work in theory, but in
> > practice it could be expensive to produce the snapshots on larger
> > objects and probably expensive to calculate the differences that way
> > too. Performance is important so I would probably just go for an
> > explicit function call to mark an attribute as having been modified
> > rather than trying to do a diff like that. (It wouldn't work for
> > rollbacks, but I can accept that.)
> 
> Premature optimization.
> 
> Unless you have been eating and breathing Python code for 15+ years, your 
> intuition of what is expensive and what isn't will probably be *way* off. 
> I've been using Python for ~15 years, and I wouldn't want to try to guess 
> what the most efficient way to do this will be.

I admit, I've only been using Python for 10 years, but I've learned a lot about optimisation and what needs optimising from my time as a game developer. This code needs to be fairly high-performing due to the role it plays in my server and the frequency with which the behaviour gets called.

> Actually I lie. I would guess that the simple, most obvious way is 
> faster: don't worry about storing what changed, just store *everything*. 
> But I could be wrong.

The use case I have is not one where that is suitable. It's not the snapshots that are important, but the changes between them.

> Fortunately, Python development is rapid enough that you can afford to 
> develop this object the straightforward way, profile your application to 
> see where the bottlenecks are, and if it turns out that the simple 
> approach is too expensive, then try something more complicated.

I don't see a more straightforward solution to the problem I have than the one I have posted. I said that a system that took snapshots of the whole object and attempted to diff them would probably perform worse, but it would probably be more complex too, given the traversal and copying requirements. 

-- 
Ben Sizer

[toc] | [prev] | [next] | [standalone]

#40883

On Wed, 06 Mar 2013 16:26:57 -0800, Ben Sizer wrote:

> On Thursday, 7 March 2013 00:07:02 UTC, Steven D'Aprano  wrote:
[...]
>> Actually I lie. I would guess that the simple, most obvious way is
>> faster: don't worry about storing what changed, just store
>> *everything*. But I could be wrong.
> 
> The use case I have is not one where that is suitable. It's not the
> snapshots that are important, but the changes between them.

I'm afraid that doesn't make much sense to me. You're performing 
calculations, and stuffing them into instance attributes, but you don't 
care about the result of the calculations, only how they differ from the 
previous result?

I obviously don't understand the underlying problem you're trying to 
solve.


>> Fortunately, Python development is rapid enough that you can afford to
>> develop this object the straightforward way, profile your application
>> to see where the bottlenecks are, and if it turns out that the simple
>> approach is too expensive, then try something more complicated.
> 
> I don't see a more straightforward solution to the problem I have than
> the one I have posted. I said that a system that took snapshots of the
> whole object and attempted to diff them would probably perform worse,
> but it would probably be more complex too, given the traversal and
> copying requirements.

Yes, and I said that your intuition of what will be fast and what will be 
slow is not necessarily trustworthy. Without testing, neither of us knows 
for sure.

Given the code you showed in the original post, I don't see that 
traversal and copying requirements are terribly complicated. You don't do 
deep-copies of attributes, so a shallow copy of the instance __dict__ 
ought to be enough. Assuming you have a well-defined "start processing" 
moment, just grab a snapshot of the dict, which will be fast, then do 
your calculations, then call get_changes:


    def snapshot(self):
        self._snapshot = self.__dict__.copy()

    def get_changes(self):
        sentinel = object()
        return dict( [ (k,v) for k,v in self.__dict__.iteritems() 
            if k == self._snapshot.get(k, sentinel) ] )


This doesn't support *deleting* attributes, but neither does your 
original version.

Obviously I don't know for sure which strategy is fastest, but since your 
version already walks the entire __dict__, this shouldn't be much slower, 
and has a good chance of being faster.

(Your version slows down *every* attribute assignment. My version does 
not.)


By the way, your original version describes the get_changes_and_clean() 
method as cleaning the dirty *flags*. But the implementation doesn't 
store flags. Misleading documentation is worse than no documentation.


But if you insist on the approach you've taken, you can simplify the 
__setattr__ method:


    def __setattr__(self, key, value):
        # If the first modification to this attribute, store the old value
        dirty = self._dirty_attributes
        if key not in dirty:
            dirty[key] = getattr(self, key, None)
        # Set the new value
        object.__setattr__(self, key, value)


You might try this (slightly) obfuscated version, which *could* be faster 
still, although I doubt it.


    def __setattr__(self, key, value):
        # If the first modification to this attribute, store the old value
        self._dirty_attributes.setdefault(key, getattr(self, key, None))
        # Set the new value
        object.__setattr__(self, key, value)


but if you really need to get every bit of performance, it's worth trying 
them both and seeing which is faster.


(P.S. I trust you know to use timeit for timing small code snippets, 
rather than rolling your own timing code?)



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#40646

Ben Sizer <kylotan@gmail.com> writes:

> I also believe that this won't catch modification to existing
> attributes as opposed to assignments: eg. if one of the attributes is
> a list and I append to it, this system won't notice. Is that something
> I can rectify easily?

It's really up to how far you wanna go: a similar use case is
implemented by SQLAlchemy__, which "instrument" builtin collection
classes to achieve the goal. But I'd not call that "easy" though :-)

ciao, lele.

__ http://www.sqlalchemy.org/trac/browser/lib/sqlalchemy/orm/collections.py
-- 
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
lele@metapensiero.it  |                 -- Fortunato Depero, 1929.

[toc] | [prev] | [next] | [standalone]

#40758

Hi,
maybe you could do this by a decorator on the setattr method. It should 
look more or less
like your implementation, but in my eyes it's a cleaner and can be reused.

Further, I would use a stack for each attribute, so that you can restore 
all previous values.

bg,
Johannes

On 03/06/2013 05:07 PM, Ben Sizer wrote:
> I am trying to make an object that can track when its attributes have been assigned new values, and which can rollback to previous values where necessary. I have the following code which I believe works, but would like to know if there are simpler ways to achieve this goal, or if there are any bugs I haven't seen yet.
>
>
> class ChangeTrackingObject(object):
>      def __init__(self):
>          self.clean()
>
>      def clean(self):
>          """Mark all attributes as unmodified."""
>          object.__setattr__(self, '_dirty_attributes', dict())
>
>      def dirty_vals(self):
>          """Returns all dirty values."""
>          return dict( [ (k,v) for k,v in self.__dict__.iteritems() if k in self._dirty_attributes]  )
>
>      def get_changes_and_clean(self):
>          """Helper that collects all the changes and returns them, cleaning the dirty flags at the same time."""
>          changes = self.dirty_vals()
>          self.clean()
>          return changes
>
>      def rollback(self):
>          """Reset attributes to their previous values."""
>          for k,v in self._dirty_attributes.iteritems():
>              object.__setattr__(self, k, v)
>          self.clean()
>
>      def __setattr__(self, key, value):
>          # If the first modification to this attribute, store the old value
>          if key not in self._dirty_attributes:
>              if key in self.__dict__:
>                  self._dirty_attributes[key] = object.__getattribute__(self, key)
>              else:
>                  self._dirty_attributes[key] = None
>          # Set the new value
>          object.__setattr__(self, key, value)
>
>
> I am aware that adding a new attribute and then calling rollback() leaves the new attribute in place with a None value - maybe I can use a special DeleteMe marker object in the _dirty_attributes dict along with a loop that calls delattr on any attribute that has that value after a rollback.
>
> I also believe that this won't catch modification to existing attributes as opposed to assignments: eg. if one of the attributes is a list and I append to it, this system won't notice. Is that something I can rectify easily?
>
> Any other comments or suggestions?
>
> Thanks,

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web