Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #79716 > unrolled thread

Lazy-evaluation lists/dictionaries

Back to article view | Back to comp.lang.python

Contents

  Lazy-evaluation lists/dictionaries Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2014-10-26 14:14 +0000
    Re: Lazy-evaluation lists/dictionaries Tim Delaney <timothy.c.delaney@gmail.com> - 2014-10-27 07:36 +1100
      Re: Lazy-evaluation lists/dictionaries Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2014-10-27 15:11 +0000
    Re: Lazy-evaluation lists/dictionaries Terry Reedy <tjreedy@udel.edu> - 2014-10-26 18:41 -0400
      Re: Lazy-evaluation lists/dictionaries Jon Ribbens <jon+usenet@unequivocal.co.uk> - 2014-10-27 15:03 +0000

#79716 — Lazy-evaluation lists/dictionaries

I have a need, in a Python C extension I am writing, for lists and
dictionaries with "lazy evaluation" - by which I mean that at least
some of the values in the lists/dictionaries are "proxy objects"
which, rather than returning as themselves, should return the thing
they are a proxy for when retrieved. This is because retrieving
the proxied objects is expensive and only a small minority of them
will actually be accessed, so retrieving them all before they are
actually accessed is massively inefficient.

With object attributes, this can be easily done with the descriptor
protocol and having properties which look like simple objects but
actually cause a method to be invoked when they are accessed. However
there doesn't seem to be any equivalent way of doing this for
retrieving items from lists or dictionaries - unfortunately, the
method implementations of list() and dict() are full of direct
accesses to the underlying data pointers rather than indirecting
through obj->tp_as_mapping->mp_subscript or whatever.

Is there any better way to do this other than simply re-implementing
these types from scratch, emulating all their methods and operations?
(i.e. using UserList/UserDict). I was under the impression that that
sort of thing was supposed to have gone since Python 2.2 or so.

[toc] | [next] | [standalone]

#79725

[Multipart message — attachments visible in raw view] — view raw

On 27 October 2014 01:14, Jon Ribbens <jon+usenet@unequivocal.co.uk> wrote:

> I have a need, in a Python C extension I am writing, for lists and
> dictionaries with "lazy evaluation" - by which I mean that at least
> some of the values in the lists/dictionaries are "proxy objects"
> which, rather than returning as themselves, should return the thing
> they are a proxy for when retrieved. This is because retrieving
> the proxied objects is expensive and only a small minority of them
> will actually be accessed, so retrieving them all before they are
> actually accessed is massively inefficient.
>

Why not put proxy objects into the list/dict? Have a look at the weakref
module for an API that may be suitable for such proxy objects (if you used
the same API, that would also allow you to transparently use weakrefs in
your lists/dicts).

Tim Delaney

[toc] | [prev] | [next] | [standalone]

#79789

On 2014-10-26, Tim Delaney <timothy.c.delaney@gmail.com> wrote:
> On 27 October 2014 01:14, Jon Ribbens <jon+usenet@unequivocal.co.uk> wrote:
>> I have a need, in a Python C extension I am writing, for lists and
>> dictionaries with "lazy evaluation" - by which I mean that at least
>> some of the values in the lists/dictionaries are "proxy objects"
>> which, rather than returning as themselves, should return the thing
>> they are a proxy for when retrieved. This is because retrieving
>> the proxied objects is expensive and only a small minority of them
>> will actually be accessed, so retrieving them all before they are
>> actually accessed is massively inefficient.
>
> Why not put proxy objects into the list/dict?

That's precisely what I am doing. The point is that when they are
retrieved they need to be resolved into the genuine objects.

> Have a look at the weakref module for an API that may be suitable
> for such proxy objects (if you used the same API, that would also
> allow you to transparently use weakrefs in your lists/dicts).

Hmm, the idea behind that appears to be to create a proxy that
emulates every possible method of every conceivable type. My
method of only emulating the list and dict methods seems to be
somewhat simpler for my purpose ;-)

[toc] | [prev] | [next] | [standalone]

#79732

On 10/26/2014 10:14 AM, Jon Ribbens wrote:
> I have a need, in a Python C extension I am writing, for lists and
> dictionaries with "lazy evaluation" - by which I mean that at least
> some of the values in the lists/dictionaries are "proxy objects"
> which, rather than returning as themselves, should return the thing
> they are a proxy for when retrieved. This is because retrieving
> the proxied objects is expensive and only a small minority of them
> will actually be accessed, so retrieving them all before they are
> actually accessed is massively inefficient.
>
> With object attributes, this can be easily done with the descriptor
> protocol and having properties which look like simple objects but
> actually cause a method to be invoked when they are accessed. However
> there doesn't seem to be any equivalent way of doing this for
> retrieving items from lists or dictionaries - unfortunately, the
> method implementations of list() and dict() are full of direct
> accesses to the underlying data pointers rather than indirecting
> through obj->tp_as_mapping->mp_subscript or whatever.
>
> Is there any better way to do this other than simply re-implementing
> these types from scratch, emulating all their methods and operations?
> (i.e. using UserList/UserDict). I was under the impression that that
> sort of thing was supposed to have gone since Python 2.2 or so.

We considered dropping UserDict and UserList for 3.0 but kept them in 
collections for cases in which subclassing does not work.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#79787

On 2014-10-26, Terry Reedy <tjreedy@udel.edu> wrote:
> On 10/26/2014 10:14 AM, Jon Ribbens wrote:
>> Is there any better way to do this other than simply re-implementing
>> these types from scratch, emulating all their methods and operations?
>> (i.e. using UserList/UserDict). I was under the impression that that
>> sort of thing was supposed to have gone since Python 2.2 or so.
>
> We considered dropping UserDict and UserList for 3.0 but kept them in 
> collections for cases in which subclassing does not work.

It seems on further investigation to be hard/impossible to subclass
Python classes from C extensions. I've gone with subclassing dict,
and reimplementing most of its methods. It helps that I only need it
to be read-only. It's a pity there's no protocol for 'dynamic'
lists/dicts though.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web