Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #40715

Re: Interesting list() un-optimization

Show all headers | View raw

Roy Smith wrote:

> I stumbled upon an interesting bit of trivia concerning lists and list
> comprehensions today.
> 
> We use mongoengine as a database model layer.  A mongoengine query
> returns an iterable object called a QuerySet.  The "obvious" way to
> create a list of the query results would be:
> 
>     my_objects = list(my_query_set)
> 
> and, indeed, that works.  But, then I found this code:
> 
>    my_objects = [obj for obj in my_query_set]
> 
> which seemed a bit silly.  I called over the guy who wrote it and asked
> him why he didn't just write it using list().  I was astounded when it
> turned out there's a good reason!
> 
> Apparently, list() has an "optimization" where it calls len() on its
> argument to try and discover the number of items it's going to put into
> the list.  Presumably, list() uses this information to pre-allocate the
> right amount of memory the first time, without any resizing.  If len()
> fails, it falls back to just iterating and resizing as needed.
> Normally, this would be a win.
> 
> The problem is, QuerySets have a __len__() method.  Calling it is a lot
> faster than iterating over the whole query set and counting the items,
> but it does result in an additional database query, which is a lot
> slower than the list resizing!  Writing the code as a list comprehension
> prevents list() from trying to optimize when it shouldn't!


Interesting discovery.  Yet isn't this as much an issue with the mongoengine 
library as with list()?  Queryset.count() can be called if the "length" of a 
resultset needs to be retrieved, so the __len__() methd seems redundant.  
And given that it's not unheard of to call list() on iterables, perhaps the 
library designers should either optimise the __len__() method, or document 
the performance implications of calling list on the queryset? 

Anyway, thanks for this thought-provoking post.

Cheers,

Kev 

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Interesting list() un-optimization Roy Smith <roy@panix.com> - 2013-03-06 22:20 -0500
  Re: Interesting list() un-optimization Dave Angel <davea@davea.name> - 2013-03-06 22:38 -0500
  Re: Interesting list() un-optimization Tim Chase <python.list@tim.thechases.com> - 2013-03-06 21:57 -0600
  Re: Interesting list() un-optimization Kev Dwyer <kevin.p.dwyer@gmail.com> - 2013-03-07 07:31 +0000
  Re: Interesting list() un-optimization Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> - 2013-03-07 11:22 +0000
  Re: Interesting list() un-optimization Ian Kelly <ian.g.kelly@gmail.com> - 2013-03-07 09:00 -0700
  Re: Interesting list() un-optimization Christian Heimes <christian@python.org> - 2013-03-07 17:20 +0100
  Re: Interesting list() un-optimization Ian Kelly <ian.g.kelly@gmail.com> - 2013-03-07 10:31 -0700
  Re: Interesting list() un-optimization Stefan Behnel <stefan_ml@behnel.de> - 2013-03-07 20:19 +0100
  Re: Interesting list() un-optimization Ian Kelly <ian.g.kelly@gmail.com> - 2013-03-07 13:26 -0700
  Re: Interesting list() un-optimization Terry Reedy <tjreedy@udel.edu> - 2013-03-07 15:29 -0500
  Re: Interesting list() un-optimization Terry Reedy <tjreedy@udel.edu> - 2013-03-07 15:34 -0500
  Re: Interesting list() un-optimization Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> - 2013-03-07 20:41 +0000
  Re: Interesting list() un-optimization Terry Reedy <tjreedy@udel.edu> - 2013-03-07 17:53 -0500
  Re: Interesting list() un-optimization Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-03-08 17:59 +0000
    Re: Interesting list() un-optimization Roy Smith <roy@panix.com> - 2013-03-08 13:45 -0500
  Re: Interesting list() un-optimization Roy Smith <roy@panix.com> - 2013-03-10 09:05 -0400
    Re: Interesting list() un-optimization Terry Reedy <tjreedy@udel.edu> - 2013-03-10 17:39 -0400
      Re: Interesting list() un-optimization Roy Smith <roy@panix.com> - 2013-03-10 18:34 -0400
        Re: Interesting list() un-optimization Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-03-10 23:35 +0000
          Re: Interesting list() un-optimization Roy Smith <roy@panix.com> - 2013-03-10 19:50 -0400

csiph-web