Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.python > #44089
| References | <sridnffI6YhxuOjMnZ2dnUVZ7tSdnZ2d@brightview.co.uk> <5175377f$0$29977$c3e8da3$5496439d@news.astraweb.com> <517545F7.5090209@nowhere.org> |
|---|---|
| From | Oscar Benjamin <oscar.j.benjamin@gmail.com> |
| Date | 2013-04-22 16:14 +0100 |
| Subject | Re: List Count |
| Newsgroups | comp.lang.python |
| Message-ID | <mailman.927.1366644147.3114.python-list@python.org> (permalink) |
On 22 April 2013 15:15, Blind Anagram <blindanagram@nowhere.org> wrote: > On 22/04/2013 14:13, Steven D'Aprano wrote: >> On Mon, 22 Apr 2013 12:58:20 +0100, Blind Anagram wrote: >> >>> I would be grateful for any advice people can offer on the fastest way >>> to count items in a sub-sequence of a large list. >>> >>> I have a list of boolean values that can contain many hundreds of >>> millions of elements for which I want to count the number of True values >>> in a sub-sequence, one from the start up to some value (say hi). >>> >>> I am currently using: >>> >>> sieve[:hi].count(True) >>> >>> but I believe this may be costly because it copies a possibly large part >>> of the sieve. [snip] > > But when using a sub-sequence, I do suffer a significant reduction in > speed for a count when compared with count on the full list. When the > list is small enough not to cause memory allocation issues this is about > 30% on 100,000,000 items. But when the list is 1,000,000,000 items, OS > memory allocation becomes an issue and the cost on my system rises to > over 600%. Have you tried using numpy? I find that it reduces the memory required to store a list of bools by a factor of 4 on my 32 bit system. I would expect that to be a factor of 8 on a 64 bit system: >>> import sys >>> a = [True] * 1000000 >>> sys.getsizeof(a) 4000036 >>> import numpy >>> a = numpy.ndarray(1000000, bool) >>> sys.getsizeof(a) # This does not include the data buffer 40 >>> a.nbytes 1000000 The numpy array also has the advantage that slicing does not actually copy the data (as has already been mentioned). On this system slicing a numpy array has a 40 byte overhead regardless of the size of the slice. > I agree that this is not a big issue but it seems to me a high price to > pay for the lack of a sieve.count(value, limit), which I feel is a > useful function (given that memoryview operations are not available for > lists). It would be very easy to subclass list and add this functionality in cython if you decide that you do need a builtin method. Oscar
Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-22 12:58 +0100
Re: List Count Dave Angel <davea@davea.name> - 2013-04-22 08:51 -0400
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-22 14:03 +0100
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-22 14:03 +0100
Re: List Count Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-04-22 13:13 +0000
Re: List Count Skip Montanaro <skip@pobox.com> - 2013-04-22 08:57 -0500
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-22 15:15 +0100
Re: List Count Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-04-22 16:14 +0100
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-22 16:50 +0100
Re: List Count Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-04-22 17:06 +0100
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-22 17:38 +0100
Re: List Count Skip Montanaro <skip@pobox.com> - 2013-04-22 12:48 -0500
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-22 20:22 +0100
Re: List Count Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-04-22 21:18 +0100
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-22 22:25 +0100
Re: List Count Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-04-23 00:06 +0100
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-23 07:45 +0100
Re: List Count Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-04-22 23:28 +0000
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-23 08:00 +0100
Re: List Count Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-04-22 22:03 +0100
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-22 22:32 +0100
Re: List Count Dave Angel <davea@davea.name> - 2013-04-22 21:47 -0400
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-23 08:02 +0100
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-22 17:38 +0100
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-22 16:50 +0100
Re: List Count Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-04-22 23:01 +0000
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-23 08:05 +0100
Re: List Count Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-04-23 12:08 +0100
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-23 12:45 +0100
Re: List Count Terry Jan Reedy <tjreedy@udel.edu> - 2013-04-23 15:01 -0400
Re: List Count Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-04-23 14:49 +0000
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-23 17:57 +0100
Re: List Count Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-04-23 18:45 +0100
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-23 19:30 +0100
Re: List Count Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-04-23 20:16 +0100
Re: List Count Terry Jan Reedy <tjreedy@udel.edu> - 2013-04-23 16:00 -0400
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-23 21:41 +0100
Re: List Count Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-04-23 21:38 +0100
Re: List Count Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-04-24 01:59 +0000
Re: List Count Blind Anagram <blindanagram@nowhere.org> - 2013-04-24 10:01 +0100
Re: List Count Peter Otten <__peter__@web.de> - 2013-04-22 15:22 +0200
Re: List Count 88888 Dihedral <dihedral88888@googlemail.com> - 2013-04-22 06:36 -0700
csiph-web