Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Date: Mon, 03 Sep 2012 21:50:57 -0400
From: Dave Angel <d@davea.name>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20120714 Thunderbird/14.0
MIME-Version: 1.0
To: Steven D'Aprano <steve+comp.lang.python@pearwood.info>
Subject: Re: set and dict iteration
References: <b8dd3aca-2a87-4124-ad6e-66a8720af99a@googlegroups.com> <mailman.3389.1345146609.4697.python-list@python.org> <7xy5le7cli.fsf@ruckus.brouhaha.com> <mailman.3404.1345158704.4697.python-list@python.org> <502dab6c$0$29978$c3e8da3$5496439d@news.astraweb.com> <fe95c29c-2289-4e9c-870e-e3c475f13459@googlegroups.com> <mailman.3435.1345240665.4697.python-list@python.org> <d4708687-2925-421a-b755-969d6dac731a@googlegroups.com> <mailman.3476.1345328046.4697.python-list@python.org> <mailman.3480.1345343315.4697.python-list@python.org> <c7452db1-5b78-4d54-81a1-1c8683631d6e@googlegroups.com> <mailman.3883.1346095064.4697.python-list@python.org> <1567e8c7-a2bb-41f4-9be8-18e9f4d063cb@googlegroups.com> <mailman.154.1346700607.27098.python-list@python.org> <mailman.155.1346702665.27098.python-list@python.org> <504558cb$0$29978$c3e8da3$5496439d@news.astraweb.com>
In-Reply-To: <504558cb$0$29978$c3e8da3$5496439d@news.astraweb.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Cc: python-list@python.org
Precedence: list
Reply-To: d@davea.name
Newsgroups: comp.lang.python
Message-ID: <mailman.162.1346723805.27098.python-list@python.org>
Lines: 61
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:28381

On 09/03/2012 09:26 PM, Steven D'Aprano wrote:
> On Mon, 03 Sep 2012 13:04:23 -0700, Aaron Brady wrote:
>
>> <snip>
>>
>> I think the technique would require two python longs and a bool in the
>> set, and a python long in the iterator.
>>
>> One long counts the number of existing (open) iterators.  Another counts
>> the version.  The bool keeps track of whether an iterator has been
>> created since the last modification, in which case the next modification
>> requires incrementing the version and resetting the flag.
> I think that is over-engineered and could be the difference between 
> having the patch accepted and having it rejected. After all, many people 
> will argue that the existing solution to the problem is good enough.
>
> Dicts are extremely common, and your patch increases both the memory 
> usage of every dict, and the overhead of every write operation 
> (__setitem__, __delitem__, update). Only a very few dicts will actually 
> need this overhead, for the rest it is waste. It is important to keep 
> that waste to a minimum or risk having the patch rejected.
>
> An unsigned C int can count up to 4,294,967,295. I propose that you say 
> that is enough iterators for anyone, and use a single, simple, version 
> counter in the dict and the iterator. If somebody exceeds that many 
> iterators to a single dict or set, 

I think you have the count confused.  it has to be a count of how many
changes have been made to the dict or set, not how many iterators exist.

> and the version field overflows by 
> exactly 2**32 versions, the results are no worse than they are now. You 
> won't be introducing any new bugs.
>
> Complicating the solution is, in my opinion, unnecessary. Why should 
> every set and dict carry the cost of incrementing TWO Python longs and a 
> flag when just a single C int is sufficient for all realistic use-cases?
>
> The opportunity for failure is extremely narrow:
>
> - I must have an iterator over a dict or set;
> - and I must have paused the iteration in some way;
> - and then I must create exactly 2**32 other iterators to the 
>   same dict or set;
> - and at some point modify the dict or set
> - and then restart the first iterator
>
> at which point some items returned by the iterator *may* be duplicated or 
> skipped (depends on the nature of the modifications).
>
>

I agree with almost your entire point, exact that the place where it
would fail to detect a bug is when somebody has modified the dict
exactly 2**32 times while an iterator is paused.  See my own response,
of 4:27pm. (my time).

-- 

DaveA