Path: csiph.com!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Newsgroups: comp.lang.python
Date: Thu, 23 Aug 2012 09:49:41 -0700 (PDT)
In-Reply-To: <mailman.3480.1345343315.4697.python-list@python.org>
Complaints-To: groups-abuse@google.com
Injection-Info: glegroupsg2000goo.googlegroups.com; posting-host=67.184.78.189; posting-account=MQ3pigoAAACeFzUFjVAePnOjOJMNlvq9
References: <b8dd3aca-2a87-4124-ad6e-66a8720af99a@googlegroups.com> <mailman.3389.1345146609.4697.python-list@python.org> <7xy5le7cli.fsf@ruckus.brouhaha.com> <mailman.3404.1345158704.4697.python-list@python.org> <502dab6c$0$29978$c3e8da3$5496439d@news.astraweb.com> <fe95c29c-2289-4e9c-870e-e3c475f13459@googlegroups.com> <mailman.3435.1345240665.4697.python-list@python.org> <d4708687-2925-421a-b755-969d6dac731a@googlegroups.com> <mailman.3476.1345328046.4697.python-list@python.org> <mailman.3480.1345343315.4697.python-list@python.org>
User-Agent: G2/1.0
MIME-Version: 1.0
Subject: Re: set and dict iteration
From: Aaron Brady <castironpi@gmail.com>
To: comp.lang.python@googlegroups.com
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
Cc: python-list@python.org
Precedence: list
Message-ID: <mailman.3721.1345740590.4697.python-list@python.org>
Lines: 107
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:27745

On Saturday, August 18, 2012 9:28:32 PM UTC-5, Aaron Brady wrote:
> On Saturday, August 18, 2012 5:14:05 PM UTC-5, MRAB wrote:
>=20
> > On 18/08/2012 21:29, Aaron Brady wrote:
>=20
> > > On Friday, August 17, 2012 4:57:41 PM UTC-5, Chris Angelico wrote:
>=20
> > >> On Sat, Aug 18, 2012 at 4:37 AM, Aaron Brady <castironpi@gmail.com> =
wrote:
>=20
> > >> > Is there a problem with hacking on the Beta?
>=20
> > >> Nope. Hack on the beta, then when the release arrives, rebase your
>=20
> > >> work onto it. I doubt that anything of this nature will be changed
>=20
> > >> between now and then.
>=20
> > >> ChrisA
>=20
> > > Thanks Chris, your post was encouraging.
>=20
> > > I have a question about involving the 'tp_clear' field of the types.
>=20
> > > http://docs.python.org/dev/c-api/typeobj.html#PyTypeObject.tp_clear
>=20
> > > '''
>=20
> > > ...The tuple type does not implement a tp_clear function, because it=
=92s possible to prove that no reference cycle can be composed entirely of =
tuples.
>=20
> > > '''
>=20
> > > I didn't follow the reasoning in the proof; the premise is necessary =
but IMHO not obviously sufficient.  Nevertheless, the earlier diagram conta=
ins an overt homogeneous reference cycle.
>=20
> > > Reposting: http://home.comcast.net/~castironpi-misc/clpy-0062%20set%2=
0iterators.png
>=20
> > > In my estimate, the 'tp_traverse' and 'tp_clear' fields of the set do=
esn't need to visit the auxiliary collection; the same fields of the iterat=
ors don't need to visit the primary set or other iterators; and references =
in the linked list don't need to be included in the iterators' reference co=
unts.
>=20
> > > Can someone who is more familiar with the cycle detector and cycle br=
eaker, help prove or disprove the above?
>=20
> > In simple terms, when you create an immutable object it can contain
>=20
> > only references to pre-existing objects, but in order to create a cycle
>=20
> > you need to make an object refer to another which is created later, so
>=20
> > it's not possible to create a cycle out of immutable objects.
>=20
> > However, using Python's C API it _is_ possible to create such a cycle,
>=20
> > by mutating an otherwise-immutable tuple (see PyTuple_SetItem and
>=20
> > PyTuple_SET_ITEM).
>=20
> Are there any precedents for storing uncounted references to PyObject's?
>=20
> One apparent problematic case is creating an iterator to a set, then addi=
ng it to the set.  However the operation is a modification, and causes the =
iterator to be removed from the secondary list before the set is examined f=
or collection.
>=20
> Otherwise, the iterator keeps a counted reference to the set, but the set=
 does not keep a counted reference to the iterator, so the iterator will al=
ways be freed first.  Therefore, the set's secondary list will be empty whe=
n the set is freed.
>=20
> Concurrent addition and deletion of iterators should be disabled, and the=
 iterators should remove themselves from the set's secondary list before th=
ey decrement their references to the set.
>=20
> Please refresh the earlier diagram; counted references are distinguished =
separately.  Reposting: http://home.comcast.net/~castironpi-misc/clpy-0062%=
20set%20iterators.png

The patch for the above is only 40-60 lines.  However it introduces two new=
 concepts.

The first is a "linked list", a classic dynamic data structure, first devel=
oped in 1955, cf. http://en.wikipedia.org/wiki/Linked_list .  Linked lists =
are absent in Python, including the standard library and CPython implementa=
tion, beyond the weak reference mechanism and garbage collector.  The "coll=
ections.deque" structure shares some of the linked list interface but uses =
arrays.

The second is "uncounted references".  The uncounted references are referen=
ces to "set iterators" exclusively, exist only internally to "set" objects,=
 and are invisible to the rest of the program.  The reason for the exceptio=
n is that iterators are unique in the Python Data Model; iterators consist =
of a single immutable reference, unlike both immutable types such as string=
s and numbers, as well as container types.  Counted references could be use=
d instead, but would be consistently wasted work for the garbage collector,=
 though the benefit to programmers' peace of mind could be significant.

Please share your opinion!  Do you agree that the internal list resolves th=
e inconsistency?  Do you agree with the strategy?  Do you agree that uncoun=
ted references are justified to introduce, or are counted references prefer=
able?