Re: Negative array indicies and slice()

Path	csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path	<ian.g.kelly@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.002
X-Spam-Evidence	'H': 1.00; 'S': 0.00; 'not': 0.07; 'objects,': 0.07; 'python': 0.09; 'collections': 0.09; 'counting': 0.09; 'garbage': 0.09; 'slices': 0.09; 'subject:()': 0.09; 'bug': 0.10; 'constructs': 0.16; 'gc.collect()': 0.16; 'iteration,': 0.16; 'oct': 0.16; 'subject:array': 0.16; 'true:': 0.16; 'wrote:': 0.17; 'appears': 0.18; 'memory': 0.18; 'import': 0.21; 'object.': 0.22; 'needed.': 0.23; 'seems': 0.23; 'header': 0.24; 'header:In-Reply- To:1': 0.25; 'am,': 0.27; 'expanding': 0.27; 'replace': 0.27; 'message-id:@mail.gmail.com': 0.27; 'contrast,': 0.29; 'objects': 0.29; "i'm": 0.29; 'code': 0.31; 'more,': 0.32; 'running': 0.32; 'certain': 0.33; 'to:addr:python-list': 0.33; 'received:google.com': 0.34; 'list': 0.35; 'received:209.85': 0.35; 'there': 0.35; 'list.': 0.35; 'but': 0.36; 'unable': 0.36; 'does': 0.37; 'why': 0.37; 'received:209': 0.37; 'subject:: ': 0.38; 'some': 0.38; 'to:addr:python.org': 0.39; 'header:Received:5': 0.40; 'your': 0.60; '30,': 0.62; 'between': 0.63; 'special': 0.73; 'cycles.': 0.84; 'to:name:python': 0.84
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=ygqh2w6UiAkyUFrSipgylhIdLs/QxKkO/iS6kaGbArc=; b=MmggHNQvYek0KkJXNQHQB2Y7wJTL5UnY+dt6byXZNB15Yb2+Adk+EAwPntfgL7CEo7 Z6pAXWPH5sPNsdvH4YFGwRxXwWgm7gl6NQ4FjpPisIfh2jB9mmVXSFLd8cLzaHcojRRK fNrkVjRT6sszcL5Wen+tLkPJrmOF/DvqHvFK9UT/5oCnEGpZKiiZOnBbGVDIR9kULJ/U 0Tt2zSZelxQB86wjHrfe47xgKpb3b5K4ufkZNWbhwbYBveEGAR9SPlgliaZf2xMMGnzN nhW2OrN6msQfHnjXgx6d6gr4Wsmp2UMLYgr5WUqEwhw2N7RKAlpifg6hzVk8FB/mdf7v PMeA==
MIME-Version	1.0
In-Reply-To	<CALwzid=5xkJ-SjzvHDStdGSnwHV-u+_QjZJLNJ8+tAs_L0k-qA@mail.gmail.com>
References	<6998a955-7b34-4f4f-b8d6-62d1028f7561@googlegroups.com> <mailman.2986.1351482209.27098.python-list@python.org> <4c024364-84df-403b-8b9e-4a4c8f06121c@googlegroups.com> <mailman.2991.1351484760.27098.python-list@python.org> <cd108b9d-e951-463e-8e50-a5a024a27c7f@googlegroups.com> <508e6649$0$29967$c3e8da3$5496439d@news.astraweb.com> <508E1BC9.3000308@r3dsolutions.com> <CAPTjJmo3uK0Hfu4h78AkM6BPXXuU9sMuc74QeKFrW4jp54MGMg@mail.gmail.com> <508EC428.5080808@r3dsolutions.com> <CALvWhxvJYfSatk7wg8zcBUHm5rHqeqqJss=N2vjCShGy9XoD5A@mail.gmail.com> <CALwzid=5xkJ-SjzvHDStdGSnwHV-u+_QjZJLNJ8+tAs_L0k-qA@mail.gmail.com>
From	Ian Kelly <ian.g.kelly@gmail.com>
Date	Tue, 30 Oct 2012 02:46:46 -0600
Subject	Re: Negative array indicies and slice()
To	Python <python-list@python.org>
Content-Type	text/plain; charset=ISO-8859-1
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.15
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.3083.1351586837.27098.python-list@python.org> (permalink)
Lines	31
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1351586837 news.xs4all.nl 6874 [2001:888:2000:d::a6]:44121
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:32477

Show key headers only | View raw

On Tue, Oct 30, 2012 at 1:21 AM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
> I'm not entirely certain why collection objects get this special
> treatment, but there you have it.

Thinking about it some more, this makes sense.  The GC header is there
to support garbage collection for the object.  Atomic types like ints
do not need this header because they do not reference other objects
and so cannot be involved in reference cycles.  For those types,
reference counting is sufficient.  For types like collections that do
reference other objects, garbage collection is needed.

Expanding on this, I suspect it is actually a bug that slice objects
are not tracked by the garbage collector.  The following code appears
to result in a memory leak:

import gc
gc.disable()
while True:
    for i in range(100):
        l = []
        s = slice(l)
        l.append(s)
        del s, l
    _ = gc.collect()

Try running that and watch your Python memory usage climb and climb.
For contrast, replace the slice with a list and observe that memory
usage does *not* climb.  On each iteration, the code constructs a
reference cycle between a slice and a list.  It seems that because
slices are not tracked by the garbage collector, it is unable to break
these cycles.

Thread

Negative array indicies and slice() andrewr3mail@gmail.com - 2012-10-28 20:12 -0700
  Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-28 21:42 -0600
    Re: Negative array indicies and slice() andrewr3mail@gmail.com - 2012-10-28 21:00 -0700
      Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-28 22:25 -0600
        Re: Negative array indicies and slice() Andrew <andrewr3mail@gmail.com> - 2012-10-29 00:54 -0700
          Re: Negative array indicies and slice() Chris Rebert <clp2@rebertia.com> - 2012-10-29 01:18 -0700
          Re: Negative array indicies and slice() Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-10-29 11:19 +0000
            Re: Negative array indicies and slice() Chris Angelico <rosuav@gmail.com> - 2012-10-29 22:32 +1100
            Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-28 21:52 -0700
            Re: Negative array indicies and slice() Chris Angelico <rosuav@gmail.com> - 2012-10-29 23:40 +1100
              Re: Negative array indicies and slice() Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-10-29 22:02 +0000
            Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-28 23:01 -0700
              Re: Negative array indicies and slice() Roy Smith <roy@panix.com> - 2012-10-29 09:52 -0400
                Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-29 08:20 -0700
                Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-29 17:01 -0600
                Re: Negative array indicies and slice() Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-10-30 00:04 +0000
                Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-29 16:54 -0700
                Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-30 02:15 -0600
            Re: Negative array indicies and slice() Chris Angelico <rosuav@gmail.com> - 2012-10-30 00:53 +1100
            Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-29 11:09 -0600
            Re: Negative array indicies and slice() Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-10-29 22:14 +0000
            Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-29 08:42 -0700
              Re: Negative array indicies and slice() Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-10-30 00:02 +0000
                Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-29 12:34 -0700
                Re: Negative array indicies and slice() Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-10-30 08:17 +0000
                Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-30 08:47 -0700
                Re: Negative array indicies and slice() Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-10-30 23:48 +0000
                Re: Negative array indicies and slice() Michael Torrie <torriem@gmail.com> - 2012-10-30 23:29 -0600
                Re: Negative array indicies and slice() Michael Torrie <torriem@gmail.com> - 2012-10-29 23:53 -0600
                Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-29 17:04 -0700
            Re: Negative array indicies and slice() Chris Angelico <rosuav@gmail.com> - 2012-10-30 09:55 +1100
            Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-29 17:07 -0600
              Re: Negative array indicies and slice() Roy Smith <roy@panix.com> - 2012-10-29 19:24 -0400
                Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-29 17:43 -0600
                Re: Negative array indicies and slice() Roy Smith <roy@panix.com> - 2012-10-29 20:17 -0400
                Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-29 18:05 -0600
            Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-29 11:00 -0700
            Re: Negative array indicies and slice() Chris Kaynor <ckaynor@zindagigames.com> - 2012-10-29 18:49 -0700
            Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-29 15:39 -0700
            Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-29 23:55 -0600
            Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-30 00:51 -0600
            Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-29 17:17 -0700
            Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-30 01:21 -0600
            Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-30 01:32 -0600
            Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-30 02:46 -0600
            Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-30 12:02 -0600
            Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-30 07:21 -0700
            Re: Negative array indicies and slice() Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-10-30 21:33 +0000
              Re: Negative array indicies and slice() Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-10-31 10:07 +0000
                Re: Negative array indicies and slice() Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-10-31 16:01 +0000
            Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-30 15:47 -0600
            Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-30 15:55 -0600
            Re: Negative array indicies and slice() Chris Angelico <rosuav@gmail.com> - 2012-10-31 09:00 +1100
            Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-30 16:02 -0600
            Re: Negative array indicies and slice() Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-10-30 23:30 +0000
          Re: Negative array indicies and slice() Ian Kelly <ian.g.kelly@gmail.com> - 2012-10-29 11:27 -0600
          Re: Negative array indicies and slice() Andrew Robinson <andrew3@r3dsolutions.com> - 2012-10-29 16:33 -0700
        Re: Negative array indicies and slice() Andrew <andrewr3mail@gmail.com> - 2012-10-29 00:54 -0700
    Re: Negative array indicies and slice() andrewr3mail@gmail.com - 2012-10-28 21:00 -0700
    Re: Negative array indicies and slice() Andrew <andrewr3mail@gmail.com> - 2012-10-28 21:09 -0700
      Re: Negative array indicies and slice() alex23 <wuwei23@gmail.com> - 2012-10-28 21:44 -0700
        Re: Negative array indicies and slice() andrewr3mail@gmail.com - 2012-10-29 01:24 -0700
          Re: Negative array indicies and slice() Chris Rebert <clp2@rebertia.com> - 2012-10-29 01:37 -0700
            Re: Negative array indicies and slice() andrewr3mail@gmail.com - 2012-10-29 01:59 -0700
              Re: Negative array indicies and slice() Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-10-29 09:36 +0000
              Re: Negative array indicies and slice() Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-10-29 10:34 +0000
            Re: Negative array indicies and slice() andrewr3mail@gmail.com - 2012-10-29 01:59 -0700
      Re: Negative array indicies and slice() Paul Rubin <no.email@nospam.invalid> - 2012-10-28 22:14 -0700
        Re: Negative array indicies and slice() andrewr3mail@gmail.com - 2012-10-29 01:08 -0700
          Re: Negative array indicies and slice() Chris Rebert <clp2@rebertia.com> - 2012-10-29 01:26 -0700
    Re: Negative array indicies and slice() Andrew <andrewr3mail@gmail.com> - 2012-10-28 21:09 -0700
  Re: Negative array indicies and slice() MRAB <python@mrabarnett.plus.com> - 2012-10-29 03:45 +0000
  Re: Negative array indicies and slice() 88888 Dihedral <dihedral88888@googlemail.com> - 2012-11-01 18:08 -0700

csiph-web