Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!goblin3!goblin2!goblin.stu.neva.ru!feeder1.cambriumusenet.nl!feed.tweaknews.nl!194.109.133.85.MISMATCH!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Tue, 25 Sep 2012 07:06:29 -0400
From: Dave Angel <d@davea.name>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20120714 Thunderbird/14.0
MIME-Version: 1.0
To: Junkshops <junkshops@gmail.com>
Subject: Re: Memory usage per top 10x usage per heapy
References: <983c532f-3ff6-4bd2-bb48-07cf4d065a4b@googlegroups.com> <5061056F.6080702@davea.name> <50613131.6030102@gmail.com>
In-Reply-To: <50613131.6030102@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Cc: python-list@python.org
Precedence: list
Reply-To: d@davea.name
Newsgroups: comp.lang.python
Message-ID: <mailman.1318.1348571218.27098.python-list@python.org>
Lines: 43
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:30065

On 09/25/2012 12:21 AM, Junkshops wrote:
>> Just curious;  which is it, two million lines, or half a million bytes?
<snip>
> 
> Sorry, that should've been a 500Mb, 2M line file.
> 
>> which machine is 2gb, the Windows machine, or the VM?
> VM. Winders is 4gb.
> 
>> ...but I would point out that just because
>> you free up the memory from the Python doesn't mean it gets released
>> back to the system.  The C runtime manages its own heap, and is pretty
>> persistent about hanging onto memory once obtained.  It's not normally a
>> problem, since most small blocks are reused.  But it can get
>> fragmented.  And i have no idea how well Virtual Box maps the Linux
>> memory map into the Windows one.
> Right, I understand that - but what's confusing me is that, given the
> memory use is (I assume) monotonically increasing, the code should never
> use more than what's reported by heapy once all the data is loaded into
> memory, given that memory released by the code to the Python runtime is
> reused. To the best of my ability to tell I'm not storing anything I
> shouldn't, so the only thing I can think of is that all the object
> creation and destruction, for some reason, it preventing reuse of
> memory. I'm at a bit of a loss regarding what to try next.

I'm not familiar with heapy, but perhaps it's missing something there.
I'm a bit surprised you aren't beyond the 2gb limit, just with the
structures you describe for the file.  You do realize that each object
has quite a few bytes of overhead, so it's not surprising to use several
times the size of a file, to store the file in an organized way.  I also
wonder if heapy has been written to take into account the larger size of
pointers in a 64bit build.

Perhaps one way to save space would be to use a long to store those md5
values.  You'd have to measure it, but I suspect it'd help (at the cost
of lots of extra hexlify-type calls).  Another thing is to make sure
that the md5 object used in your two maps is the same object, and not
just one with the same value.


-- 

DaveA