Path: csiph.com!eternal-september.org!feeder.eternal-september.org!border1.nntp.ams1.giganews.com!nntp.giganews.com!newsfeed.xs4all.nl!newsfeed8.news.xs4all.nl!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Terry Reedy <tjreedy@udel.edu>
Subject: Re: memory control in Python
Date: Sat, 15 Aug 2015 13:28:19 -0400
References: <CAM5e_kbbTAEHE=mi0WFXutJ5G2A-UV4wLsAPMcBT5XrCqhr3MA@mail.gmail.com> <87io8hc8z3.fsf@handshake.de>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0
In-Reply-To: <87io8hc8z3.fsf@handshake.de>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.31.1439659711.4764.python-list@python.org>
Lines: 41
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:95395

On 8/15/2015 3:21 AM, dieter wrote:
> Ping Liu <yanzhipingliu@gmail.com> writes:
>> ...
>> For small cases, Python works well. But if we consider longer time period.
>> then it would fail due to the memory usage issues. We have tested several
>> case studies to check the memory use for different time period, including
>> 1) 2 hours in one day, 2) 24 hours in one day, 3) 20 days with 24 hours
>> each day, as well as 4) 30 days with 24 hours each day. The first 3 cases
>> are feasible while the last case gives out the memory error.
>>
>> When we are testing the forth case, the memory error comes out while
>> creating the inequality constraints. The problem size is 1) Aeq: 12 * 26,
>> Aineq: 30 * 26; 2) Aeq: 144*268, Aineq:316*268; 3) Aeq: 2880*5284, Aineq:
>> 6244*5284; 4) Aeq: 4320 * 7924, Aineq is unknown due to the memory error.
>>
>> The solver is CPLEX (academic). It turns out that the solver is taking a
>> lot of memory as you can see in the memory test report. for the first three
>> cases, different memory usage is observed, and it grows up dramatically
>> with the increase of the time period. 1) solver memory usage: 25.6 MB, 2)
>> 19.5 MB; 3) solver memory usage: 830.0742 MB.

Make sure that the solver is using numpy arrays.

> The C implementation of Python (called "CPython") does not use
> memory compaction and places most of its objects on the heap.
> Those implementations tend to suffer from memory fragmentation in
> long running processes with large memory use.

This can be helped somewhat by reusing arrays instead of deleting and 
creating a new one.


> You may consider "Jython" (a Python implementation in "Java") or
> "IronPython" (a ".Net" based Python implementation) to get a
> Python implementation with memory compaction.
>


-- 
Terry Jan Reedy