Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #95395 > unrolled thread

Re: memory control in Python

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: memory control in Python Terry Reedy <tjreedy@udel.edu> - 2015-08-15 13:28 -0400

#95395 — Re: memory control in Python

On 8/15/2015 3:21 AM, dieter wrote:
> Ping Liu <yanzhipingliu@gmail.com> writes:
>> ...
>> For small cases, Python works well. But if we consider longer time period.
>> then it would fail due to the memory usage issues. We have tested several
>> case studies to check the memory use for different time period, including
>> 1) 2 hours in one day, 2) 24 hours in one day, 3) 20 days with 24 hours
>> each day, as well as 4) 30 days with 24 hours each day. The first 3 cases
>> are feasible while the last case gives out the memory error.
>>
>> When we are testing the forth case, the memory error comes out while
>> creating the inequality constraints. The problem size is 1) Aeq: 12 * 26,
>> Aineq: 30 * 26; 2) Aeq: 144*268, Aineq:316*268; 3) Aeq: 2880*5284, Aineq:
>> 6244*5284; 4) Aeq: 4320 * 7924, Aineq is unknown due to the memory error.
>>
>> The solver is CPLEX (academic). It turns out that the solver is taking a
>> lot of memory as you can see in the memory test report. for the first three
>> cases, different memory usage is observed, and it grows up dramatically
>> with the increase of the time period. 1) solver memory usage: 25.6 MB, 2)
>> 19.5 MB; 3) solver memory usage: 830.0742 MB.

Make sure that the solver is using numpy arrays.

> The C implementation of Python (called "CPython") does not use
> memory compaction and places most of its objects on the heap.
> Those implementations tend to suffer from memory fragmentation in
> long running processes with large memory use.

This can be helped somewhat by reusing arrays instead of deleting and 
creating a new one.


> You may consider "Jython" (a Python implementation in "Java") or
> "IronPython" (a ".Net" based Python implementation) to get a
> Python implementation with memory compaction.
>


-- 
Terry Jan Reedy

[toc] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web