Re: memory control in Python

Path	csiph.com!news.swapon.de!eternal-september.org!feeder.eternal-september.org!border1.nntp.ams1.giganews.com!nntp.giganews.com!newsfeed.xs4all.nl!newsfeed8.news.xs4all.nl!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path	<python-python-list@m.gmane.org>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.024
X-Spam-Evidence	'H': 0.95; 'S': 0.00; 'subject:Python': 0.05; 'report.': 0.07; 'ping': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'python': 0.10; '"ironpython"': 0.16; 'feasible': 0.16; 'liu': 0.16; 'received:80.91.229.3': 0.16; 'received:dip0.t-ipconnect.de': 0.16; 'received:plane.gmane.org': 0.16; 'received:t-ipconnect.de': 0.16; 'solver': 0.16; 'memory': 0.17; 'testing': 0.25; 'header:User-Agent:1': 0.26; 'header:X -Complaints-To:1': 0.26; 'error': 0.27; 'tend': 0.27; 'turns': 0.27; 'issues.': 0.29; 'objects': 0.29; 'creating': 0.30; 'error.': 0.31; 'problem': 0.33; 'case,': 0.34; 'running': 0.34; 'gives': 0.35; 'fail': 0.35; 'unknown': 0.35; 'but': 0.36; 'cases': 0.36; 'to:addr:python-list': 0.36; 'subject:: ': 0.37; 'received:org': 0.37; 'charset:us-ascii': 0.37; 'several': 0.38; 'test': 0.39; 'does': 0.39; 'well.': 0.40; 'to:addr:python.org': 0.40; 'received:de': 0.40; 'studies': 0.63; 'different': 0.63; 'day.': 0.63; 'places': 0.64; 'hours': 0.65; 'due': 0.65; 'day,': 0.65; 'received:217': 0.66; '26,': 0.72; 'increase': 0.73; 'forth': 0.79; 'dramatically': 0.84; 'suffer': 0.93; 'period.': 0.95
X-Injected-Via-Gmane	http://gmane.org/
To	python-list@python.org
From	dieter <dieter@handshake.de>
Subject	Re: memory control in Python
Date	Sat, 15 Aug 2015 09:21:52 +0200
References	<CAM5e_kbbTAEHE=mi0WFXutJ5G2A-UV4wLsAPMcBT5XrCqhr3MA@mail.gmail.com>
Mime-Version	1.0
Content-Type	text/plain; charset=us-ascii
X-Gmane-NNTP-Posting-Host	pd9e08ba4.dip0.t-ipconnect.de
User-Agent	Gnus/5.1008 (Gnus v5.10.8) XEmacs/21.4.22 (linux)
Cancel-Lock	sha1:TDl7xGXMscNcIPOCooX7xs/lG9A=
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.20+
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.24.1439623320.4764.python-list@python.org> (permalink)
Lines	29
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1439623320 news.xs4all.nl 2829 [2001:888:2000:d::a6]:40903
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:95390

Show key headers only | View raw

Ping Liu <yanzhipingliu@gmail.com> writes:
> ...
> For small cases, Python works well. But if we consider longer time period.
> then it would fail due to the memory usage issues. We have tested several
> case studies to check the memory use for different time period, including
> 1) 2 hours in one day, 2) 24 hours in one day, 3) 20 days with 24 hours
> each day, as well as 4) 30 days with 24 hours each day. The first 3 cases
> are feasible while the last case gives out the memory error.
>
> When we are testing the forth case, the memory error comes out while
> creating the inequality constraints. The problem size is 1) Aeq: 12 * 26,
> Aineq: 30 * 26; 2) Aeq: 144*268, Aineq:316*268; 3) Aeq: 2880*5284, Aineq:
> 6244*5284; 4) Aeq: 4320 * 7924, Aineq is unknown due to the memory error.
>
> The solver is CPLEX (academic). It turns out that the solver is taking a
> lot of memory as you can see in the memory test report. for the first three
> cases, different memory usage is observed, and it grows up dramatically
> with the increase of the time period. 1) solver memory usage: 25.6 MB, 2)
> 19.5 MB; 3) solver memory usage: 830.0742 MB.

The C implementation of Python (called "CPython") does not use
memory compaction and places most of its objects on the heap.
Those implementations tend to suffer from memory fragmentation in
long running processes with large memory use.

You may consider "Jython" (a Python implementation in "Java") or
"IronPython" (a ".Net" based Python implementation) to get a
Python implementation with memory compaction.

Back to comp.lang.python | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Re: memory control in Python dieter <dieter@handshake.de> - 2015-08-15 09:21 +0200
  Re: memory control in Python Ping Liu <yanzhipingliu@gmail.com> - 2015-08-17 15:09 -0700
    Re: memory control in Python Chris Angelico <rosuav@gmail.com> - 2015-08-18 10:13 +1000
    Re: memory control in Python dieter <dieter@handshake.de> - 2015-08-18 08:02 +0200
    Re: memory control in Python Laura Creighton <lac@openend.se> - 2015-08-18 08:16 +0200
    Re: memory control in Python Rustom Mody <rustompmody@gmail.com> - 2015-08-18 01:56 -0700
      Re: memory control in Python Laura Creighton <lac@openend.se> - 2015-08-18 11:13 +0200

csiph-web