Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #95390 > unrolled thread

Re: memory control in Python

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: memory control in Python dieter <dieter@handshake.de> - 2015-08-15 09:21 +0200
    Re: memory control in Python Ping Liu <yanzhipingliu@gmail.com> - 2015-08-17 15:09 -0700
      Re: memory control in Python Chris Angelico <rosuav@gmail.com> - 2015-08-18 10:13 +1000
      Re: memory control in Python dieter <dieter@handshake.de> - 2015-08-18 08:02 +0200
      Re: memory control in Python Laura Creighton <lac@openend.se> - 2015-08-18 08:16 +0200
      Re: memory control in Python Rustom Mody <rustompmody@gmail.com> - 2015-08-18 01:56 -0700
        Re: memory control in Python Laura Creighton <lac@openend.se> - 2015-08-18 11:13 +0200

#95390 — Re: memory control in Python

Ping Liu <yanzhipingliu@gmail.com> writes:
> ...
> For small cases, Python works well. But if we consider longer time period.
> then it would fail due to the memory usage issues. We have tested several
> case studies to check the memory use for different time period, including
> 1) 2 hours in one day, 2) 24 hours in one day, 3) 20 days with 24 hours
> each day, as well as 4) 30 days with 24 hours each day. The first 3 cases
> are feasible while the last case gives out the memory error.
>
> When we are testing the forth case, the memory error comes out while
> creating the inequality constraints. The problem size is 1) Aeq: 12 * 26,
> Aineq: 30 * 26; 2) Aeq: 144*268, Aineq:316*268; 3) Aeq: 2880*5284, Aineq:
> 6244*5284; 4) Aeq: 4320 * 7924, Aineq is unknown due to the memory error.
>
> The solver is CPLEX (academic). It turns out that the solver is taking a
> lot of memory as you can see in the memory test report. for the first three
> cases, different memory usage is observed, and it grows up dramatically
> with the increase of the time period. 1) solver memory usage: 25.6 MB, 2)
> 19.5 MB; 3) solver memory usage: 830.0742 MB.

The C implementation of Python (called "CPython") does not use
memory compaction and places most of its objects on the heap.
Those implementations tend to suffer from memory fragmentation in
long running processes with large memory use.

You may consider "Jython" (a Python implementation in "Java") or
"IronPython" (a ".Net" based Python implementation) to get a
Python implementation with memory compaction.

[toc] | [next] | [standalone]

#95447

Hi, Dieter,

If I move from Python to Jython or IronPython, do I need to retool whatever I have done? If so, that may take quite a long time. This may make the reimplementation impossible. 

[toc] | [prev] | [next] | [standalone]

#95453

On Tue, Aug 18, 2015 at 8:09 AM, Ping Liu <yanzhipingliu@gmail.com> wrote:
> If I move from Python to Jython or IronPython, do I need to retool whatever I have done? If so, that may take quite a long time. This may make the reimplementation impossible.

You're not moving from Python to something else; you're moving from
CPython to something else. It's like moving from Borland's C compiler
to Watcom's C compiler - all your code should still run unchanged.
There will be differences, but the bulk of your code shouldn't need
changing. With Python interpreters, the usual difference is extension
libraries - CPython can call on a bunch of things implemented in
native code, Jython can call on a bunch of things implemented in Java,
etc.

ChrisA

[toc] | [prev] | [next] | [standalone]

#95457

Ping Liu <yanzhipingliu@gmail.com> writes:

> If I move from Python to Jython or IronPython, do I need to retool whatever I have done? If so, that may take quite a long time. This may make the reimplementation impossible. 

As Chris already pointed out, you are still using Python -- i.e. the
base language does not change.

However, extension packages, such as "NumPy", might not be supported.
If you are using extension packages, carefully check whether they
are supported for a different Python implementation before you decide
a switch.

[toc] | [prev] | [next] | [standalone]

#95458

In a message of Tue, 18 Aug 2015 10:13:57 +1000, Chris Angelico writes:
>On Tue, Aug 18, 2015 at 8:09 AM, Ping Liu <yanzhipingliu@gmail.com> wrote:
>> If I move from Python to Jython or IronPython, do I need to retool whatever I have done? If so, that may take quite a long time. This may make the reimplementation impossible.
>
>You're not moving from Python to something else; you're moving from
>CPython to something else. It's like moving from Borland's C compiler
>to Watcom's C compiler - all your code should still run unchanged.
>There will be differences, but the bulk of your code shouldn't need
>changing. With Python interpreters, the usual difference is extension
>libraries - CPython can call on a bunch of things implemented in
>native code, Jython can call on a bunch of things implemented in Java,
>etc.
>
>ChrisA

Unless, as I expect, what he has done uses Numpy and or SciPy a lot.
Enthought is no longer supporting NumPy for IronPython (and it never
worked all that well, anyway, I am told ... but I never used it myself).
Even the maintainer of Jnumeric (which is trying to do Numeric not
NumPy) thinks the Jnumeric project should die.
http://stackoverflow.com/questions/18832169/numpy-analog-for-jython

We already know that Ping has a really big C extension he needs to work
with -- CPLEX, and well, here he may be in luck as there are java
versions of CPLEX and there is something called the CPLEX/Concert .NET
API  which may -- I never tried this -- let him work with Iron Python.
But that is one lirary.  If he has many more C extensions he needs to
use, then this could mean retooling them.

Laura

[toc] | [prev] | [next] | [standalone]

#95459

On Tuesday, August 18, 2015 at 3:40:11 AM UTC+5:30, Ping Liu wrote:
> Hi, Dieter,
> 
> If I move from Python to Jython or IronPython, do I need to retool whatever I have done? If so, that may take quite a long time. This may make the reimplementation impossible.

Hi Ping

There is a message from Laura Creighton that may be useful to you
that googlegroups shows in the thread: 
"python implementation of a new integer encoding algorithm."
Thought I'd mention in case you are not seeing other threads.

[How she (her mail client) manages to befuddle googlegroups thusly is 
quite a mystery...
]

[toc] | [prev] | [next] | [standalone]

#95460

In a message of Tue, 18 Aug 2015 01:56:16 -0700, Rustom Mody writes:
>[How she (her mail client) manages to befuddle googlegroups thusly is 
>quite a mystery...
>]

For me as well, as all I am doing is just replying to the mail ...  And
I haven't changed my mail client at all in years and years ...

Laura

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web