Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #78677

Re: "High water" Memory fragmentation still a thing?

Show all headers | View raw

On Sat, Oct 4, 2014 at 11:02 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Chris Angelico wrote:
>
>> In theory, a high
>> level language like Python would be allowed to move objects around to
>> compact memory, but CPython doesn't do this, and there's no proof that
>> it'd really help anything anyway.
>
> I welcome correction, but I understand that both the JVM and the CLR memory
> managers move memory around. That's why Jython and IronPython use
> sequential integers as object IDs, since memory locations are not fixed.

Right; I should have made it clearer that there's no proof that it'd
help anything *in CPython*. Removing the GIL is periodically proposed,
too, but there's no proof that its removal would benefit CPython; it's
not just that nobody's gotten around to writing a memory compactor for
CPython.

> Ironically, the cost of restarting the process periodically is likely to be
> orders of magnitude more expensive than that of moving a few blocks of
> memory around from time to time. Especially on Windows, where starting
> processes is expensive, but even on Linux you have to shut the running
> application down, then start it up again and rebuild all the internal data
> structures that you just tore down...

Maybe. But you deal with a number of things all at once:

1) Code updates (including interpreter updates)
2) Compaction of Python objects
3) Disposal of anything that got "high level leaked" - unintended
longevity caused by a global reference of some sort
4) Cleanup of low-level allocations that don't go through the Python
memory manager
etc etc etc.

So, yes, it's expensive. And sometimes it's not even possible (there's
no way to retain socket connections across a restart, AFAIK). But it's
there if you want it.

Personally, I like to keep processes running, but that's me. :)

ChrisA

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Re: "High water" Memory fragmentation still a thing? Chris Angelico <rosuav@gmail.com> - 2014-10-04 09:26 +1000
  Re: "High water" Memory fragmentation still a thing? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-10-04 11:02 +1000
    Re: "High water" Memory fragmentation still a thing? Chris Angelico <rosuav@gmail.com> - 2014-10-04 12:22 +1000
      Re: "High water" Memory fragmentation still a thing? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-10-04 15:48 +1000
        Re: "High water" Memory fragmentation still a thing? Chris Angelico <rosuav@gmail.com> - 2014-10-04 15:57 +1000
        Re: "High water" Memory fragmentation still a thing? Sturla Molden <sturla.molden@gmail.com> - 2014-10-04 17:51 +0000
    Re: "High water" Memory fragmentation still a thing? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-10-04 16:20 -0400
    Re: "High water" Memory fragmentation still a thing? mm0fmf <none@mailinator.com> - 2014-10-05 20:23 +0100
      Re: "High water" Memory fragmentation still a thing? Steven D'Aprano <steve@pearwood.info> - 2014-10-06 07:20 +0000
  Re: "High water" Memory fragmentation still a thing? bryanjugglercryptographer@yahoo.com - 2014-10-06 09:21 -0700
    Re: "High water" Memory fragmentation still a thing? Chris Angelico <rosuav@gmail.com> - 2014-10-07 03:43 +1100
      Re: "High water" Memory fragmentation still a thing? bryanjugglercryptographer@yahoo.com - 2014-10-08 10:28 -0700
        Re: "High water" Memory fragmentation still a thing? Emile van Sebille <emile@fenx.com> - 2014-10-09 14:39 -0700
        Re: "High water" Memory fragmentation still a thing? Chris Angelico <rosuav@gmail.com> - 2014-10-10 08:42 +1100
        Re: "High water" Memory fragmentation still a thing? dieter <dieter@handshake.de> - 2014-10-10 08:09 +0200
        Re: "High water" Memory fragmentation still a thing? Chris Angelico <rosuav@gmail.com> - 2014-10-10 17:52 +1100

csiph-web