Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #40609 > unrolled thread

sync databse table based on current directory data without losign previous values

Back to article view | Back to comp.lang.python

Contents

  sync databse table based on current directory data without losign previous values Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-03-05 23:45 -0800
    Re: sync databse table based on current directory data without losign previous values Lele Gaifax <lele@metapensiero.it> - 2013-03-06 09:19 +0100
      Re: sync databse table based on current directory data without losign previous values Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-03-06 00:57 -0800
        Re: sync databse table based on current directory data without losign previous values Lele Gaifax <lele@metapensiero.it> - 2013-03-06 10:24 +0100
          Re: sync databse table based on current directory data without losign previous values Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-03-06 01:41 -0800
            Re: sync databse table based on current directory data without losign previous values Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-03-06 01:43 -0800
              Re: sync databse table based on current directory data without losign previous values Bryan Devaney <bryan.devaney@gmail.com> - 2013-03-06 02:15 -0800
              Re: sync databse table based on current directory data without losign previous values Bryan Devaney <bryan.devaney@gmail.com> - 2013-03-06 02:15 -0800
            Re: sync databse table based on current directory data without losign previous values Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-03-06 01:43 -0800
            Re: sync databse table based on current directory data without losign previous values Lele Gaifax <lele@metapensiero.it> - 2013-03-06 11:27 +0100
            Re: sync databse table based on current directory data without losign previous values Dave Angel <davea@davea.name> - 2013-03-06 08:31 -0500
            Re: sync databse table based on current directory data without losign previous values Lele Gaifax <lele@metapensiero.it> - 2013-03-06 15:16 +0100
          Re: sync databse table based on current directory data without losign previous values Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-03-06 01:41 -0800
      Re: sync databse table based on current directory data without losign previous values Νίκος Γκρ33κ <nikos.gr33k@gmail.com> - 2013-03-06 00:57 -0800
    Set x to  to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Wong Wah Meng-R32813 <r32813@freescale.com> - 2013-03-06 10:11 +0000
      Re: Set x to  to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Bryan Devaney <bryan.devaney@gmail.com> - 2013-03-06 02:25 -0800
        RE: Set x to  to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Wong Wah Meng-R32813 <r32813@freescale.com> - 2013-03-06 12:31 +0000
        Re: Set x to  to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Dave Angel <davea@davea.name> - 2013-03-06 08:18 -0500
        Re: Set x to  to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Dave Angel <davea@davea.name> - 2013-03-06 08:25 -0500
      Re: Set x to  to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Bryan Devaney <bryan.devaney@gmail.com> - 2013-03-06 02:25 -0800
      Re: Set x to  to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-03-06 23:34 +0000
        RE: Set x to  to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Wong Wah Meng-R32813 <r32813@freescale.com> - 2013-03-07 06:33 +0000
        Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Chris Angelico <rosuav@gmail.com> - 2013-03-07 18:19 +1100
        RE: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Wong Wah Meng-R32813 <r32813@freescale.com> - 2013-03-08 09:08 +0000
        Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-03-08 19:40 -0500
        RE: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Wong Wah Meng-R32813 <r32813@freescale.com> - 2013-03-09 08:07 +0000
          Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Grant Edwards <invalid@invalid.invalid> - 2013-03-09 19:18 +0000
            Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Roy Smith <roy@panix.com> - 2013-03-09 15:04 -0500
              Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Grant Edwards <invalid@invalid.invalid> - 2013-03-09 20:35 +0000
                Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Roy Smith <roy@panix.com> - 2013-03-09 16:44 -0500
                  Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Grant Edwards <invalid@invalid.invalid> - 2013-03-11 14:27 +0000
        Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Dave Angel <davea@davea.name> - 2013-03-09 06:02 -0500
        Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Isaac To <isaac.to@gmail.com> - 2013-03-09 23:02 +0800
    Re: Set x to  to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Terry Reedy <tjreedy@udel.edu> - 2013-03-06 05:59 -0500
    Re: sync databse table based on current directory data without losign previous values Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-03-06 11:52 +0000
    RE: Set x to  to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64) Wong Wah Meng-R32813 <r32813@freescale.com> - 2013-03-06 12:36 +0000
    Re: sync databse table based on current directory data without losign previous values Chris Angelico <rosuav@gmail.com> - 2013-03-07 00:40 +1100
    Re: sync databse table based on current directory data without losign previous values "Michael Ross" <gmx@ross.cx> - 2013-03-06 15:04 +0100
      Re: sync databse table based on current directory data without losign previous values nagia.retsina@gmail.com - 2013-03-06 08:09 -0800
      Re: sync databse table based on current directory data without losign previous values nagia.retsina@gmail.com - 2013-03-06 08:09 -0800

Page 2 of 2 — ← Prev page 1 [2]

#40669 — Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64)

On Wed, 06 Mar 2013 10:11:12 +0000, Wong Wah Meng-R32813 wrote:

> Hello there,
> 
> I am using python 2.7.1 built on HP-11.23 a Itanium 64 bit box.
> 
> I discovered following behavior whereby the python process doesn't seem
> to release memory utilized even after a variable is set to None, and
> "deleted". I use glance tool to monitor the memory utilized by this
> process. Obviously after the for loop is executed, the memory used by
> this process has hiked to a few MB. However, after "del" is executed to
> both I and str variables, the memory of that process still stays at
> where it was.
> 
> Any idea why?

Python does not guarantee to return memory to the operating system. 
Whether it does or not depends on the OS, but as a general rule, you 
should expect that it will not.


>>>> for i in range(100000L):
> ...     str=str+"%s"%(i,)


You should never build large strings in that way. It risks being 
horribly, horribly slow on some combinations of OS, Python implementation 
and version.

Instead, you should do this:

items = ["%s" % i for i in range(100000)]
s = ''.join(items)


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#40710 — RE: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64)

Python does not guarantee to return memory to the operating system. 
Whether it does or not depends on the OS, but as a general rule, you should expect that it will not.


>>>> for i in range(100000L):
> ...     str=str+"%s"%(i,)


You should never build large strings in that way. It risks being 
horribly, horribly slow on some combinations of OS, Python implementation 
and version.

Instead, you should do this:

items = ["%s" % i for i in range(100000)]
s = ''.join(items)

[] The example is written for illustration purpose. Thanks for pointing out a better way of achieving the same result. Yes it seems so that the OS thinks the piece allocated to Python should not be taken back unless the process dies. :(

[toc] | [prev] | [next] | [standalone]

#40713 — Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64)

On Thu, Mar 7, 2013 at 5:33 PM, Wong Wah Meng-R32813
<r32813@freescale.com> wrote:
> [] The example is written for illustration purpose. Thanks for pointing out a better way of achieving the same result. Yes it seems so that the OS thinks the piece allocated to Python should not be taken back unless the process dies. :(

Don't be too bothered by that. That memory will be reused by Python
for subsequent allocations.

ChrisA

[toc] | [prev] | [next] | [standalone]

#40854 — RE: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64)

The problem is my server hits memory usage threshold, and starts giving me errors like Oracle unable to spawn off new session stating Out of Memory error and what not. I won't be bothered much if I have the luxury of available memory for other processes to use. If only if the UNIX understand my concerns and release the allocation when I issue gc.collect() or the gc.collect() takes place. :)



 


On Thu, Mar 7, 2013 at 5:33 PM, Wong Wah Meng-R32813 <r32813@freescale.com> wrote:
> [] The example is written for illustration purpose. Thanks for 
> pointing out a better way of achieving the same result. Yes it seems 
> so that the OS thinks the piece allocated to Python should not be 
> taken back unless the process dies. :(

Don't be too bothered by that. That memory will be reused by Python for subsequent allocations.

[toc] | [prev] | [next] | [standalone]

#40913 — Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64)

On Fri, 8 Mar 2013 09:08:15 +0000, Wong Wah Meng-R32813
<r32813@freescale.com> declaimed the following in
gmane.comp.python.general:

> The problem is my server hits memory usage threshold, and starts giving me errors like Oracle unable to spawn off new session stating Out of Memory error and what not. I won't be bothered much if I have the luxury of available memory for other processes to use. If only if the UNIX understand my concerns and release the allocation when I issue gc.collect() or the gc.collect() takes place. :)
>

	If the memory usage is continually growing, you have something else
that is a problem -- something is holding onto objects. Even if Python
is not returning memory to the OS, it should be reusing the memory it
has if objects are being freed.
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#40937 — RE: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64)

	If the memory usage is continually growing, you have something else that is a problem -- something is holding onto objects. Even if Python is not returning memory to the OS, it should be reusing the memory it has if objects are being freed.
-- 
[] Yes I have verified my python application is reusing the memory (just that it doesn't reduce once it has grown) and my python process doesn't have any issue to run even though it is seen taking up more than 2G in footprint. My problem is capacity planning on the server whereby since my python process doesn't release memory back to the OS, the OS wasn't able to allocate memory when a new process is spawn off.   

[toc] | [prev] | [next] | [standalone]

#40968 — Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64)

On 2013-03-09, Wong Wah Meng-R32813 <r32813@freescale.com> wrote:

Your entire post is in your signature block. Don't do that. Many
people have newsreaders or e-mail clinets configured to hide or ignore
signature blocks.

>Yes I have verified my python application is reusing the memory (just
>that it doesn't reduce once it has grown) and my python process
>doesn't have any issue to run even though it is seen taking up more
>than 2G in footprint.

Then there's nothing wrong with Python.

>My problem is capacity planning on the server whereby since my python
>process doesn't release memory back to the OS,

In Unix there is no way to release heap memory (which is what you're
talking about) back to the OS except for terminating the process.

>the OS wasn't able to allocate memory when a new process is spawn off.

You need to either get more memory, change your Python program to use
less memory, or have your Python program terminate in order to return
memory to the OS.

Perhaps you can put the memory hungery portion of your processing into
a subprocess that terminates when it's completed its work?

-- 
Grant Edwards               grant.b.edwards        Yow! Don't hit me!!  I'm in
                                  at               the Twilight Zone!!!
                              gmail.com            

[toc] | [prev] | [next] | [standalone]

#40972 — Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64)

In article <khg1v3$l7q$1@reader2.panix.com>,
 Grant Edwards <invalid@invalid.invalid> wrote:

> In Unix there is no way to release heap memory (which is what you're
> talking about) back to the OS except for terminating the process.

That's not quite true.  The man page for BRK(2) (at least on the Linux 
box I happen to have handy) says:

"brk() and sbrk() change the location of the program break, which 
defines the  end  of  the process's data segment (i.e., the program 
break is the first location after the end of the uninitialized data 
segment).  Increasing the program break has the  effect  of  allocating 
memory to the process; decreasing the break deallocates memory."

So, in theory, it's possible.  I just ran this C program:

#include <unistd.h>
#include <time.h>

int main(int argc, char** argv) {
    struct timespec t;
    t.tv_sec = 10;
    t.tv_nsec = 0;

    nanosleep(&t, NULL);
    sbrk(500 * 1024 * 1024);

    nanosleep(&t, NULL);
    sbrk(-500 * 1024 * 1024);

    nanosleep(&t, NULL);
}

while watching the process with ps.  I could see the process appear and 
for the first 10 seconds it had a VSZ of 4156.  Then, for the next 10 
seconds, the VSZ was 516156, then it went back down to 4156.

$ while sleep 1; do ps augx | grep a.out | grep -v grep; done
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0 516156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0 516156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0 516156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0 516156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0 516156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0 516156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0 516156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0 516156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0 516156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out
roy       6657  0.0  0.0   4156   356 pts/10   S+   19:56   0:00 ./a.out

In practice, unless you go to extraordinary lengths (i.e. avoid almost 
all of the standard C library), the break is going to be managed by 
malloc(), and malloc() provides no way to give memory back (insert 
handwave here about mmap, and another handwave about various versions of 
malloc).  But, that's due to the way malloc() manages its arena, not 
anything fundamental about how Unix works.

[toc] | [prev] | [next] | [standalone]

#40975 — Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64)

On 2013-03-09, Roy Smith <roy@panix.com> wrote:
> In article <khg1v3$l7q$1@reader2.panix.com>,
>  Grant Edwards <invalid@invalid.invalid> wrote:
>
>> In Unix there is no way to release heap memory (which is what you're
>> talking about) back to the OS except for terminating the process.
>
> That's not quite true.  The man page for BRK(2) (at least on the Linux 
> box I happen to have handy) says:
>
> "brk() and sbrk() change the location of the program break, which 
> defines the  end  of  the process's data segment (i.e., the program 
> break is the first location after the end of the uninitialized data 
> segment).  Increasing the program break has the  effect  of  allocating 
> memory to the process; decreasing the break deallocates memory."
>
> So, in theory, it's possible.

Well spotted.  I would have bet money (OK, not a lot), that sbrk()
only accepted positive increment values.  I seem to have conflated
what sbrk() will accept and the fact that the C library's malloc/free
calls never call sbrk() with a nagative number.

[... nicely done demonstraction of negative sbrk() parameter usage ...]

> In practice, unless you go to extraordinary lengths (i.e. avoid
> almost all of the standard C library), the break is going to be
> managed by malloc(), and malloc() provides no way to give memory back
> (insert handwave here about mmap, and another handwave about various
> versions of malloc).  But, that's due to the way malloc() manages its
> arena, not anything fundamental about how Unix works.

Indeed.

What I should have said was that there's no way to return to the OS
memory obtained via calls to malloc() et al, and those are the calls
that "good" C programmers (like the maintainers of CPython) use.

In theory, you could modify the C library so that calls to free()
might return memory to the OS.  Sometimes.  If you're lucky.

The problem is that if there's _one_byte_ of memory in use at the
"end" of the heap region, it doesn't matter if there's an unused 16GB
chunk in the middle -- it can't be returned to the OS.

-- 
Grant Edwards               grant.b.edwards        Yow! Look DEEP into the
                                  at               OPENINGS!!  Do you see any
                              gmail.com            ELVES or EDSELS ... or a
                                                   HIGHBALL?? ...

[toc] | [prev] | [next] | [standalone]

#40980 — Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64)

In article <khg6f3$cfr$1@reader2.panix.com>,
 Grant Edwards <invalid@invalid.invalid> wrote:

> What I should have said was that there's no way to return to the OS
> memory obtained via calls to malloc() et al.

That's true (for certain values of "et al").

> and those are the calls that "good" C programmers (like the
> maintainers of CPython) use.

Well, there is mmap, which is exposed via the Python mmap module.  
Python doesn't have anything like C++'s "placement new", so there's no 
way to use that memory to hold generic Python objects, but you can 
certainly use mmap to allocate a large chunk of memory, use it, and then 
give it back to the OS.  For example:

#!/usr/bin/env python                                                                                                                                               

import mmap
import time

time.sleep(5)

f = open('my-500-mbyte-text-file')
data = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ)

count = 0
while 1:
    line = data.readline()
    if not line:
        break
    count += 1

print count
time.sleep(5)
data.close()
time.sleep(5)

When I run that and watch the process size (like I did with the previous 
example), you can see the process grow when mmap() is called, and shrink 
again when the segment is closed.

I have to admit, in all the years I've been using Python, this is the 
first time I've ever used the mmap module.  Even for long running 
processes, the automatic memory management that Python provides out of 
the box has always been good enough for me.  But, it's nice to know mmap 
is there if I need to do something unusual.

[toc] | [prev] | [next] | [standalone]

#41063 — Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64)

On 2013-03-09, Roy Smith <roy@panix.com> wrote:
> In article <khg6f3$cfr$1@reader2.panix.com>,
>  Grant Edwards <invalid@invalid.invalid> wrote:
>
>> What I should have said was that there's no way to return to the OS
>> memory obtained via calls to malloc() et al.
>
> That's true (for certain values of "et al").
>
>> and those are the calls that "good" C programmers (like the
>> maintainers of CPython) use.
>
> Well, there is mmap, which is exposed via the Python mmap module.  
> Python doesn't have anything like C++'s "placement new", so there's
> no way to use that memory to hold generic Python objects, but you can
> certainly use mmap to allocate a large chunk of memory, use it, and
> then give it back to the OS.

[example]

I was surprised to find that the object returned by mmap.mmap()
supported file semantics (seek, tell, read, etc.) in addition to byte
buffer semantics.  Usually, the reason one maps a file is that one
doesn't want to use file semantics (which are already supported by the
file object) but wants instead to use buffer semantics.

Using memmap to obtain a "returnable" chunk of memory seems a bit
obtuse, since it requires creating an underlying file of appropriate
size that ends up being superfluous.

-- 
Grant Edwards               grant.b.edwards        Yow! I wonder if I ought
                                  at               to tell them about my
                              gmail.com            PREVIOUS LIFE as a COMPLETE
                                                   STRANGER?

[toc] | [prev] | [next] | [standalone]

#40940 — Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64)

On 03/09/2013 03:07 AM, Wong Wah Meng-R32813 wrote:
>
>
>  Yes I have verified my python application is reusing the memory (just that it doesn't reduce once it has grown) and my python process doesn't have any issue to run even though it is seen taking up more than 2G in footprint. My problem is capacity planning on the server whereby since my python process doesn't release memory back to the OS, the OS wasn't able to allocate memory when a new process is spawn off>

So is the server running out of disk space?  If not, why not increase 
the swapfile(s) size?  That's what's being held onto, not RAM.

If you are short on disk space, and therefore cannot afford to increase 
the swapfile size, then you may want to plan the Python program's 
execution around that constraint.

Discounting the play-program that started this thread, is it possible 
that your actual app unnecessarily uses lots of space during one phase 
of execution, and therefore is saddled with that space at other times? 
For example, are you using a large list to hold a data file when an 
iterator would do just as well?

If that phase of execution cannot be eliminated, but it's a fleeting 
time, perhaps that part of the execution can be launched from the main 
app as a separate process.  When the process ends, the memory is freed.

-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#40948 — Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64)

[Multipart message — attachments visible in raw view] — view raw

In general, it is hard for any process to return the memory the OS allocate
to it back to the OS, short of exiting the whole process.  The only case
that this works reliably is when the process allocates a chunk of memory by
mmap (which is chosen by libc if it malloc or calloc a large chunk of
memory), and that whole chunk is not needed any more.  In that case the
process can munmap it.  Evidently you are not see that in your program.
What you allocate might be too small (so libc choose to allocate it using
another system call "sbrk"), or that the allocated memory also hold other
objects not freed.

If you want to reduce the footprint of a long running program that
periodically allocates a large chunk of memory, the "easiest" solution is
to fork a different process to achieve the computations that needs the
memory.  That way, you can exit the process after you complete the
computation, and at that point all memory allocated to it is guaranteed to
be freed to the OS.

Modules like multiprocessing probably make the idea sufficiently easy to
implement.


On Sat, Mar 9, 2013 at 4:07 PM, Wong Wah Meng-R32813
<r32813@freescale.com>wrote:

>
>
>         If the memory usage is continually growing, you have something
> else that is a problem -- something is holding onto objects. Even if Python
> is not returning memory to the OS, it should be reusing the memory it has
> if objects are being freed.
> --
> [] Yes I have verified my python application is reusing the memory (just
> that it doesn't reduce once it has grown) and my python process doesn't
> have any issue to run even though it is seen taking up more than 2G in
> footprint. My problem is capacity planning on the server whereby since my
> python process doesn't release memory back to the OS, the OS wasn't able to
> allocate memory when a new process is spawn off.
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#40627 — Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64)

On 3/6/2013 5:11 AM, Wong Wah Meng-R32813 wrote:
> Hello there,
>
> I am using python 2.7.1 built on HP-11.23 a Itanium 64 bit box.
>
> I discovered following behavior whereby the python process doesn't
> seem to release memory utilized even after a variable is set to None,
> and "deleted". I use glance tool to monitor the memory utilized by
> this process. Obviously after the for loop is executed, the memory
> used by this process has hiked to a few MB. However, after "del" is
> executed to both I and str variables, the memory of that process
> still stays at where it was.

Whether memory freed by deleting an object is returned to and taken by 
the OS depends on the OS and other factors like like the size and layout 
of the freed memory, probably the history of memory use, and for 
CPython, the C compiler's malloc/free implementation. At various times, 
the Python memory handlers have been rewritten to encourage/facilitate 
memory return, but Python cannot control the process.

> for i in range(100000L):
>      str=str+"%s"%(i,)
> i=None; str=None   # not necessary
 > del i; del str

Reusing built-in names for unrelated purposes is generally a bad idea, 
although the final deletion does restore access to the builtin.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#40629

On 06/03/2013 07:45, Νίκος Γκρ33κ wrote:
> I'am using this snipper to read a current directory and insert all filenames into a databse and then display them.
>
> But what happens when files are get removed form the directory?
> The inserted records into databse remain.
> How can i update  the databse to only contain the existing filenames without losing the previous stored data?
>
> Here is what i ahve so far:
>
> ==================================
> path = "/home/nikos/public_html/data/files/"
>
> #read the containing folder and insert new filenames
> for result in os.walk(path):

You were told yesterday at least twice that os.walk returns a tuple but 
you still insist on refusing to take any notice of our replies when it 
suits you, preferring instead to waste everbody's time with these 
questions.  Or are you trying to get into the Guinness Book of World 
Records for the laziest bastard on the planet?

> 	for filename in result[2]:
> 		try:
> 			#find the needed counter for the page URL
> 			cur.execute('''SELECT URL FROM files WHERE URL = %s''', (filename,) )
> 			data = cur.fetchone()        #URL is unique, so should only be one
> 			
> 			if not data:
> 				#first time for file; primary key is automatic, hit is defaulted
> 				cur.execute('''INSERT INTO files (URL, host, lastvisit) VALUES (%s, %s, %s)''', (filename, host, date) )
> 		except MySQLdb.Error, e:
> 			print ( "Query Error: ", sys.exc_info()[1].excepinfo()[2] )
> ======================
>
> Thank you.
>

-- 
Cheers.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#40631 — RE: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64)

Thanks for youre reply. I built python 2.7.1 binary myself on the HP box and I wasn't aware there is any configuration or setup that I need to modify in order to activate or engage the garbage collection (or even setting the memory size used). Probably you are right it leaves it to the OS itself (in this case HP-UX) to clean it up as after python removes the reference to the address of the variables the OS still thinks the python process should still owns it until the process exits. 

Regards,
Wah Meng 

-----Original Message-----
From: Python-list [mailto:python-list-bounces+wahmeng=freescale.com@python.org] On Behalf Of Terry Reedy
Sent: Wednesday, March 06, 2013 7:00 PM
To: python-list@python.org
Subject: Re: Set x to to None and del x doesn't release memory in python 2.7.1 (HPUX 11.23, ia64)

On 3/6/2013 5:11 AM, Wong Wah Meng-R32813 wrote:
> Hello there,
>
> I am using python 2.7.1 built on HP-11.23 a Itanium 64 bit box.
>
> I discovered following behavior whereby the python process doesn't 
> seem to release memory utilized even after a variable is set to None, 
> and "deleted". I use glance tool to monitor the memory utilized by 
> this process. Obviously after the for loop is executed, the memory 
> used by this process has hiked to a few MB. However, after "del" is 
> executed to both I and str variables, the memory of that process still 
> stays at where it was.

Whether memory freed by deleting an object is returned to and taken by the OS depends on the OS and other factors like like the size and layout of the freed memory, probably the history of memory use, and for CPython, the C compiler's malloc/free implementation. At various times, the Python memory handlers have been rewritten to encourage/facilitate memory return, but Python cannot control the process.

> for i in range(100000L):
>      str=str+"%s"%(i,)
> i=None; str=None   # not necessary
 > del i; del str

Reusing built-in names for unrelated purposes is generally a bad idea, although the final deletion does restore access to the builtin.

--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list

[toc] | [prev] | [next] | [standalone]

#40636

On Wed, Mar 6, 2013 at 10:52 PM, Mark Lawrence <breamoreboy@yahoo.co.uk> wrote:
> On 06/03/2013 07:45, Νίκος Γκρ33κ wrote:
>> blah blah blah
>> blah blah blah
>> blah blah blah
> You were told yesterday at least twice that os.walk returns a tuple but you
> still insist on refusing to take any notice of our replies when it suits
> you, preferring instead to waste everbody's time with these questions.  Or
> are you trying to get into the Guinness Book of World Records for the
> laziest bastard on the planet?

This is the same person who posts under the name Ferrous Cranus. His
threads are open-ended time sinks; he generally expects everyone else
to do the work, while his contribution is to complain that the
provided code doesn't fit some arbitrary set of restrictions that
don't always even make sense.

Guinness might be interested in him as "Most Successful Troll",
though. He certainly absorbed a lot of my dev time before I gave up on
him, and by the look of things, he's still getting other people to do
his work for him.

ChrisA

[toc] | [prev] | [next] | [standalone]

#40638

On Wed, 06 Mar 2013 12:52:00 +0100, Mark Lawrence  
<breamoreboy@yahoo.co.uk> wrote:

> On 06/03/2013 07:45, Νίκος Γκρ33κ wrote:
>> I'am using this snipper to read a current directory and insert all  
>> filenames into a databse and then display them.
>>
>> But what happens when files are get removed form the directory?
>> The inserted records into databse remain.
>> How can i update  the databse to only contain the existing filenames  
>> without losing the previous stored data?
>>
>> Here is what i ahve so far:
>>
>> ==================================
>> path = "/home/nikos/public_html/data/files/"
>>
>> #read the containing folder and insert new filenames
>> for result in os.walk(path):
>
> You were told yesterday at least twice that os.walk returns a tuple but  
> you still insist on refusing to take any notice of our replies when it  
> suits you, preferring instead to waste everbody's time with these  
> questions.  Or are you trying to get into the Guinness Book of World  
> Records for the laziest bastard on the planet?

Hold on a sec ...

He has

for result in os.walk(path):
	for filename in result[2]:

So he *did* take notice of that.


Nikos:
Expectation is to iterate through a tuple like this:

for dirpath, dirnames, filenames in os.walk(path):
	...

[toc] | [prev] | [next] | [standalone]

#40642

Τη Τετάρτη, 6 Μαρτίου 2013 4:04:26 μ.μ. UTC+2, ο χρήστης Michael Ross έγραψε:
> On Wed, 06 Mar 2013 12:52:00 +0100, Mark Lawrence  
> 
> <breamoreboy@yahoo.co.uk> wrote:
> 
> 
> 
> > On 06/03/2013 07:45, Νίκος Γκρ33κ wrote:
> 
> >> I'am using this snipper to read a current directory and insert all  
> 
> >> filenames into a databse and then display them.
> 
> >>
> 
> >> But what happens when files are get removed form the directory?
> 
> >> The inserted records into databse remain.
> 
> >> How can i update  the databse to only contain the existing filenames  
> 
> >> without losing the previous stored data?
> 
> >>
> 
> >> Here is what i ahve so far:
> 
> >>
> 
> >> ==================================
> 
> >> path = "/home/nikos/public_html/data/files/"
> 
> >>
> 
> >> #read the containing folder and insert new filenames
> 
> >> for result in os.walk(path):
> 
> >
> 
> > You were told yesterday at least twice that os.walk returns a tuple but  
> 
> > you still insist on refusing to take any notice of our replies when it  
> 
> > suits you, preferring instead to waste everbody's time with these  
> 
> > questions.  Or are you trying to get into the Guinness Book of World  
> 
> > Records for the laziest bastard on the planet?
> 
> 
> 
> Hold on a sec ...
> 
> 
> 
> He has
> 
> 
> 
> for result in os.walk(path):
> 
> 	for filename in result[2]:
> 
> 
> 
> So he *did* take notice of that.
> 
> 
> 
> 
> 
> Nikos:
> 
> Expectation is to iterate through a tuple like this:
> 
> 
> 
> for dirpath, dirnames, filenames in os.walk(path):
> 
> 	...

Thank you Michael, yes i ahve understood that myself yesterday after one of the guys here have shown the output of os.walk(path) in IDLE. 
So, yes in fact i was in need for the 3rd item in the tuple, so to get hold on to the files.

Thank you for supporting me, some ppl here think i'am a troll and don't try thinks or ignore evrything but 'some_ppl's_opinion != True'

I'am just sometimes persistant on having thing my way that why i was calling my self Ferrous cRanus, which i changes it since it was annoying....

[toc] | [prev] | [next] | [standalone]

#40653

Τη Τετάρτη, 6 Μαρτίου 2013 4:04:26 μ.μ. UTC+2, ο χρήστης Michael Ross έγραψε:
> On Wed, 06 Mar 2013 12:52:00 +0100, Mark Lawrence  
> 
> <breamoreboy@yahoo.co.uk> wrote:
> 
> 
> 
> > On 06/03/2013 07:45, Νίκος Γκρ33κ wrote:
> 
> >> I'am using this snipper to read a current directory and insert all  
> 
> >> filenames into a databse and then display them.
> 
> >>
> 
> >> But what happens when files are get removed form the directory?
> 
> >> The inserted records into databse remain.
> 
> >> How can i update  the databse to only contain the existing filenames  
> 
> >> without losing the previous stored data?
> 
> >>
> 
> >> Here is what i ahve so far:
> 
> >>
> 
> >> ==================================
> 
> >> path = "/home/nikos/public_html/data/files/"
> 
> >>
> 
> >> #read the containing folder and insert new filenames
> 
> >> for result in os.walk(path):
> 
> >
> 
> > You were told yesterday at least twice that os.walk returns a tuple but  
> 
> > you still insist on refusing to take any notice of our replies when it  
> 
> > suits you, preferring instead to waste everbody's time with these  
> 
> > questions.  Or are you trying to get into the Guinness Book of World  
> 
> > Records for the laziest bastard on the planet?
> 
> 
> 
> Hold on a sec ...
> 
> 
> 
> He has
> 
> 
> 
> for result in os.walk(path):
> 
> 	for filename in result[2]:
> 
> 
> 
> So he *did* take notice of that.
> 
> 
> 
> 
> 
> Nikos:
> 
> Expectation is to iterate through a tuple like this:
> 
> 
> 
> for dirpath, dirnames, filenames in os.walk(path):
> 
> 	...

Thank you Michael, yes i ahve understood that myself yesterday after one of the guys here have shown the output of os.walk(path) in IDLE. 
So, yes in fact i was in need for the 3rd item in the tuple, so to get hold on to the files.

Thank you for supporting me, some ppl here think i'am a troll and don't try thinks or ignore evrything but 'some_ppl's_opinion != True'

I'am just sometimes persistant on having thing my way that why i was calling my self Ferrous cRanus, which i changes it since it was annoying....

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.python

csiph-web