Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.python > #41525 > unrolled thread
| Started by | Bodhi <amitdev@gmail.com> |
|---|---|
| First post | 2013-03-19 08:47 -0700 |
| Last post | 2013-03-19 09:36 -0700 |
| Articles | 7 — 2 participants |
Back to article view | Back to comp.lang.python
Wierd behavior of gc.collect Bodhi <amitdev@gmail.com> - 2013-03-19 08:47 -0700
Re: Wierd behavior of gc.collect Dave Angel <davea@davea.name> - 2013-03-19 12:02 -0400
Re: Wierd behavior of gc.collect Bodhi <amitdev@gmail.com> - 2013-03-19 09:36 -0700
Re: Wierd behavior of gc.collect Dave Angel <davea@davea.name> - 2013-03-19 13:13 -0400
Re: Wierd behavior of gc.collect Bodhi <amitdev@gmail.com> - 2013-03-19 21:51 -0700
Re: Wierd behavior of gc.collect Bodhi <amitdev@gmail.com> - 2013-03-19 21:51 -0700
Re: Wierd behavior of gc.collect Bodhi <amitdev@gmail.com> - 2013-03-19 09:36 -0700
| From | Bodhi <amitdev@gmail.com> |
|---|---|
| Date | 2013-03-19 08:47 -0700 |
| Subject | Wierd behavior of gc.collect |
| Message-ID | <76cc4613-8086-4b36-a8bd-8ad0f21a19e0@googlegroups.com> |
I have a python process that does some operations and is supposed to release memory after those. The issue is that memory is not released (as seen through top). So I do a gc.collect() to see if there is any cycle etc. Immediately after doing the collect memory usage drops as expected, but strangely gc.collect() returns 0. This means I cannot find out what the problem is by setting the debug option on gc which is what I usually do to figure out issues like this. Maybe its that my understanding about it is incorrect, but if gc.collect returned 0, how come some memory was freed?
[toc] | [next] | [standalone]
| From | Dave Angel <davea@davea.name> |
|---|---|
| Date | 2013-03-19 12:02 -0400 |
| Message-ID | <mailman.3525.1363708963.2939.python-list@python.org> |
| In reply to | #41525 |
On 03/19/2013 11:47 AM, Bodhi wrote: > I have a python process that does some operations and is supposed to release memory after those. The issue is that memory is not released (as seen through top). So I do a gc.collect() to see if there is any cycle etc. Immediately after doing the collect memory usage drops as expected, but strangely gc.collect() returns 0. > This means I cannot find out what the problem is by setting the debug option on gc which is what I usually do to figure out issues like this. > > Maybe its that my understanding about it is incorrect, but if gc.collect returned 0, how come some memory was freed? > To put it simply, top won't in general show you that things are freed. The C libraries for malloc and free will reuse the memory, but not usually release it to the operating system. So it's not usually going to show up in 'top.' There was a long thread on this quite recently, but I can't seem to find it right now. -- DaveA
[toc] | [prev] | [next] | [standalone]
| From | Bodhi <amitdev@gmail.com> |
|---|---|
| Date | 2013-03-19 09:36 -0700 |
| Message-ID | <dd31732d-a26f-4a8e-b9d5-0e76bd4c3b55@googlegroups.com> |
| In reply to | #41532 |
I know this, but my question is what does gc.collect do which results in the c library to free memory? Usually it is because of unreferenced objects in a cycle or something, but here that doesn't seem to be the case. On Tuesday, March 19, 2013 9:32:27 PM UTC+5:30, Dave Angel wrote: > On 03/19/2013 11:47 AM, Bodhi wrote: > > > I have a python process that does some operations and is supposed to release memory after those. The issue is that memory is not released (as seen through top). So I do a gc.collect() to see if there is any cycle etc. Immediately after doing the collect memory usage drops as expected, but strangely gc.collect() returns 0. > > > This means I cannot find out what the problem is by setting the debug option on gc which is what I usually do to figure out issues like this. > > > > > > Maybe its that my understanding about it is incorrect, but if gc.collect returned 0, how come some memory was freed? > > > > > > > To put it simply, top won't in general show you that things are freed. > > The C libraries for malloc and free will reuse the memory, but not > > usually release it to the operating system. So it's not usually going > > to show up in 'top.' > > > > There was a long thread on this quite recently, but I can't seem to find > > it right now. > > > > > > -- > > DaveA
[toc] | [prev] | [next] | [standalone]
| From | Dave Angel <davea@davea.name> |
|---|---|
| Date | 2013-03-19 13:13 -0400 |
| Message-ID | <mailman.3529.1363713205.2939.python-list@python.org> |
| In reply to | #41535 |
On 03/19/2013 12:36 PM, Bodhi wrote: > I know this, but my question is what does gc.collect do which results in the c library to free memory? Usually it is because of unreferenced objects in a cycle or something, but here that doesn't seem to be the case. > As I said, python calls the C free() function, whether it's when an object's ref-count goes to zero, or whether it's during a gc call, where circular refs are freed. But free() does not necessarily release the memory to the OS. And the times it does depends on which C library is being used, and what OS it's running on. If the freed memory affects top in some situations, it's a C library detail. I've written a replacement C allocator in the past for Windows that used a different scheme for blocks over a certain threshold, and when those blocks were freed, it gave them back to the OS. But such blocks were multiples of 64k, which was the increment for VirtualAlloc. -- DaveA
[toc] | [prev] | [next] | [standalone]
| From | Bodhi <amitdev@gmail.com> |
|---|---|
| Date | 2013-03-19 21:51 -0700 |
| Message-ID | <c01b7af9-ccbc-43fc-9240-0e764c47a1cb@googlegroups.com> |
| In reply to | #41536 |
Thanks for the info. I now suspect that the free lists are taking up the memory which won't be released unless we do a collect. I'm verifying that. On Tuesday, March 19, 2013 10:43:11 PM UTC+5:30, Dave Angel wrote: > On 03/19/2013 12:36 PM, Bodhi wrote: > > > I know this, but my question is what does gc.collect do which results in the c library to free memory? Usually it is because of unreferenced objects in a cycle or something, but here that doesn't seem to be the case. > > > > > > > As I said, python calls the C free() function, whether it's when an > > object's ref-count goes to zero, or whether it's during a gc call, where > > circular refs are freed. > > > > But free() does not necessarily release the memory to the OS. And the > > times it does depends on which C library is being used, and what OS it's > > running on. > > > > If the freed memory affects top in some situations, it's a C library > > detail. I've written a replacement C allocator in the past for Windows > > that used a different scheme for blocks over a certain threshold, and > > when those blocks were freed, it gave them back to the OS. But such > > blocks were multiples of 64k, which was the increment for VirtualAlloc. > > > > > > > > > > -- > > DaveA
[toc] | [prev] | [next] | [standalone]
| From | Bodhi <amitdev@gmail.com> |
|---|---|
| Date | 2013-03-19 21:51 -0700 |
| Message-ID | <mailman.3545.1363755746.2939.python-list@python.org> |
| In reply to | #41536 |
Thanks for the info. I now suspect that the free lists are taking up the memory which won't be released unless we do a collect. I'm verifying that. On Tuesday, March 19, 2013 10:43:11 PM UTC+5:30, Dave Angel wrote: > On 03/19/2013 12:36 PM, Bodhi wrote: > > > I know this, but my question is what does gc.collect do which results in the c library to free memory? Usually it is because of unreferenced objects in a cycle or something, but here that doesn't seem to be the case. > > > > > > > As I said, python calls the C free() function, whether it's when an > > object's ref-count goes to zero, or whether it's during a gc call, where > > circular refs are freed. > > > > But free() does not necessarily release the memory to the OS. And the > > times it does depends on which C library is being used, and what OS it's > > running on. > > > > If the freed memory affects top in some situations, it's a C library > > detail. I've written a replacement C allocator in the past for Windows > > that used a different scheme for blocks over a certain threshold, and > > when those blocks were freed, it gave them back to the OS. But such > > blocks were multiples of 64k, which was the increment for VirtualAlloc. > > > > > > > > > > -- > > DaveA
[toc] | [prev] | [next] | [standalone]
| From | Bodhi <amitdev@gmail.com> |
|---|---|
| Date | 2013-03-19 09:36 -0700 |
| Message-ID | <mailman.3530.1363713588.2939.python-list@python.org> |
| In reply to | #41532 |
I know this, but my question is what does gc.collect do which results in the c library to free memory? Usually it is because of unreferenced objects in a cycle or something, but here that doesn't seem to be the case. On Tuesday, March 19, 2013 9:32:27 PM UTC+5:30, Dave Angel wrote: > On 03/19/2013 11:47 AM, Bodhi wrote: > > > I have a python process that does some operations and is supposed to release memory after those. The issue is that memory is not released (as seen through top). So I do a gc.collect() to see if there is any cycle etc. Immediately after doing the collect memory usage drops as expected, but strangely gc.collect() returns 0. > > > This means I cannot find out what the problem is by setting the debug option on gc which is what I usually do to figure out issues like this. > > > > > > Maybe its that my understanding about it is incorrect, but if gc.collect returned 0, how come some memory was freed? > > > > > > > To put it simply, top won't in general show you that things are freed. > > The C libraries for malloc and free will reuse the memory, but not > > usually release it to the operating system. So it's not usually going > > to show up in 'top.' > > > > There was a long thread on this quite recently, but I can't seem to find > > it right now. > > > > > > -- > > DaveA
[toc] | [prev] | [standalone]
Back to top | Article view | comp.lang.python
csiph-web