Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #48255

Re: Memory usage steadily going up while pickling objects

Show all headers | View raw

Giorgos Tzampanakis wrote:

> I have a program that saves lots (about 800k) objects into a shelve
> database (I'm using sqlite3dbm for this since all the default python dbm
> packages seem to be unreliable and effectively unusable, but this is
> another discussion).
> 
> The process takes about 10-15 minutes. During that time I see memory usage
> steadily rising, sometimes resulting in a MemoryError. Now, there is a
> chance that my code is keeping unneeded references to the stored objects,
> but I have debugged it thoroughly and haven't found any.
> 
> So I'm beginning to suspect that the pickle module might be keeping an
> internal cache of objects being pickled. Is this true?

Pickler/Unpickler objects use a cache to maintain object identity, but at 
least shelve in the standard library uses a new Pickler/Unpickler for each 
set/get operation. 

I don't have sqlite3dbm, but you can try the following:

>>> import shelve
>>> class A: pass
... 
>>> a = A()
>>> s = shelve.open("tmp.shelve")
>>> s["x"] = s["y"] = a
>>> s["x"] is s["y"]
False

If you are getting True there must be a cache. One way to enable a cache 
yourself is writeback:

>>> s = shelve.open("tmp.shelve", writeback=True)
>>> s["x"] = s["y"] = a
>>> s["x"] is s["y"]
True

You didn't do that, I guess?

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Memory usage steadily going up while pickling objects Giorgos Tzampanakis <giorgos.tzampanakis@gmail.com> - 2013-06-14 23:04 +0000
  Re: Memory usage steadily going up while pickling objects Dave Angel <davea@davea.name> - 2013-06-14 21:52 -0400
    Re: Memory usage steadily going up while pickling objects Giorgos Tzampanakis <giorgos.tzampanakis@gmail.com> - 2013-06-15 09:41 +0000
      Re: Memory usage steadily going up while pickling objects Peter Otten <__peter__@web.de> - 2013-06-15 12:18 +0200
        Re: Memory usage steadily going up while pickling objects Giorgos Tzampanakis <giorgos.tzampanakis@gmail.com> - 2013-06-15 11:06 +0000
      Re: Memory usage steadily going up while pickling objects dieter <dieter@handshake.de> - 2013-06-16 08:26 +0200
  Re: Memory usage steadily going up while pickling objects Peter Otten <__peter__@web.de> - 2013-06-15 08:37 +0200
    Re: Memory usage steadily going up while pickling objects Giorgos Tzampanakis <giorgos.tzampanakis@gmail.com> - 2013-06-15 09:37 +0000

csiph-web