Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #70268

Re: MemoryError in data conversion

Show key headers only | View raw

Gregory Ewing wrote:

> Mok-Kong Shen wrote:
>> I have yet a question out of curiosity: Why is my 2nd list structure,
>> that apparently is too complex for handling by eval and json, seemingly
>> not a problem for pickle?
> 
> Pickle is intended for arbitrary data structures, so it
> is designed to be able to handle deeply-nested and/or
> recursive data. Eval only has to handle nesting to depths
> likely to be encountered in source code. Apparently the
> json parser also assumes you're not going to be using
> very deep nesting.

But pickle does have the same limitation:

>>> def check(load, dump):
...     items = []
...     try:
...             for i in range(10**6):
...                     assert load(dump(items)) == items
...                     items = [items]
...     except RuntimeError:
...             return i
... 
>>> check(json.loads, json.dumps)
994
>>> check(pickle.loads, pickle.dumps)
499

Mok-Kong Shen, for pickle and json you can increase the limit a bit: 

>>> import sys
>>> sys.setrecursionlimit(2000)
>>> check(json.loads, json.dumps)
1994
>>> check(pickle.loads, pickle.dumps)
999

But be careful, if you choose the limit too high you'll see Python react 
like any other C program:

>>> sys.setrecursionlimit(100000)
>>> items = []
>>> for i in range(100000):
...     items = [items]
... 
>>> s = pickle.dumps(items)
Segmentation fault

For literal_eval() the limit is unfortunately hard-coded in the C source.

Mok-Kong Shen wrote:

> What I need is to have my (complicated) list to be put into
> a bytearray, do some proceesing, transfer it to the recipient.
> The recipient reverses the processing, obtains a bytearray
> that is the same as my original one and gets from it the same
> list as mine. Using pickle I can manage to do it (in fact I
> did try and succeed with my 2nd list), but that's IMHO a
> rather unnatural/awkward work-around. (It means that I have
> to pickle out the list to a file and read in the content of
> the file in order to have it as a bytearray etc. etc.)
 
The limit has been increased before, see

http://bugs.python.org/issue1881

and maybe you can get the core developers to increase it again, but 
generally speaking the existence of a recursion limit is the price you pay 
for easy interfacing with C. 

literal_eval() could allocate its stack dynamically, but that would probably 
complicate the code so that return on investment is questionable.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

MemoryError in data conversion Mok-Kong Shen <mok-kong.shen@t-online.de> - 2014-04-14 03:46 +0200
  Re: MemoryError in data conversion dieter <dieter@handshake.de> - 2014-04-14 08:14 +0200
  Re: MemoryError in data conversion Peter Otten <__peter__@web.de> - 2014-04-14 09:46 +0200
    Re: MemoryError in data conversion Mok-Kong Shen <mok-kong.shen@t-online.de> - 2014-04-14 14:26 +0200
      Re: MemoryError in data conversion Peter Otten <__peter__@web.de> - 2014-04-14 15:59 +0200
        Re: MemoryError in data conversion Mok-Kong Shen <mok-kong.shen@t-online.de> - 2014-04-14 23:20 +0200
          Re: MemoryError in data conversion Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-04-15 11:51 +1200
            Re: MemoryError in data conversion Mok-Kong Shen <mok-kong.shen@t-online.de> - 2014-04-15 10:55 +0200
              Re: MemoryError in data conversion Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-04-16 00:08 +1200
            Re: MemoryError in data conversion Peter Otten <__peter__@web.de> - 2014-04-15 11:59 +0200
  Re:MemoryError in data conversion Dave Angel <davea@davea.name> - 2014-04-14 08:42 -0400

csiph-web