Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed1a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Peter Otten <__peter__@web.de>
Subject: Re: MemoryError in data conversion
Date: Tue, 15 Apr 2014 11:59 +0200
Organization: None
References: <lifelc$ccj$1@news.albasani.net> <mailman.9239.1397461622.18130.python-list@python.org> <ligk5n$3jp$1@news.albasani.net> <mailman.9250.1397484008.18130.python-list@python.org> <lihjf0$4t6$1@news.albasani.net> <br3ajeFdnj3U1@mid.individual.net>
Mime-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7Bit
User-Agent: KNode/4.11.5
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.9278.1397555958.18130.python-list@python.org>
Lines: 76
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:70268

Gregory Ewing wrote:

> Mok-Kong Shen wrote:
>> I have yet a question out of curiosity: Why is my 2nd list structure,
>> that apparently is too complex for handling by eval and json, seemingly
>> not a problem for pickle?
> 
> Pickle is intended for arbitrary data structures, so it
> is designed to be able to handle deeply-nested and/or
> recursive data. Eval only has to handle nesting to depths
> likely to be encountered in source code. Apparently the
> json parser also assumes you're not going to be using
> very deep nesting.

But pickle does have the same limitation:

>>> def check(load, dump):
...     items = []
...     try:
...             for i in range(10**6):
...                     assert load(dump(items)) == items
...                     items = [items]
...     except RuntimeError:
...             return i
... 
>>> check(json.loads, json.dumps)
994
>>> check(pickle.loads, pickle.dumps)
499

Mok-Kong Shen, for pickle and json you can increase the limit a bit: 

>>> import sys
>>> sys.setrecursionlimit(2000)
>>> check(json.loads, json.dumps)
1994
>>> check(pickle.loads, pickle.dumps)
999

But be careful, if you choose the limit too high you'll see Python react 
like any other C program:

>>> sys.setrecursionlimit(100000)
>>> items = []
>>> for i in range(100000):
...     items = [items]
... 
>>> s = pickle.dumps(items)
Segmentation fault

For literal_eval() the limit is unfortunately hard-coded in the C source.

Mok-Kong Shen wrote:

> What I need is to have my (complicated) list to be put into
> a bytearray, do some proceesing, transfer it to the recipient.
> The recipient reverses the processing, obtains a bytearray
> that is the same as my original one and gets from it the same
> list as mine. Using pickle I can manage to do it (in fact I
> did try and succeed with my 2nd list), but that's IMHO a
> rather unnatural/awkward work-around. (It means that I have
> to pickle out the list to a file and read in the content of
> the file in order to have it as a bytearray etc. etc.)
 
The limit has been increased before, see

http://bugs.python.org/issue1881

and maybe you can get the core developers to increase it again, but 
generally speaking the existence of a recursion limit is the price you pay 
for easy interfacing with C. 

literal_eval() could allocate its stack dynamically, but that would probably 
complicate the code so that return on investment is questionable.