Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #12468 > unrolled thread

PC locks up with list operations

Back to article view | Back to comp.lang.python

Contents

  PC locks up with list operations Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-08-31 22:33 +1000
    Re: PC locks up with list operations Chris Withers <chris@simplistix.co.uk> - 2011-08-31 13:40 +0100
      Re: PC locks up with list operations Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-08-31 22:47 +1000
        Re: PC locks up with list operations Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2011-09-01 11:31 +1200
          Re: PC locks up with list operations Tim Chase <python.list@tim.thechases.com> - 2011-08-31 19:53 -0500
        Re: PC locks up with list operations Nobody <nobody@nowhere.com> - 2011-09-12 13:19 +0100
      Re: PC locks up with list operations Robert Spanjaard <spamtrap@arumes.com> - 2011-08-31 16:06 +0200
      Re: PC locks up with list operations John Nagle <nagle@animats.com> - 2011-09-15 12:27 -0700
    Re: PC locks up with list operations Benjamin Kaplan <benjamin.kaplan@case.edu> - 2011-08-31 08:49 -0400
      Re: PC locks up with list operations Carl Banks <pavlovevidence@gmail.com> - 2011-09-13 06:31 -0700
    Re: PC locks up with list operations Peter Otten <__peter__@web.de> - 2011-08-31 14:56 +0200
    Re: PC locks up with list operations Rodrick Brown <rodrick.brown@gmail.com> - 2011-08-31 10:13 -0400
    Re: PC locks up with list operations Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-09-12 18:44 +1000
      Re: PC locks up with list operations Roy Smith <roy@panix.com> - 2011-09-12 07:40 -0400
        Re: PC locks up with list operations Terry Reedy <tjreedy@udel.edu> - 2011-09-12 16:53 -0400

#12468 — PC locks up with list operations

Twice in a couple of weeks, I have locked up my PC by running a Python 2.5
script that tries to create a list that is insanely too big.

In the first case, I (stupidly) did something like:

mylist = [0]*12345678901234

After leaving the machine for THREE DAYS (!!!) I eventually was able to get
to a console and kill the Python process. Amazingly, it never raised
MemoryError in that time.

The second time was a little less stupid, but not much:

mylist = []
for x in itertools.combinations_with_replacement(some_big_list, 20):
    mylist.append(func(x))

After three hours, the desktop is still locked up. I'm waiting to see what
happens in the morning before rebooting.

Apart from "Then don't do that!", is there anything I can do to prevent this
sort of thing in the future? Like instruct Python not to request more
memory than my PC has?

I am using Linux desktops; both incidents were with Python 2.5. Do newer
versions of Python respond to this sort of situation more gracefully?



-- 
Steven

[toc] | [next] | [standalone]

#12470

On 31/08/2011 13:33, Steven D'Aprano wrote:
> I am using Linux desktops; both incidents were with Python 2.5. Do newer
> versions of Python respond to this sort of situation more gracefully?

Ironically, Windows does better here and dumps you out with a 
MemoryError before slowly recovering.

Linux seems to fair badly when programs use more memory than physically 
available. Perhaps there's some per-process thing that can be used to 
limit things on Linux?

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

[toc] | [prev] | [next] | [standalone]

#12471

Chris Withers wrote:

> On 31/08/2011 13:33, Steven D'Aprano wrote:
>> I am using Linux desktops; both incidents were with Python 2.5. Do newer
>> versions of Python respond to this sort of situation more gracefully?
> 
> Ironically, Windows does better here and dumps you out with a
> MemoryError before slowly recovering.
> 
> Linux seems to fair badly when programs use more memory than physically
> available. Perhaps there's some per-process thing that can be used to
> limit things on Linux?

As far as I know, ulimit ("user limit") won't help. It can limit the amount
of RAM available to a process, but that just makes the process start using
virtual memory more quickly. It can also limit the amount of virtual memory
used by the shell, but not of other processes. In other words, Linux will
try really, really, really hard to give you the 84 gigabytes you've asked
for on a 2 GB system, even if it means DOSing your system for a month.

Of course, I would be very happy to learn I'm wrong.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#12532

Steven D'Aprano wrote:

> As far as I know, ulimit ("user limit") won't help. It can limit the amount
> of RAM available to a process, but that just makes the process start using
> virtual memory more quickly.

ulimit -v is supposed to set the maximum amount of virtual
memory the process can use.

> It can also limit the amount of virtual memory
> used by the shell, but not of other processes.

That doesn't sound right. Not sure about Linux, but the
man page for sh on Darwin says:

     Provides  control  over the resources available to the shell and
     to processes started by it, on systems that allow such  control.

The Python process should also be able to set its own
limits using resource.setrlimit().

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#12538

On 08/31/11 18:31, Gregory Ewing wrote:
> The Python process should also be able to set its own
> limits using resource.setrlimit().

A new corner of stdlib that I've never poked at.  Thanks for the 
suggestion.  Disappointed though that it doesn't seem to have 
docstrings on the functions, so I had to wade back out to the 
online docs to probe at it.  Granted, after the fact, they were 
pretty obvious, but it would be nice if 
"help(resource.getrlimit)" gave me a hint as to what that one 
expected parameter should have been.

-tim

   import resource as r
   token = "RLIMIT_"
   for item in dir(r):
     if item.startswith(token):
       print "%s:" % item[len(token):],
       print "%s hard/%s soft" % r.getrlimit(getattr(r, item))

[toc] | [prev] | [next] | [standalone]

#13173

On Wed, 31 Aug 2011 22:47:59 +1000, Steven D'Aprano wrote:

>> Linux seems to fair badly when programs use more memory than physically
>> available. Perhaps there's some per-process thing that can be used to
>> limit things on Linux?
> 
> As far as I know, ulimit ("user limit") won't help. It can limit the amount
> of RAM available to a process, but that just makes the process start using
> virtual memory more quickly. It can also limit the amount of virtual memory
> used by the shell, but not of other processes. In other words, Linux will
> try really, really, really hard to give you the 84 gigabytes you've asked
> for on a 2 GB system, even if it means DOSing your system for a month.
> 
> Of course, I would be very happy to learn I'm wrong.

Resource limits set by ulimit are inherited by any processes spawned from
the shell. They also affect the shell itself, but a shell process
shouldn't require a lot of resources. You can use a subshell if you want
to impose limits on a specific process.

For Python, setting the limit on virtual memory (RAM + swap) to no more
than the amount of physical RAM is probably a wise move. Some processes
can use swap effectively, but the typical Python program probably can't.
There are exceptions, e.g. if most of the memory is accounted for by large
NumPy arrays and you're careful about the operations which are performed
upon them. But using large amounts of memory for many small objects is
almost bound to result in swap-thrashing.

One problem with doing this automatically (e.g. in .pythonrc) is the
inheritance issue; any processes spawned from the interpreter will also be
resource limited. Similarly, any binary libraries loaded into the
interpreter will be subject to the process' resource limits. Consequently,
there would be some advantages to the Python interpreter having its own
resource-limiting mechanism.

[toc] | [prev] | [next] | [standalone]

#12476

On 08/31/2011 02:40 PM, Chris Withers wrote:
> On 31/08/2011 13:33, Steven D'Aprano wrote:
>> I am using Linux desktops; both incidents were with Python 2.5. Do newer
>> versions of Python respond to this sort of situation more gracefully?
>
> Ironically, Windows does better here and dumps you out with a
> MemoryError before slowly recovering.

I think it's a little premature to make such a statement based on a single 
user experience. I've used Linux for six years now, and it NEVER locked up, 
even when a program leaks memory like hell.
I can't duplicate the OP's behaviour because my Python (2.6.5, 64 bit) does 
generate an instant MemoryError (which answers the question, ofcourse), but 
I've used a VLC version that had a nasty leak. My swap space is on a SSD, 
and VLC filled it completely in about 10 seconds. But then, VLC got killed 
automatically, and the system recovered gracefully.
On a system with the swapspace on a regular HD, the same will happen, but 
it'll take more time. And while VLC is filling the swap space, the system 
does not lock up at all. You can still keep working, and (for example) fire 
up the process manager and kill VLC yourself.

 >>> mylist = [0]*12345678901234
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
MemoryError

-- 
Regards, Robert                                      http://www.arumes.com

[toc] | [prev] | [next] | [standalone]

#13347

On 8/31/2011 5:40 AM, Chris Withers wrote:
> On 31/08/2011 13:33, Steven D'Aprano wrote:
>> I am using Linux desktops; both incidents were with Python 2.5. Do newer
>> versions of Python respond to this sort of situation more gracefully?
>
> Ironically, Windows does better here and dumps you out with a
> MemoryError before slowly recovering.
>
> Linux seems to fair badly when programs use more memory than physically
> available.

    Linux virtual memory has some known design problems.  For example,
you can crash many Linux systems with a program which opens large
files and accesses them randomly, combined with another program which
is spawning processes that need a fair amount of memory.  The disk
caching system tries to use all unused memory, and when process
startup is consuming pages, it's possible to get into a situation
where a page is needed, the task requesting it can't block, and
a lock preventing freeing a file cache page is set.

    Arguably, paging to disk is obsolete. RAM is too cheap and
disk is too slow.

					John Nagle

[toc] | [prev] | [next] | [standalone]

#12472

On Wed, Aug 31, 2011 at 8:40 AM, Chris Withers <chris@simplistix.co.uk> wrote:
>
> On 31/08/2011 13:33, Steven D'Aprano wrote:
>>
>> I am using Linux desktops; both incidents were with Python 2.5. Do newer
>> versions of Python respond to this sort of situation more gracefully?
>
> Ironically, Windows does better here and dumps you out with a MemoryError before slowly recovering.
>
> Linux seems to fair badly when programs use more memory than physically available. Perhaps there's some per-process thing that can be used to limit things on Linux?
>
> cheers,
>
> Chris
>
> --

32-bit or 64-bit Python? A 32-bit program will crash once memory hits
2GB. A 64-bit program will just keep consuming RAM until your computer
starts thrashing. The problem isn't your program using more RAM than
you have, just more RAM than you have free. Last time I faced a
situation like this, I just decided it was better to stick to the
32-bit program and let it crash if it got too big.


> Simplistix - Content Management, Batch Processing & Python Consulting
>            - http://www.simplistix.co.uk
> --
> http://mail.python.org/mailman/listinfo/python-list

[toc] | [prev] | [next] | [standalone]

#13230

On Wednesday, August 31, 2011 5:49:24 AM UTC-7, Benjamin Kaplan wrote:
> 32-bit or 64-bit Python? A 32-bit program will crash once memory hits
> 2GB. A 64-bit program will just keep consuming RAM until your computer
> starts thrashing. The problem isn't your program using more RAM than
> you have, just more RAM than you have free. Last time I faced a
> situation like this, I just decided it was better to stick to the
> 32-bit program and let it crash if it got too big.

On my 64-bit Linux system, I got a memory error in under a second, no thrashing.

I have no swap.  It's overrated.


Carl Banks

[toc] | [prev] | [next] | [standalone]

#12473

Steven D'Aprano wrote:

> Twice in a couple of weeks, I have locked up my PC by running a Python 2.5
> script that tries to create a list that is insanely too big.
> 
> In the first case, I (stupidly) did something like:
> 
> mylist = [0]*12345678901234
> 
> After leaving the machine for THREE DAYS (!!!) I eventually was able to
> get to a console and kill the Python process. Amazingly, it never raised
> MemoryError in that time.
> 
> The second time was a little less stupid, but not much:
> 
> mylist = []
> for x in itertools.combinations_with_replacement(some_big_list, 20):
>     mylist.append(func(x))
> 
> After three hours, the desktop is still locked up. I'm waiting to see what
> happens in the morning before rebooting.
> 
> Apart from "Then don't do that!", is there anything I can do to prevent
> this sort of thing in the future? Like instruct Python not to request more
> memory than my PC has?
> 
> I am using Linux desktops; both incidents were with Python 2.5. Do newer
> versions of Python respond to this sort of situation more gracefully?

If you are starting these scripts from the shell, how about ulimit?

$ ulimit -v 40000
$ python -c'print range(10**5)[-1]'
99999
$ python -c'print range(10**6)[-1]'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
MemoryError
$

[toc] | [prev] | [next] | [standalone]

#12477

$ man limits.conf 

Sent from my iPhone

On Aug 31, 2011, at 8:33 AM, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:

> Twice in a couple of weeks, I have locked up my PC by running a Python 2.5
> script that tries to create a list that is insanely too big.
> 
> In the first case, I (stupidly) did something like:
> 
> mylist = [0]*12345678901234
> 
> After leaving the machine for THREE DAYS (!!!) I eventually was able to get
> to a console and kill the Python process. Amazingly, it never raised
> MemoryError in that time.
> 
> The second time was a little less stupid, but not much:
> 
> mylist = []
> for x in itertools.combinations_with_replacement(some_big_list, 20):
>    mylist.append(func(x))
> 
> After three hours, the desktop is still locked up. I'm waiting to see what
> happens in the morning before rebooting.
> 
> Apart from "Then don't do that!", is there anything I can do to prevent this
> sort of thing in the future? Like instruct Python not to request more
> memory than my PC has?
> 
> I am using Linux desktops; both incidents were with Python 2.5. Do newer
> versions of Python respond to this sort of situation more gracefully?
> 
> 
> 
> -- 
> Steven
> 
> -- 
> http://mail.python.org/mailman/listinfo/python-list

[toc] | [prev] | [next] | [standalone]

#13163

On Wed, 31 Aug 2011 10:33 pm Steven D'Aprano wrote:

> Twice in a couple of weeks, I have locked up my PC by running a Python 2.5
> script that tries to create a list that is insanely too big.
> 
> In the first case, I (stupidly) did something like:
> 
> mylist = [0]*12345678901234
[...]
> Apart from "Then don't do that!", is there anything I can do to prevent
> this sort of thing in the future? Like instruct Python not to request more
> memory than my PC has?


For anyone who may care, I can report that ulimit under Linux will help with
this situation.

[steve@wow-wow ~]$ ulimit -v 200000
[steve@wow-wow ~]$ python
Python 2.5 (r25:51908, Nov  6 2007, 16:54:01)
[GCC 4.1.2 20070925 (Red Hat 4.1.2-27)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> L = range(100000000)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError

(Of course, I would have eventually got a MemoryError even without the
ulimit. *Eventually.* After much thrashing and processes crashing and
pain.)


Does anyone else think it would be useful for Python's memory manager to
enforce user-settable limits? Even just a simple thing like "never try to
allocate more than N bytes at once" would probably go a long way to prevent
a certain class of accidental (or deliberate) DOSes.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#13169

In article <4e6dc66e$0$29986$c3e8da3$5496439d@news.astraweb.com>,
 Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:

> > mylist = [0]*12345678901234
> [...]
> > Apart from "Then don't do that!", is there anything I can do to prevent
> > this sort of thing in the future? Like instruct Python not to request more
> > memory than my PC has?
> 
> 
> For anyone who may care, I can report that ulimit under Linux will help with
> this situation.
> [...]
> Does anyone else think it would be useful for Python's memory manager to
> enforce user-settable limits?

Not me.  You've already discovered that ulimit does exactly what you 
want.  Why would be gained by having Python duplicate this functionality?

[toc] | [prev] | [next] | [standalone]

#13199

On 9/12/2011 7:40 AM, Roy Smith wrote:
> In article<4e6dc66e$0$29986$c3e8da3$5496439d@news.astraweb.com>,
>   Steven D'Aprano<steve+comp.lang.python@pearwood.info>  wrote:
>
>>> mylist = [0]*12345678901234
>> [...]
>>> Apart from "Then don't do that!", is there anything I can do to prevent
>>> this sort of thing in the future? Like instruct Python not to request more
>>> memory than my PC has?
>>
>>
>> For anyone who may care, I can report that ulimit under Linux will help with
>> this situation.
>> [...]
>> Does anyone else think it would be useful for Python's memory manager to
>> enforce user-settable limits?
>
> Not me.  You've already discovered that ulimit does exactly what you
> want.  Why would be gained by having Python duplicate this functionality?

Having it on Windows.

-- 
Terry Jan Reedy

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web