Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #40137 > unrolled thread

Speeding up Python's exit

Back to article view | Back to comp.lang.python

Contents

  Speeding up Python's exit Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-02-28 16:49 +0000
    Re: Speeding up Python's exit Neil Cerutti <neilc@norwich.edu> - 2013-02-28 17:02 +0000
    Re: Speeding up Python's exit Chris Angelico <rosuav@gmail.com> - 2013-03-01 04:06 +1100
    Re: Speeding up Python's exit Chris Angelico <rosuav@gmail.com> - 2013-03-01 04:33 +1100
    Re: Speeding up Python's exit Devin Jeanpierre <jeanpierreda@gmail.com> - 2013-02-28 12:33 -0500
    Re: Speeding up Python's exit Devin Jeanpierre <jeanpierreda@gmail.com> - 2013-02-28 12:31 -0500
    Re: Speeding up Python's exit Grant Edwards <invalid@invalid.invalid> - 2013-03-01 01:17 +0000
      Re: Speeding up Python's exit Antoine Pitrou <solipsis@pitrou.net> - 2013-03-01 19:10 +0000
      Re: Speeding up Python's exit Dave Angel <davea@davea.name> - 2013-03-01 14:17 -0500
      Re: Speeding up Python's exit Antoine Pitrou <solipsis@pitrou.net> - 2013-03-01 22:51 +0000
        Re: Speeding up Python's exit Ross Ridge <rridge@csclub.uwaterloo.ca> - 2013-03-03 18:27 -0500
      Re: Speeding up Python's exit Jason Swails <jason.swails@gmail.com> - 2013-03-01 21:51 -0500
    Re: Speeding up Python's exit Antoine Pitrou <solipsis@pitrou.net> - 2013-03-01 14:59 +0000

#40137 — Speeding up Python's exit

I just quit an interactive session using Python 2.7 on Linux. It took in 
excess of twelve minutes to exit, with the load average going well past 9 
for much of that time.

I think the reason it took so long was that Python was garbage-collecting 
a giant dict with 10 million entries, each one containing a list of the 
form [1, [2, 3], 4]. But still, that's terribly slow -- ironically, it 
took longer to dispose of the dict (12+ minutes) than it took to create 
it in the first place (approx 3 minutes, with a maximum load of 4). 

Can anyone explain why this was so painfully slow, and what (if anything) 
I can do to avoid it in the future?

I know there is a function os._exit which effectively kills the Python 
interpreter dead immediately, without doing any cleanup. What are the 
consequences of doing this? I assume that the memory used by the Python 
process will be reclaimed by the operating system, but other resources 
such as opened files may not be.


-- 
Steven

[toc] | [next] | [standalone]

#40138

On 2013-02-28, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
> Can anyone explain why this was so painfully slow, and what (if
> anything) I can do to avoid it in the future?

I think your explanation makes sense. Maybe the nested nature of
the strings was causing it to churn looking for circular
references?

Disabling gc before exiting might do the trick, assuming you're
assiduously managing other resources with context managers.

gc.disable()
exit()


-- 
Neil Cerutti

[toc] | [prev] | [next] | [standalone]

#40140

On Fri, Mar 1, 2013 at 3:49 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> I think the reason it took so long was that Python was garbage-collecting
> a giant dict with 10 million entries, each one containing a list of the
> form [1, [2, 3], 4]. But still, that's terribly slow -- ironically, it
> took longer to dispose of the dict (12+ minutes) than it took to create
> it in the first place (approx 3 minutes, with a maximum load of 4).

Leaving the question of just *why* you have so much in your dict.....
but anyway.

Is it any different if you create a deliberate reference loop and then
stuff it into some module somewhere? That would force it to be kept
until interpreter shutdown, and then a cyclic garbage collection after
that, which quite probably would be never run. A stupid trick,
perhaps, but it might work; I tested it with a dummy class with a
__del__ method and it wasn't called. Putting it into some other module
may not be necessary, but I don't know what happens with the
interactive interpreter and what gets freed up when.

ChrisA

[toc] | [prev] | [next] | [standalone]

#40144

On Fri, Mar 1, 2013 at 4:31 AM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
> On Thu, Feb 28, 2013 at 12:06 PM, Chris Angelico <rosuav@gmail.com> wrote:
>> Is it any different if you create a deliberate reference loop and then
>> stuff it into some module somewhere? That would force it to be kept
>> until interpreter shutdown, and then a cyclic garbage collection after
>> that, which quite probably would be never run. A stupid trick,
>> perhaps, but it might work; I tested it with a dummy class with a
>> __del__ method and it wasn't called. Putting it into some other module
>> may not be necessary, but I don't know what happens with the
>> interactive interpreter and what gets freed up when.
>
> __del__ is never called for cyclic references.

D'oh. Test is flawed, then. But is the theory plausible? That the
cycle detector won't be called on exit after other modules get freed?

ChrisA

[toc] | [prev] | [next] | [standalone]

#40145

On Thu, Feb 28, 2013 at 12:31 PM, Devin Jeanpierre
<jeanpierreda@gmail.com> wrote:
> On Thu, Feb 28, 2013 at 12:06 PM, Chris Angelico <rosuav@gmail.com> wrote:
>> Is it any different if you create a deliberate reference loop and then
>> stuff it into some module somewhere? That would force it to be kept
>> until interpreter shutdown, and then a cyclic garbage collection after
>> that, which quite probably would be never run. A stupid trick,
>> perhaps, but it might work; I tested it with a dummy class with a
>> __del__ method and it wasn't called. Putting it into some other module
>> may not be necessary, but I don't know what happens with the
>> interactive interpreter and what gets freed up when.
>
> __del__ is never called for cyclic references.

Sorry, I posted too early. Not only is __del__ never called, but
__del__ is the reason the cycles aren't collected. I don't know if
your trick will work without __del__.

-- Devin

[toc] | [prev] | [next] | [standalone]

#40146

On Thu, Feb 28, 2013 at 12:06 PM, Chris Angelico <rosuav@gmail.com> wrote:
> Is it any different if you create a deliberate reference loop and then
> stuff it into some module somewhere? That would force it to be kept
> until interpreter shutdown, and then a cyclic garbage collection after
> that, which quite probably would be never run. A stupid trick,
> perhaps, but it might work; I tested it with a dummy class with a
> __del__ method and it wasn't called. Putting it into some other module
> may not be necessary, but I don't know what happens with the
> interactive interpreter and what gets freed up when.

__del__ is never called for cyclic references.

-- Devin

[toc] | [prev] | [next] | [standalone]

#40208

On 2013-02-28, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:

> I know there is a function os._exit which effectively kills the
> Python interpreter dead immediately, without doing any cleanup. What
> are the consequences of doing this?

You loose any data you haven't saved to disk.

> I assume that the memory used by the Python process will be reclaimed
> by the operating system, but other resources such as opened files may
> not be.

All open files (including sockets, pipes, serial ports, etc) will be
flushed (from an OS standpoint) and closed.  If you've closed all the
files you've written to, there should be no danger in just pulling the
plug.

-- 
Grant

[toc] | [prev] | [next] | [standalone]

#40282

Grant Edwards <invalid <at> invalid.invalid> writes:
> 
> > I assume that the memory used by the Python process will be reclaimed
> > by the operating system, but other resources such as opened files may
> > not be.
> 
> All open files (including sockets, pipes, serial ports, etc) will be
> flushed (from an OS standpoint) and closed.

According to POSIX, no, open files will not be flushed:

“The _Exit() and _exit() functions shall not call functions registered with 
atexit() nor any registered signal handlers. Open streams shall not be flushed. 
Whether open streams are closed (without flushing) is implementation-defined.”

http://pubs.opengroup.org/onlinepubs/9699919799/functions/_exit.html

(under the hood, os._exit() calls C _exit())

Regards

Antoine.

[toc] | [prev] | [next] | [standalone]

#40283

On 03/01/2013 02:10 PM, Antoine Pitrou wrote:
> Grant Edwards <invalid <at> invalid.invalid> writes:
>>  <snip>
>> All open files (including sockets, pipes, serial ports, etc) will be
>> flushed (from an OS standpoint) and closed.
>
> According to POSIX, no, open files will not be flushed:
>
> “The _Exit() and _exit() functions shall not call functions registered with
> atexit() nor any registered signal handlers. Open streams shall not be flushed.
> Whether open streams are closed (without flushing) is implementation-defined.”
>

Note he didn't say the python buffers would be flushed.  It's the OS 
buffers that are flushed.


-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#40300

Dave Angel <davea <at> davea.name> writes:
> 
> Note he didn't say the python buffers would be flushed.  It's the OS 
> buffers that are flushed.

Now please read my message again. The OS buffers are *not* flushed according
to POSIX.

[toc] | [prev] | [next] | [standalone]

#40428

Antoine Pitrou  <solipsis@pitrou.net> wrote:
>Now please read my message again. The OS buffers are *not* flushed according
>to POSIX.

POSIX says open *streams* might not be flushed.  POSIX streams are C
FILE * streams and generally aren't regarded as being part of the OS.

When you call os._exit() in a Python program any unwritten data still
in Python's own file buffers will be lost.  Any unwritten data still
in the C library's FILE * buffers will be lost.  Any data successfuly
written through a POSIX file descriptor (eg. using the write() function)
will not be lost becasue os._exit() was used.

Note that this doesn't mean that OS buffers will flushed when os._exit()
is called.  Data that hasn't yet been physically written to disk, hasn't
be successfully transmitted over the network, or otherwise hasn't been
fully comitted could still be lost.  However, exiting Python normally
doesn't change this.  Only the Python process's own internal buffers are
flushed, the OS doesn't change its handling of its buffers.  If you want
written data to be fully committed before exiting you need to use other
OS services that guarantee this.

					Ross Ridge

-- 
 l/  //	  Ross Ridge -- The Great HTMU
[oo][oo]  rridge@csclub.uwaterloo.ca
-()-/()/  http://www.csclub.uwaterloo.ca/~rridge/ 
 db  //	  

[toc] | [prev] | [next] | [standalone]

#40313

[Multipart message — attachments visible in raw view] — view raw

On Fri, Mar 1, 2013 at 5:51 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:

> Dave Angel <davea <at> davea.name> writes:
> >
> > Note he didn't say the python buffers would be flushed.  It's the OS
> > buffers that are flushed.
>
> Now please read my message again. The OS buffers are *not* flushed
> according
> to POSIX.
>

I have observed this behavior on some Linux systems with a Fortran program
that terminated abnormally (via a kill signal).  Other Linux systems I've
used appear to flush their file buffers to disk in the event of a kill
signal, it really depends on the system.

If a file object's destructor is not called when the Python interpreter
exits and it's up to the OS to flush the file buffers to disk, you can't be
sure that it will do so.  And as Antoine pointed out, POSIX standard
doesn't require that they do.

All the best,
Jason

[toc] | [prev] | [next] | [standalone]

#40261

Steven D'Aprano <steve+comp.lang.python <at> pearwood.info> writes:
> 
> I just quit an interactive session using Python 2.7 on Linux. It took in 
> excess of twelve minutes to exit, with the load average going well past 9 
> for much of that time.
> 
> I think the reason it took so long was that Python was garbage-collecting 
> a giant dict with 10 million entries, each one containing a list of the 
> form [1, [2, 3], 4]. But still, that's terribly slow -- ironically, it 
> took longer to dispose of the dict (12+ minutes) than it took to create 
> it in the first place (approx 3 minutes, with a maximum load of 4). 
> 
> Can anyone explain why this was so painfully slow, and what (if anything) 
> I can do to avoid it in the future?

You are basically asking people to guess where your performance problem
comes from, without even providing a snippet so that people can reproduce ;)

> I know there is a function os._exit which effectively kills the Python 
> interpreter dead immediately, without doing any cleanup. What are the 
> consequences of doing this? I assume that the memory used by the Python 
> process will be reclaimed by the operating system, but other resources 
> such as opened files may not be.

The OS always disposes of per-process resources when the process terminates
(except if the OS is buggy ;-)). However, file buffers will not be flushed,
atexit handlers and other destructors will not be called, database
transactions will be abandoned (rolled back), etc.

Regards

Antoine.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web