Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #83708 > unrolled thread

Re: Track down SIGABRT

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Track down SIGABRT Israel Brewster <israel@ravnalaska.net> - 2015-01-13 08:05 -0900
    Re: Track down SIGABRT Paul Rubin <no.email@nospam.invalid> - 2015-01-13 09:31 -0800
      Re: Track down SIGABRT Israel Brewster <israel@ravnalaska.net> - 2015-01-23 11:19 -0900
      Re: Track down SIGABRT dieter <dieter@handshake.de> - 2015-01-26 08:17 +0100

#83708 — Re: Track down SIGABRT

[Multipart message — attachments visible in raw view] — view raw

On Jan 13, 2015, at 6:27 AM, William Ray Wing <wrw@mac.com> wrote:

> 
>> On Jan 9, 2015, at 12:40 PM, Israel Brewster <israel@ravnalaska.net> wrote:
>> 
>> I have a long-running python/CherryPy Web App server process that I am running on Mac OS X 10.8.5. Python 2.7.2 running in 32-bit mode (for now, I have the code in place to change over to 64 bit, but need to schedule the downtime to do it). On the 6th of this month, during normal operation from what I can tell, and after around 33 days of trouble-free uptime, the python process crashed with a SIGABRT. I restarted the process, and everything looked good again until yesterday, when it again crashed with a SIGABRT. The crash dump the system gave me doesn't tell me much, other than that it looks like python is calling some C function when it crashes. I've attached the crash report, in case it can mean something more to someone else.
>> 
>> Can anyone give me some hints as to how to track down the cause of this crash? It's especially problematic since I can't mess with the live server for testing, and it is quite a while between crashes, making it difficult, if not impossible, to reproduce in testing. Thanks.
>> -----------------------------------------------
>> Israel Brewster
>> Systems Analyst II
>> Ravn Alaska
>> 5245 Airport Industrial Rd
>> Fairbanks, AK 99709
>> (907) 450-7293
>> -----------------------------------------------
> 
> 
> Can you run the application in an IDE?

Yes - I run it through Wing during development. I don't think that would be such a good option for my production machine, however. If it gets really bad I'll consider it though - that should at least tell me where it is crashing.

> 
> -Bill

-----------------------------------------------
Israel Brewster
Systems Analyst II
Ravn Alaska
5245 Airport Industrial Rd
Fairbanks, AK 99709
(907) 450-7293
-----------------------------------------------

[toc] | [next] | [standalone]

#83712

Israel Brewster <israel@ravnalaska.net> writes:
>         when it again crashed with a SIGABRT. The crash dump the
>         system gave me doesn't tell me much, other than that it looks
>         like python is calling some C function when it crashes. I've
>         attached the crash report, in case it can mean something more
>         to someone else.

Somehow I missed the original post: did you get a core dump?  The
next thing to do is examine it under gdb.

>         
>         Can anyone give me some hints as to how to track down the
>         cause of this crash? 

If it's recurring, recompile python with -g to generate debugging
symbols, so you'll have an easier time examining the dump.  You can even
run the whole interpreter under gdb, though doing that for 33 days might
be uncharted territory.

[toc] | [prev] | [next] | [standalone]

#84453

[Multipart message — attachments visible in raw view] — view raw

On Jan 13, 2015, at 8:31 AM, Paul Rubin <no.email@nospam.invalid> wrote:

> Israel Brewster <israel@ravnalaska.net> writes:
>>        when it again crashed with a SIGABRT. The crash dump the
>>        system gave me doesn't tell me much, other than that it looks
>>        like python is calling some C function when it crashes. I've
>>        attached the crash report, in case it can mean something more
>>        to someone else.
> 
> Somehow I missed the original post: did you get a core dump?  The
> next thing to do is examine it under gdb.

And somehow I missed your response, so I guess that makes us even :-) I did get a core dump and stack trace, that I attached to my original post. I'll attach the new one from today to this message.

Running it through GDB didn't seem to give me much more information than that crash report - I can see that it is in thread 0, and the call stack is identical to what the crash report shows, but I can't see where in the python code it is.

Memory usage is relatively minimal - at the time of today's crash, top was reporting only around 84MB used - actually down a couple of MB from where it was yesterday.

> 
>> 
>>        Can anyone give me some hints as to how to track down the
>>        cause of this crash? 
> 
> If it's recurring, recompile python with -g to generate debugging
> symbols, so you'll have an easier time examining the dump.  You can even
> run the whole interpreter under gdb, though doing that for 33 days might
> be uncharted territory.

May just have to try it though :-)

In looking back over my GIT change logs, the only change I find that even remotely seems like it might be the issue is a call to del(). Not saying I think that is the issue, min you, but the rest of the changes I made were to things like SQL queries, that might change the data somewhat but not the functioning. Of course, that said I was making more regular changes before, maybe it simply never remained running long enough.

> -- 
> https://mail.python.org/mailman/listinfo/python-list

[toc] | [prev] | [next] | [standalone]

#84590

Israel Brewster <israel@ravnalaska.net> writes:
> ...
> Running it through GDB didn't seem to give me much more information than that crash report - I can see that it is in thread 0, and the call stack is identical to what the crash report shows, but I can't see where in the python code it is.
> ...
> Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
> 0   libsystem_kernel.dylib        	0x98176be6 __select + 10
> 1   time.so                       	0x0052fdda 0x52f000 + 3546
> 2   org.python.python             	0x0007e90a PyCFunction_Call + 98
> 3   org.python.python             	0x000ccef4 PyEval_EvalFrameEx + 8182
> 4   org.python.python             	0x000caeb2 PyEval_EvalCodeEx + 1777

This tells you: the crash happens somewhere in a call to a function
defined in "time.so".
I suppose that all such functions are harmless in themselves.


I see two potential causes:

  *  a stack overflow
     some Python builds are made with quite a small thread runtime stack.
     However, you would usually get a "SIGSEGV" rather than
     a "SIGABRT".

  *  a bad pointer passed in to the function.
     Again, I would expect rather a "SIGSEGV" than a "SIGABRT" --
     but maybe "__select" has a special sanity check for its arguments
     and raises "SIGABRT" when the check fails.

In order to determine where in your Python code the problem happens,
you likely need a Python with debugging symbols (system installed
Python interpreters usually are "stripped", e.g. they no longer
have debugging symbols).

Python comes with GDB macros able to show you information
at the Python (rather than the C) level. Here (Ubuntu 12.4),
it is at "/usr/share/doc/python2.7/gdbinit.gz".
Those macros need debugging symbols.
Read the top of this file, for how to get those macros defined
in your GDB session.


I doubt that being able to relate the problem to code
at the Python level will help you much. Your "SIGABRT" problem
is likely not caused on the Python level (but at C level).

If it is not caused by a runtime stack overflow, the cause is likely
a memory corruption, made non locally (far away) in some C extension.
Using a tool to track down memory corruption might help you
(however, in my experience, it is usually a huge task).

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web