Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1271287 > unrolled thread
| Started by | Stefan Priebe - Profihost AG <s.priebe@profihost.ag> |
|---|---|
| First post | 2015-11-17 15:50 +0100 |
| Last post | 2015-11-19 10:50 +0100 |
| Articles | 20 on this page of 28 — 5 participants |
Back to article view | Back to linux.kernel
Asterisk deadlocks since Kernel 4.1 Stefan Priebe - Profihost AG <s.priebe@profihost.ag> - 2015-11-17 15:50 +0100
Re: Asterisk deadlocks since Kernel 4.1 Thomas Gleixner <tglx@linutronix.de> - 2015-11-17 20:20 +0100
Re: Asterisk deadlocks since Kernel 4.1 Stefan Priebe <s.priebe@profihost.ag> - 2015-11-17 20:30 +0100
Re: Asterisk deadlocks since Kernel 4.1 Thomas Gleixner <tglx@linutronix.de> - 2015-11-17 20:50 +0100
Re: Asterisk deadlocks since Kernel 4.1 Stefan Priebe <s.priebe@profihost.ag> - 2015-11-18 21:30 +0100
Re: Asterisk deadlocks since Kernel 4.1 Hannes Frederic Sowa <hannes@stressinduktion.org> - 2015-11-18 22:10 +0100
Re: Asterisk deadlocks since Kernel 4.1 Stefan Priebe <s.priebe@profihost.ag> - 2015-11-18 22:30 +0100
Re: Asterisk deadlocks since Kernel 4.1 Hannes Frederic Sowa <hannes@stressinduktion.org> - 2015-11-18 22:30 +0100
Re: Asterisk deadlocks since Kernel 4.1 Stefan Priebe - Profihost AG <s.priebe@profihost.ag> - 2015-11-19 10:40 +0100
Re: Asterisk deadlocks since Kernel 4.1 Florian Weimer <fweimer@redhat.com> - 2015-11-18 22:20 +0100
Re: Asterisk deadlocks since Kernel 4.1 Stefan Priebe <s.priebe@profihost.ag> - 2015-11-18 22:30 +0100
Re: Asterisk deadlocks since Kernel 4.1 Florian Weimer <fweimer@redhat.com> - 2015-11-19 10:40 +0100
Re: Asterisk deadlocks since Kernel 4.1 Stefan Priebe <s.priebe@profihost.ag> - 2015-11-18 22:40 +0100
Re: Asterisk deadlocks since Kernel 4.1 Hannes Frederic Sowa <hannes@stressinduktion.org> - 2015-11-18 22:50 +0100
Re: Asterisk deadlocks since Kernel 4.1 Stefan Priebe <s.priebe@profihost.ag> - 2015-11-18 22:50 +0100
Re: Asterisk deadlocks since Kernel 4.1 Hannes Frederic Sowa <hannes@stressinduktion.org> - 2015-11-18 23:00 +0100
Re: Asterisk deadlocks since Kernel 4.1 Stefan Priebe - Profihost AG <s.priebe@profihost.ag> - 2015-11-19 10:50 +0100
Re: Asterisk deadlocks since Kernel 4.1 Stefan Priebe - Profihost AG <s.priebe@profihost.ag> - 2015-11-19 11:00 +0100
Re: Asterisk deadlocks since Kernel 4.1 Hannes Frederic Sowa <hannes@stressinduktion.org> - 2015-11-19 12:50 +0100
Re: Asterisk deadlocks since Kernel 4.1 Stefan Priebe - Profihost AG <s.priebe@profihost.ag> - 2015-11-19 12:50 +0100
Re: Asterisk deadlocks since Kernel 4.1 Hannes Frederic Sowa <hannes@stressinduktion.org> - 2015-11-19 13:50 +0100
Re: Asterisk deadlocks since Kernel 4.1 Stefan Priebe - Profihost AG <s.priebe@profihost.ag> - 2015-11-19 13:50 +0100
Re: Asterisk deadlocks since Kernel 4.1 Florian Weimer <fweimer@redhat.com> - 2015-11-19 14:20 +0100
Re: Asterisk deadlocks since Kernel 4.1 Stefan Priebe <s.priebe@profihost.ag> - 2015-11-19 21:00 +0100
Re: Asterisk deadlocks since Kernel 4.1 Stefan Priebe - Profihost AG <s.priebe@profihost.ag> - 2015-11-23 13:50 +0100
Re: Asterisk deadlocks since Kernel 4.1 Hannes Frederic Sowa <hannes@stressinduktion.org> - 2015-11-23 14:00 +0100
Re: Asterisk deadlocks since Kernel 4.1 Stefan Priebe - Profihost AG <s.priebe@profihost.ag> - 2015-11-24 14:40 +0100
Re: Asterisk deadlocks since Kernel 4.1 Florian Weimer <fweimer@redhat.com> - 2015-11-19 10:50 +0100
Page 1 of 2 [1] 2 Next page →
| From | Stefan Priebe - Profihost AG <s.priebe@profihost.ag> |
|---|---|
| Date | 2015-11-17 15:50 +0100 |
| Subject | Asterisk deadlocks since Kernel 4.1 |
| Message-ID | <qvPK2-6hQ-11@gated-at.bofh.it> |
Hello,
since Upgrading our Asterisk System from Kernel 3.18.17 to 4.1.13 it
deadlocks every few hours (kill -9 is the only thing working). Booting
with 3.18 again let it run smooth again.
An strace shows asterisk is looping like this:
[pid 6068] timerfd_gettime(8, , {it_interval={0, 20000000},
it_value={0, 140592906050976}}) = 0
[pid 6068] read(8, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 6068] poll([{fd=9, events=POLLIN}, {fd=8, events=POLLIN|POLLPRI}],
2, 1000) = 1 ([{fd=8, revents=POLLIN}])
[pid 6068] timerfd_gettime(8, , {it_interval={0, 20000000},
it_value={0, 140592906050976}}) = 0
[pid 6068] read(8, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 6068] poll([{fd=9, events=POLLIN}, {fd=8, events=POLLIN|POLLPRI}],
2, 1000) = 1 ([{fd=8, revents=POLLIN}])
[pid 6068] timerfd_gettime(8, , {it_interval={0, 20000000},
it_value={0, 140592906050976}}) = 0
[pid 6068] read(8, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 6068] poll([{fd=9, events=POLLIN}, {fd=8, events=POLLIN|POLLPRI}],
2, 1000) = 1 ([{fd=8, revents=POLLIN}])
[pid 6068] timerfd_gettime(8, , {it_interval={0, 20000000},
it_value={0, 140592906050976}}) = 0
[pid 6068] read(8, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 6068] poll([{fd=9, events=POLLIN}, {fd=8, events=POLLIN|POLLPRI}],
2, 1000) = 1 ([{fd=8, revents=POLLIN}])
[pid 6068] timerfd_gettime(8, , {it_interval={0, 20000000},
it_value={0, 140592906050976}}) = 0
[pid 6068] read(8, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 6068] poll([{fd=9, events=POLLIN}, {fd=8, events=POLLIN|POLLPRI}],
2, 1000) = 1 ([{fd=8, revents=POLLIN}])
[pid 6068] timerfd_gettime(8, , {it_interval={0, 20000000},
it_value={0, 140592906050976}}) = 0
[pid 6068] read(8, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 6068] poll([{fd=9, events=POLLIN}, {fd=8, events=POLLIN|POLLPRI}],
2, 1000) = 1 ([{fd=8, revents=POLLIN}])
[pid 6068] timerfd_gettime(8, , {it_interval={0, 20000000},
it_value={0, 140592906050976}}) = 0
[pid 6068] read(8, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 6068] poll([{fd=9, events=POLLIN}, {fd=8, events=POLLIN|POLLPRI}],
2, 1000) = 1 ([{fd=8, revents=POLLIN}])
[pid 6068] timerfd_gettime(8, , {it_interval={0, 20000000},
it_value={0, 140592906050976}}) = 0
[pid 6068] read(8, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 6068] poll([{fd=9, events=POLLIN}, {fd=8, events=POLLIN|POLLPRI}],
2, 1000) = 1 ([{fd=8, revents=POLLIN}])
[pid 6068] timerfd_gettime(8, , {it_interval={0, 20000000},
it_value={0, 140592906050976}}) = 0
[pid 6068] read(8, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 6068] poll([{fd=9, events=POLLIN}, {fd=8, events=POLLIN|POLLPRI}],
2, 1000) = 1 ([{fd=8, revents=POLLIN}])
[pid 6068] timerfd_gettime(8, , {it_interval={0, 20000000},
it_value={0, 140592906050976}}) = 0
fd 8 is:
lrwx------ 1 root root 64 Nov 17 15:27 /proc/6025/fd/8 ->
anon_inode:[timerfd]
# cat /proc/6025/stack
[<ffffffff941d60e9>] poll_schedule_timeout+0x49/0x70
[<ffffffff941d76c7>] do_sys_poll+0x3d7/0x590
[<ffffffff941d78bc>] do_restart_poll+0x3c/0x70
[<ffffffff9408e7ff>] sys_restart_syscall+0x1f/0x30
[<ffffffff9463f1ee>] system_call_fastpath+0x12/0x71
[<ffffffffffffffff>] 0xffffffffffffffff
Any ideas how to debug this?
Greets,
Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [next] | [standalone]
| From | Thomas Gleixner <tglx@linutronix.de> |
|---|---|
| Date | 2015-11-17 20:20 +0100 |
| Message-ID | <qvTXk-J9-13@gated-at.bofh.it> |
| In reply to | #1271287 |
On Tue, 17 Nov 2015, Stefan Priebe - Profihost AG wrote:
> since Upgrading our Asterisk System from Kernel 3.18.17 to 4.1.13 it
> deadlocks every few hours (kill -9 is the only thing working). Booting
> with 3.18 again let it run smooth again.
>
> An strace shows asterisk is looping like this:
>
> [pid 6068] timerfd_gettime(8, , {it_interval={0, 20000000},
> it_value={0, 140592906050976}}) = 0
> [pid 6068] read(8, "\1\0\0\0\0\0\0\0", 8) = 8
> [pid 6068] poll([{fd=9, events=POLLIN}, {fd=8, events=POLLIN|POLLPRI}],
> 2, 1000) = 1 ([{fd=8, revents=POLLIN}])
> [pid 6068] timerfd_gettime(8, , {it_interval={0, 20000000},
> it_value={0, 140592906050976}}) = 0
> [pid 6068] read(8, "\1\0\0\0\0\0\0\0", 8) = 8
>
> fd 8 is:
>
> lrwx------ 1 root root 64 Nov 17 15:27 /proc/6025/fd/8 ->
> anon_inode:[timerfd]
>
>
> # cat /proc/6025/stack
> [<ffffffff941d60e9>] poll_schedule_timeout+0x49/0x70
> [<ffffffff941d76c7>] do_sys_poll+0x3d7/0x590
> [<ffffffff941d78bc>] do_restart_poll+0x3c/0x70
> [<ffffffff9408e7ff>] sys_restart_syscall+0x1f/0x30
> [<ffffffff9463f1ee>] system_call_fastpath+0x12/0x71
> [<ffffffffffffffff>] 0xffffffffffffffff
>
> Any ideas how to debug this?
fd 8 is probably not really interesting. That looks like a interval
timer firing periodically. So it probably waits for fd 9 ...
Thanks,
tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Stefan Priebe <s.priebe@profihost.ag> |
|---|---|
| Date | 2015-11-17 20:30 +0100 |
| Message-ID | <qvU6Z-N7-3@gated-at.bofh.it> |
| In reply to | #1271589 |
Am 17.11.2015 um 20:15 schrieb Thomas Gleixner:
> On Tue, 17 Nov 2015, Stefan Priebe - Profihost AG wrote:
>> since Upgrading our Asterisk System from Kernel 3.18.17 to 4.1.13 it
>> deadlocks every few hours (kill -9 is the only thing working). Booting
>> with 3.18 again let it run smooth again.
>>
>> An strace shows asterisk is looping like this:
>>
>> [pid 6068] timerfd_gettime(8, , {it_interval={0, 20000000},
>> it_value={0, 140592906050976}}) = 0
>> [pid 6068] read(8, "\1\0\0\0\0\0\0\0", 8) = 8
>> [pid 6068] poll([{fd=9, events=POLLIN}, {fd=8, events=POLLIN|POLLPRI}],
>> 2, 1000) = 1 ([{fd=8, revents=POLLIN}])
>> [pid 6068] timerfd_gettime(8, , {it_interval={0, 20000000},
>> it_value={0, 140592906050976}}) = 0
>> [pid 6068] read(8, "\1\0\0\0\0\0\0\0", 8) = 8
>>
>> fd 8 is:
>>
>> lrwx------ 1 root root 64 Nov 17 15:27 /proc/6025/fd/8 ->
>> anon_inode:[timerfd]
>>
>>
>> # cat /proc/6025/stack
>> [<ffffffff941d60e9>] poll_schedule_timeout+0x49/0x70
>> [<ffffffff941d76c7>] do_sys_poll+0x3d7/0x590
>> [<ffffffff941d78bc>] do_restart_poll+0x3c/0x70
>> [<ffffffff9408e7ff>] sys_restart_syscall+0x1f/0x30
>> [<ffffffff9463f1ee>] system_call_fastpath+0x12/0x71
>> [<ffffffffffffffff>] 0xffffffffffffffff
>>
>> Any ideas how to debug this?
>
> fd 8 is probably not really interesting. That looks like a interval
> timer firing periodically. So it probably waits for fd 9 ...
I've now also two gdb backtraces from two crashes:
http://pastebin.com/raw.php?i=yih5jNt8
http://pastebin.com/raw.php?i=kGEcvH4T
Stefan
>
> Thanks,
>
> tglx
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Thomas Gleixner <tglx@linutronix.de> |
|---|---|
| Date | 2015-11-17 20:50 +0100 |
| Message-ID | <qvUqm-TY-13@gated-at.bofh.it> |
| In reply to | #1271590 |
On Tue, 17 Nov 2015, Stefan Priebe wrote: > I've now also two gdb backtraces from two crashes: > http://pastebin.com/raw.php?i=yih5jNt8 > > http://pastebin.com/raw.php?i=kGEcvH4T They don't tell me anything as I have no idea of the inner workings of asterisk. You might be better of to talk to the asterisk folks to help you track down what that thing is waiting for, so we can actually look at a well defined area. Thanks, tglx -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Stefan Priebe <s.priebe@profihost.ag> |
|---|---|
| Date | 2015-11-18 21:30 +0100 |
| Message-ID | <qwhwC-7V2-27@gated-at.bofh.it> |
| In reply to | #1271616 |
Am 17.11.2015 um 20:43 schrieb Thomas Gleixner:
> On Tue, 17 Nov 2015, Stefan Priebe wrote:
>> I've now also two gdb backtraces from two crashes:
>> http://pastebin.com/raw.php?i=yih5jNt8
>>
>> http://pastebin.com/raw.php?i=kGEcvH4T
>
> They don't tell me anything as I have no idea of the inner workings of
> asterisk. You might be better of to talk to the asterisk folks to help
> you track down what that thing is waiting for, so we can actually look
> at a well defined area.
The asterisk guys told me it's a livelock asterisk is waiting for
getaddrinfo / recvmsg.
Thread 2 (Thread 0x7fbe989c6700 (LWP 12890)):
#0 0x00007fbeb9eb487d in recvmsg () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fbeb9ed4fcc in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fbeb9ed544a in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007fbeb9e92007 in getaddrinfo () from
/lib/x86_64-linux-gnu/libc.so.6
#4 0x00000000004f2e81 in ast_sockaddr_resolve
(addrs=addrs@entry=0x7fbe989c48d8, str=<optimized out>,
str@entry=0x7fbe989c4a80 "10.12.12.92:2052", flags=flags@entry=0,
family=2) at netsock2.c:268
#5 0x00007fbeb095fb8b in ast_sockaddr_resolve_first_af
(family=<optimized out>, flag=0, name=0x7fbe989c4a80 "10.12.12.92:2052",
addr=0x7fbeb4114948) at chan_sip.c:30797
#6 ast_sockaddr_resolve_first_transport (transport=<optimized out>,
flag=0, name=0x7fbe989c4a80 "10.12.12.92:2052",
addr=0x7fbeb4114948) at chan_sip.c:30828
#7 set_destination (uri=<optimized out>, p=0x7fbeb4114438) at
chan_sip.c:10602
#8 reqprep (req=req@entry=0x7fbe989c4fc0, p=p@entry=0x7fbeb4114438,
sipmethod=sipmethod@entry=8, seqno=<optimized out>,
seqno@entry=0, newbranch=newbranch@entry=1) at chan_sip.c:10925
---Type <return> to continue, or q <return> to quit---
#9 0x00007fbeb0969968 in transmit_request_with_auth
(p=p@entry=0x7fbeb4114438, newbranch=1, reliable=XMIT_RELIABLE, seqno=0,
sipmethod=8) at chan_sip.c:14296
#10 0x00007fbeb098b704 in sip_hangup (ast=<optimized out>) at
chan_sip.c:6822
#11 0x0000000000477b55 in ast_hangup (chan=chan@entry=0x7fbeb40ef018) at
channel.c:2887
#12 0x000000000050ce1b in __ast_pbx_run (c=c@entry=0x7fbeb40ef018,
args=args@entry=0x0) at pbx.c:5729
#13 0x000000000050e426 in pbx_thread (data=data@entry=0x7fbeb40ef018) at
pbx.c:5821
#14 0x0000000000548aaa in dummy_start (data=<optimized out>) at utils.c:1151
#15 0x00007fbeb965eb50 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007fbeb9eb395d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#17 0x0000000000000000 in ?? ()
In both cases, the thread is waiting in recvmsg.
Stefan
>
> Thanks,
>
> tglx
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Hannes Frederic Sowa <hannes@stressinduktion.org> |
|---|---|
| Date | 2015-11-18 22:10 +0100 |
| Message-ID | <qwi9k-8vZ-25@gated-at.bofh.it> |
| In reply to | #1272539 |
On Wed, Nov 18, 2015, at 21:23, Stefan Priebe wrote: > > Am 17.11.2015 um 20:43 schrieb Thomas Gleixner: > > On Tue, 17 Nov 2015, Stefan Priebe wrote: > >> I've now also two gdb backtraces from two crashes: > >> http://pastebin.com/raw.php?i=yih5jNt8 > >> > >> http://pastebin.com/raw.php?i=kGEcvH4T > > > > They don't tell me anything as I have no idea of the inner workings of > > asterisk. You might be better of to talk to the asterisk folks to help > > you track down what that thing is waiting for, so we can actually look > > at a well defined area. > > The asterisk guys told me it's a livelock asterisk is waiting for > getaddrinfo / recvmsg. > > Thread 2 (Thread 0x7fbe989c6700 (LWP 12890)): > #0 0x00007fbeb9eb487d in recvmsg () from /lib/x86_64-linux-gnu/libc.so.6 > #1 0x00007fbeb9ed4fcc in ?? () from /lib/x86_64-linux-gnu/libc.so.6 > #2 0x00007fbeb9ed544a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 > #3 0x00007fbeb9e92007 in getaddrinfo () from > /lib/x86_64-linux-gnu/libc.so.6 Florian Weimer recently patched glibc to detect filedescriptors which have been reused incorrectly in the glibc name resolving routines: <https://sourceware.org/ml/libc-alpha/2015-10/msg00865.html> Could you determine the type of filedescriptor in /proc/pid/fd and /proc/net/netlink by matching inodes? Thanks, Hannes -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Stefan Priebe <s.priebe@profihost.ag> |
|---|---|
| Date | 2015-11-18 22:30 +0100 |
| Message-ID | <qwisG-br-21@gated-at.bofh.it> |
| In reply to | #1272577 |
Am 18.11.2015 um 22:00 schrieb Hannes Frederic Sowa: > > > On Wed, Nov 18, 2015, at 21:23, Stefan Priebe wrote: >> >> Am 17.11.2015 um 20:43 schrieb Thomas Gleixner: >>> On Tue, 17 Nov 2015, Stefan Priebe wrote: >>>> I've now also two gdb backtraces from two crashes: >>>> http://pastebin.com/raw.php?i=yih5jNt8 >>>> >>>> http://pastebin.com/raw.php?i=kGEcvH4T >>> >>> They don't tell me anything as I have no idea of the inner workings of >>> asterisk. You might be better of to talk to the asterisk folks to help >>> you track down what that thing is waiting for, so we can actually look >>> at a well defined area. >> >> The asterisk guys told me it's a livelock asterisk is waiting for >> getaddrinfo / recvmsg. >> >> Thread 2 (Thread 0x7fbe989c6700 (LWP 12890)): >> #0 0x00007fbeb9eb487d in recvmsg () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x00007fbeb9ed4fcc in ?? () from /lib/x86_64-linux-gnu/libc.so.6 >> #2 0x00007fbeb9ed544a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 >> #3 0x00007fbeb9e92007 in getaddrinfo () from >> /lib/x86_64-linux-gnu/libc.so.6 > > Florian Weimer recently patched glibc to detect filedescriptors which > have been reused incorrectly in the glibc name resolving routines: > <https://sourceware.org/ml/libc-alpha/2015-10/msg00865.html> > > Could you determine the type of filedescriptor in /proc/pid/fd and > /proc/net/netlink by matching inodes? you mean just: la /proc/$pid/fd and cat /proc/net/netlink Stefan > Thanks, > Hannes > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Hannes Frederic Sowa <hannes@stressinduktion.org> |
|---|---|
| Date | 2015-11-18 22:30 +0100 |
| Message-ID | <qwisG-br-25@gated-at.bofh.it> |
| In reply to | #1272589 |
On Wed, Nov 18, 2015, at 22:20, Stefan Priebe wrote: > you mean just: > la /proc/$pid/fd ls -l /proc/pid/fd/ the numbers in brackets in return from readlink are the inode numbers. > and > > cat /proc/net/netlink Exactly, last row is the inode number. Bye, Hannes -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Stefan Priebe - Profihost AG <s.priebe@profihost.ag> |
|---|---|
| Date | 2015-11-19 10:40 +0100 |
| Message-ID | <qwtR9-7uw-11@gated-at.bofh.it> |
| In reply to | #1272591 |
Am 18.11.2015 um 22:22 schrieb Hannes Frederic Sowa: > On Wed, Nov 18, 2015, at 22:20, Stefan Priebe wrote: >> you mean just: >> la /proc/$pid/fd > > ls -l /proc/pid/fd/ > > the numbers in brackets in return from readlink are the inode numbers. > >> and >> >> cat /proc/net/netlink > > Exactly, last row is the inode number. It has stopped locking since i've set: net.ipv6.conf.all.disable_ipv6=1 Stefan -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Florian Weimer <fweimer@redhat.com> |
|---|---|
| Date | 2015-11-18 22:20 +0100 |
| Message-ID | <qwij0-7N-19@gated-at.bofh.it> |
| In reply to | #1272539 |
On 11/18/2015 09:23 PM, Stefan Priebe wrote: > > Am 17.11.2015 um 20:43 schrieb Thomas Gleixner: >> On Tue, 17 Nov 2015, Stefan Priebe wrote: >>> I've now also two gdb backtraces from two crashes: >>> http://pastebin.com/raw.php?i=yih5jNt8 >>> >>> http://pastebin.com/raw.php?i=kGEcvH4T >> >> They don't tell me anything as I have no idea of the inner workings of >> asterisk. You might be better of to talk to the asterisk folks to help >> you track down what that thing is waiting for, so we can actually look >> at a well defined area. > > The asterisk guys told me it's a livelock asterisk is waiting for > getaddrinfo / recvmsg. > > Thread 2 (Thread 0x7fbe989c6700 (LWP 12890)): > #0 0x00007fbeb9eb487d in recvmsg () from /lib/x86_64-linux-gnu/libc.so.6 > #1 0x00007fbeb9ed4fcc in ?? () from /lib/x86_64-linux-gnu/libc.so.6 > #2 0x00007fbeb9ed544a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 > #3 0x00007fbeb9e92007 in getaddrinfo () from > /lib/x86_64-linux-gnu/libc.so.6 Stefan, please try to get a backtrace with debugging information. It is likely that this is the make_request/__check_pf functionality in glibc, but it would be nice to get some certainty. Which glibc version do you use? Has it got a fix for CVE-2013-7423? So far, the only known cause for a hang in this place (that is, lack of return from recvmsg) is incorrect file descriptor use. (CVE-2013-7423 is such an issue in glibc itself.) The kernel upgrade could change scheduling behavior, and the actual bug might have been latent before. Theoretically, recvmsg could also hang if the Netlink query was dropped by the kernel, or the final packet in the response was dropped. We never saw that happen, even under extreme load, but I didn't test with recent kernels. The glibc change Hannes mentioned won't detect the hang, but if there is incorrect file descriptor reuse going on, it is possible that the new assert catches it. Florian -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Stefan Priebe <s.priebe@profihost.ag> |
|---|---|
| Date | 2015-11-18 22:30 +0100 |
| Message-ID | <qwisG-br-15@gated-at.bofh.it> |
| In reply to | #1272585 |
Am 18.11.2015 um 22:18 schrieb Florian Weimer: > On 11/18/2015 09:23 PM, Stefan Priebe wrote: >> >> Am 17.11.2015 um 20:43 schrieb Thomas Gleixner: >>> On Tue, 17 Nov 2015, Stefan Priebe wrote: >>>> I've now also two gdb backtraces from two crashes: >>>> http://pastebin.com/raw.php?i=yih5jNt8 >>>> >>>> http://pastebin.com/raw.php?i=kGEcvH4T >>> >>> They don't tell me anything as I have no idea of the inner workings of >>> asterisk. You might be better of to talk to the asterisk folks to help >>> you track down what that thing is waiting for, so we can actually look >>> at a well defined area. >> >> The asterisk guys told me it's a livelock asterisk is waiting for >> getaddrinfo / recvmsg. >> >> Thread 2 (Thread 0x7fbe989c6700 (LWP 12890)): >> #0 0x00007fbeb9eb487d in recvmsg () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x00007fbeb9ed4fcc in ?? () from /lib/x86_64-linux-gnu/libc.so.6 >> #2 0x00007fbeb9ed544a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 >> #3 0x00007fbeb9e92007 in getaddrinfo () from >> /lib/x86_64-linux-gnu/libc.so.6 > > Stefan, > > please try to get a backtrace with debugging information. It is likely > that this is the make_request/__check_pf functionality in glibc, but it > would be nice to get some certainty. > > Which glibc version do you use? Has it got a fix for CVE-2013-7423? It's Debians 2.13-38+deb7u8 Debians issue tracker says it is fixed: https://security-tracker.debian.org/tracker/CVE-2013-7423 > So far, the only known cause for a hang in this place (that is, lack of > return from recvmsg) is incorrect file descriptor use. (CVE-2013-7423 > is such an issue in glibc itself.) The kernel upgrade could change > scheduling behavior, and the actual bug might have been latent before. > > Theoretically, recvmsg could also hang if the Netlink query was dropped > by the kernel, or the final packet in the response was dropped. We > never saw that happen, even under extreme load, but I didn't test with > recent kernels. The load is very low in this system. Just 30 phones and only 1-6 calling. > The glibc change Hannes mentioned won't detect the hang, but if there is > incorrect file descriptor reuse going on, it is possible that the new > assert catches it. > > Florian > Stefan -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Florian Weimer <fweimer@redhat.com> |
|---|---|
| Date | 2015-11-19 10:40 +0100 |
| Message-ID | <qwtR9-7uw-7@gated-at.bofh.it> |
| In reply to | #1272588 |
On 11/18/2015 10:23 PM, Stefan Priebe wrote: >> please try to get a backtrace with debugging information. It is likely >> that this is the make_request/__check_pf functionality in glibc, but it >> would be nice to get some certainty. >> >> Which glibc version do you use? Has it got a fix for CVE-2013-7423? > > It's Debians 2.13-38+deb7u8 Debians issue tracker says it is fixed: > https://security-tracker.debian.org/tracker/CVE-2013-7423 I checked, and the patch is there and applied. >> Theoretically, recvmsg could also hang if the Netlink query was dropped >> by the kernel, or the final packet in the response was dropped. We >> never saw that happen, even under extreme load, but I didn't test with >> recent kernels. > > The load is very low in this system. Just 30 phones and only 1-6 calling. This is rather odd. I'm not sure if we can do anything simple on the glibc to help to debug this. If something in the process consumes the Netlink message from the kernel (incorrectly reading from the internal glibc file descriptor), this is fairly difficult detect inside glibc (it would require rather large changes which do not currently exist at all). Florian -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Stefan Priebe <s.priebe@profihost.ag> |
|---|---|
| Date | 2015-11-18 22:40 +0100 |
| Message-ID | <qwiCm-f0-29@gated-at.bofh.it> |
| In reply to | #1272585 |
Am 18.11.2015 um 22:18 schrieb Florian Weimer:
> On 11/18/2015 09:23 PM, Stefan Priebe wrote:
>>
>> Am 17.11.2015 um 20:43 schrieb Thomas Gleixner:
>>> On Tue, 17 Nov 2015, Stefan Priebe wrote:
>>>> I've now also two gdb backtraces from two crashes:
>>>> http://pastebin.com/raw.php?i=yih5jNt8
>>>>
>>>> http://pastebin.com/raw.php?i=kGEcvH4T
>>>
>>> They don't tell me anything as I have no idea of the inner workings of
>>> asterisk. You might be better of to talk to the asterisk folks to help
>>> you track down what that thing is waiting for, so we can actually look
>>> at a well defined area.
>>
>> The asterisk guys told me it's a livelock asterisk is waiting for
>> getaddrinfo / recvmsg.
>>
>> Thread 2 (Thread 0x7fbe989c6700 (LWP 12890)):
>> #0 0x00007fbeb9eb487d in recvmsg () from /lib/x86_64-linux-gnu/libc.so.6
>> #1 0x00007fbeb9ed4fcc in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>> #2 0x00007fbeb9ed544a in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>> #3 0x00007fbeb9e92007 in getaddrinfo () from
>> /lib/x86_64-linux-gnu/libc.so.6
>
> Stefan,
>
> please try to get a backtrace with debugging information. It is likely
> that this is the make_request/__check_pf functionality in glibc, but it
> would be nice to get some certainty.
sorry here it is. What I'm wondering is why is there ipv6 stuff? I don't
have ipv6 except for link local. Could it be this one?
https://bugzilla.redhat.com/show_bug.cgi?id=505105#c79
Thread 31 (Thread 0x7f295c011700 (LWP 26654)):
#0 0x00007f295de3287d in recvmsg () at
../sysdeps/unix/syscall-template.S:82
#1 0x00007f295de52fcc in make_request (fd=35, pid=26631,
seen_ipv4=<optimized out>, seen_ipv6=<optimized out>,
in6ai=<optimized out>, in6ailen=<optimized out>) at
../sysdeps/unix/sysv/linux/check_pf.c:119
#2 0x00007f295de5344a in __check_pf (seen_ipv4=0x7f295c00e85f,
seen_ipv6=0x7f295c00e85e, in6ai=0x7f295c00e840,
in6ailen=0x7f295c00e838) at ../sysdeps/unix/sysv/linux/check_pf.c:271
#3 0x00007f295de10007 in *__GI_getaddrinfo (name=0x7f295c00e8b0
"10.12.12.55", service=0x7f295c00e8bc "2135",
hints=0x7f295c00e910, pai=0x7f295c00e908) at
../sysdeps/posix/getaddrinfo.c:2389
#4 0x000000000050287e in ast_sockaddr_resolve (addrs=0x7f295c00e9d0,
str=0x7f295c00ea30 "10.12.12.55:2135", flags=0, family=2)
at netsock2.c:268
#5 0x00007f2958963ba2 in ast_sockaddr_resolve_first_af
(addr=0x7f29300591d8, name=0x7f295c00ea30 "10.12.12.55:2135", flag=0,
family=2) at chan_sip.c:30689
#6 0x00007f2958963cb5 in ast_sockaddr_resolve_first_transport
(addr=0x7f29300591d8, name=0x7f295c00ea30 "10.12.12.55:2135",
flag=0, transport=1) at chan_sip.c:30720
#7 0x00007f29588fd3cc in set_destination (p=0x7f2930058cc8,
uri=0x7f29300576e8 "sip:9052@10.12.12.55:2135;line=to7a729l")
at chan_sip.c:10455
#8 0x00007f29588fe6e0 in reqprep (req=0x7f295c00fee0, p=0x7f2930058cc8,
sipmethod=4, seqno=287, newbranch=1) at chan_sip.c:10778
#9 0x00007f295890a201 in transmit_state_notify (p=0x7f2930058cc8,
state=1, full=1, timeout=0) at chan_sip.c:13259
#10 0x00007f29589141bb in cb_extensionstate (context=0x7f295c010cd0
"hints", exten=0x7f295c010c80 "9052QS", state=1,
data=0x7f2930058cc8) at chan_sip.c:15117
#11 0x000000000050ebf6 in handle_statechange (datap=0x7f293acef830) at
pbx.c:4972
#12 0x0000000000555f8e in tps_processing_function (data=0x1f24f28) at
taskprocessor.c:327
#13 0x0000000000569280 in dummy_start (data=0x1ed76f0) at utils.c:1173
#14 0x00007f295d5dcb50 in start_thread (arg=<optimized out>) at
pthread_create.c:304
#15 0x00007f295de3195d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#16 0x0000000000000000 in ?? ()
>
> Which glibc version do you use? Has it got a fix for CVE-2013-7423?
>
> So far, the only known cause for a hang in this place (that is, lack of
> return from recvmsg) is incorrect file descriptor use. (CVE-2013-7423
> is such an issue in glibc itself.) The kernel upgrade could change
> scheduling behavior, and the actual bug might have been latent before.
>
> Theoretically, recvmsg could also hang if the Netlink query was dropped
> by the kernel, or the final packet in the response was dropped. We
> never saw that happen, even under extreme load, but I didn't test with
> recent kernels.
>
> The glibc change Hannes mentioned won't detect the hang, but if there is
> incorrect file descriptor reuse going on, it is possible that the new
> assert catches it.
>
> Florian
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Hannes Frederic Sowa <hannes@stressinduktion.org> |
|---|---|
| Date | 2015-11-18 22:50 +0100 |
| Message-ID | <qwiM1-iJ-3@gated-at.bofh.it> |
| In reply to | #1272600 |
On Wed, Nov 18, 2015, at 22:36, Stefan Priebe wrote: > sorry here it is. What I'm wondering is why is there ipv6 stuff? I don't > have ipv6 except for link local. Could it be this one? > > https://bugzilla.redhat.com/show_bug.cgi?id=505105#c79 > > Thread 31 (Thread 0x7f295c011700 (LWP 26654)): > #0 0x00007f295de3287d in recvmsg () at > ../sysdeps/unix/syscall-template.S:82 > #1 0x00007f295de52fcc in make_request (fd=35, pid=26631, > seen_ipv4=<optimized out>, seen_ipv6=<optimized out>, > in6ai=<optimized out>, in6ailen=<optimized out>) at > ../sysdeps/unix/sysv/linux/check_pf.c:119 > #2 0x00007f295de5344a in __check_pf (seen_ipv4=0x7f295c00e85f, > seen_ipv6=0x7f295c00e85e, in6ai=0x7f295c00e840, > in6ailen=0x7f295c00e838) at > ../sysdeps/unix/sysv/linux/check_pf.c:271 > #3 0x00007f295de10007 in *__GI_getaddrinfo (name=0x7f295c00e8b0 > "10.12.12.55", service=0x7f295c00e8bc "2135", > hints=0x7f295c00e910, pai=0x7f295c00e908) at > ../sysdeps/posix/getaddrinfo.c:2389 Can you also get a /proc/pid/stack stacktrace? Thanks, Hannes -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Stefan Priebe <s.priebe@profihost.ag> |
|---|---|
| Date | 2015-11-18 22:50 +0100 |
| Message-ID | <qwiM1-iJ-5@gated-at.bofh.it> |
| In reply to | #1272602 |
Am 18.11.2015 um 22:40 schrieb Hannes Frederic Sowa: > On Wed, Nov 18, 2015, at 22:36, Stefan Priebe wrote: >> sorry here it is. What I'm wondering is why is there ipv6 stuff? I don't >> have ipv6 except for link local. Could it be this one? >> >> https://bugzilla.redhat.com/show_bug.cgi?id=505105#c79 >> >> Thread 31 (Thread 0x7f295c011700 (LWP 26654)): >> #0 0x00007f295de3287d in recvmsg () at >> ../sysdeps/unix/syscall-template.S:82 >> #1 0x00007f295de52fcc in make_request (fd=35, pid=26631, >> seen_ipv4=<optimized out>, seen_ipv6=<optimized out>, >> in6ai=<optimized out>, in6ailen=<optimized out>) at >> ../sysdeps/unix/sysv/linux/check_pf.c:119 >> #2 0x00007f295de5344a in __check_pf (seen_ipv4=0x7f295c00e85f, >> seen_ipv6=0x7f295c00e85e, in6ai=0x7f295c00e840, >> in6ailen=0x7f295c00e838) at >> ../sysdeps/unix/sysv/linux/check_pf.c:271 >> #3 0x00007f295de10007 in *__GI_getaddrinfo (name=0x7f295c00e8b0 >> "10.12.12.55", service=0x7f295c00e8bc "2135", >> hints=0x7f295c00e910, pai=0x7f295c00e908) at >> ../sysdeps/posix/getaddrinfo.c:2389 > > Can you also get a /proc/pid/stack stacktrace? Sure: For the thread: # cat /proc/26631/task/26654/stack [<ffffffffb34e5cdb>] __skb_recv_datagram+0x52b/0x5d0 [<ffffffffb34e5db2>] skb_recv_datagram+0x32/0x40 [<ffffffffb35227bc>] netlink_recvmsg+0x4c/0x300 [<ffffffffb34d3863>] sock_recvmsg+0x13/0x20 [<ffffffffb34d679e>] ___sys_recvmsg+0xee/0x230 [<ffffffffb34d7739>] __sys_recvmsg+0x49/0x90 [<ffffffffb34d7792>] SyS_recvmsg+0x12/0x20 [<ffffffffb363f1ee>] system_call_fastpath+0x12/0x71 [<ffffffffffffffff>] 0xffffffffffffffff Stefan > > Thanks, > Hannes > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Hannes Frederic Sowa <hannes@stressinduktion.org> |
|---|---|
| Date | 2015-11-18 23:00 +0100 |
| Message-ID | <qwiVH-ms-1@gated-at.bofh.it> |
| In reply to | #1272603 |
On Wed, Nov 18, 2015, at 22:42, Stefan Priebe wrote: > > Am 18.11.2015 um 22:40 schrieb Hannes Frederic Sowa: > > On Wed, Nov 18, 2015, at 22:36, Stefan Priebe wrote: > >> sorry here it is. What I'm wondering is why is there ipv6 stuff? I don't > >> have ipv6 except for link local. Could it be this one? > >> > >> https://bugzilla.redhat.com/show_bug.cgi?id=505105#c79 > >> > >> Thread 31 (Thread 0x7f295c011700 (LWP 26654)): > >> #0 0x00007f295de3287d in recvmsg () at > >> ../sysdeps/unix/syscall-template.S:82 > >> #1 0x00007f295de52fcc in make_request (fd=35, pid=26631, > >> seen_ipv4=<optimized out>, seen_ipv6=<optimized out>, > >> in6ai=<optimized out>, in6ailen=<optimized out>) at > >> ../sysdeps/unix/sysv/linux/check_pf.c:119 > >> #2 0x00007f295de5344a in __check_pf (seen_ipv4=0x7f295c00e85f, > >> seen_ipv6=0x7f295c00e85e, in6ai=0x7f295c00e840, > >> in6ailen=0x7f295c00e838) at > >> ../sysdeps/unix/sysv/linux/check_pf.c:271 > >> #3 0x00007f295de10007 in *__GI_getaddrinfo (name=0x7f295c00e8b0 > >> "10.12.12.55", service=0x7f295c00e8bc "2135", > >> hints=0x7f295c00e910, pai=0x7f295c00e908) at > >> ../sysdeps/posix/getaddrinfo.c:2389 > > > > Can you also get a /proc/pid/stack stacktrace? > > Sure: > > For the thread: > # cat /proc/26631/task/26654/stack > [<ffffffffb34e5cdb>] __skb_recv_datagram+0x52b/0x5d0 > [<ffffffffb34e5db2>] skb_recv_datagram+0x32/0x40 > [<ffffffffb35227bc>] netlink_recvmsg+0x4c/0x300 > [<ffffffffb34d3863>] sock_recvmsg+0x13/0x20 > [<ffffffffb34d679e>] ___sys_recvmsg+0xee/0x230 > [<ffffffffb34d7739>] __sys_recvmsg+0x49/0x90 > [<ffffffffb34d7792>] SyS_recvmsg+0x12/0x20 > [<ffffffffb363f1ee>] system_call_fastpath+0x12/0x71 > [<ffffffffffffffff>] 0xffffffffffffffff Ok, we are definitely in netlink code and thus have a correct(? at least a netlink) netlink fd which is blocking and waiting for an answer. I research the glibc code a little bit but I think that the occurrences of ipv6 parameters are just generic glibc code. Let me try to reproduce that. Thanks, Hannes -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Stefan Priebe - Profihost AG <s.priebe@profihost.ag> |
|---|---|
| Date | 2015-11-19 10:50 +0100 |
| Message-ID | <qwu0N-7xT-5@gated-at.bofh.it> |
| In reply to | #1272600 |
Am 19.11.2015 um 10:44 schrieb Florian Weimer: > On 11/18/2015 10:36 PM, Stefan Priebe wrote: > >>> please try to get a backtrace with debugging information. It is likely >>> that this is the make_request/__check_pf functionality in glibc, but it >>> would be nice to get some certainty. >> >> sorry here it is. What I'm wondering is why is there ipv6 stuff? I don't >> have ipv6 except for link local. > > glibc needs to know if the system has global unicast addresses if it > receives AAAA records. > > It's curious that net.ipv6.conf.all.disable_ipv6=1 makes the problem go > away. Even with that setting, the kernel seems to send two Netlink > responses. So either this is enough to narrow the window for the race > so that no longer triggers, or there is a genuine kernel issue with > supplying the requested IPv6 Netlink response. No idea it also goes away by downgrading to 3.18 again. Stefan >> Could it be this one? >> >> https://bugzilla.redhat.com/show_bug.cgi?id=505105#c79 > > No, that's on the DNS/UDP side, not in the Netlink code. > > Florian > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Stefan Priebe - Profihost AG <s.priebe@profihost.ag> |
|---|---|
| Date | 2015-11-19 11:00 +0100 |
| Message-ID | <qwuat-7Bg-5@gated-at.bofh.it> |
| In reply to | #1272968 |
OK it had a livelock again. It just took more time.
So here is the data:
# la /proc/2598/fd
total 0
dr-x------ 2 root root 0 Nov 19 06:53 .
dr-xr-xr-x 7 callweaver callweaver 0 Nov 18 22:38 ..
lrwx------ 1 root root 64 Nov 19 06:54 0 -> /dev/null
lrwx------ 1 root root 64 Nov 19 06:54 1 -> /dev/null
lrwx------ 1 root root 64 Nov 19 06:54 10 -> socket:[12066]
lr-x------ 1 root root 64 Nov 19 06:54 11 -> anon_inode:inotify
lr-x------ 1 root root 64 Nov 19 06:54 12 -> pipe:[12181]
l-wx------ 1 root root 64 Nov 19 06:54 13 -> pipe:[12181]
lrwx------ 1 root root 64 Nov 19 10:56 14 -> socket:[510853]
lrwx------ 1 root root 64 Nov 19 10:56 15 -> socket:[510854]
lrwx------ 1 root root 64 Nov 19 10:56 16 ->
anon_inode:[timerfd]
lr-x------ 1 root root 64 Nov 19 10:56 17 -> pipe:[510856]
l-wx------ 1 root root 64 Nov 19 10:56 18 -> pipe:[510856]
lrwx------ 1 root root 64 Nov 19 10:56 19 -> socket:[208723]
lrwx------ 1 root root 64 Nov 19 06:54 2 -> /dev/null
l-wx------ 1 root root 64 Nov 19 10:56 20 ->
/var/log/asterisk/queue_log
lrwx------ 1 root root 64 Nov 19 10:56 21 -> socket:[199595]
lrwx------ 1 root root 64 Nov 19 10:56 22 -> socket:[510873]
lr-x------ 1 root root 64 Nov 19 10:56 23 -> anon_inode:inotify
lrwx------ 1 root root 64 Nov 19 10:56 24 -> socket:[525349]
lrwx------ 1 root root 64 Nov 19 10:56 25 -> socket:[525350]
lrwx------ 1 root root 64 Nov 19 10:56 26 -> socket:[510874]
lrwx------ 1 root root 64 Nov 19 10:56 27 ->
anon_inode:[timerfd]
lr-x------ 1 root root 64 Nov 19 10:56 28 -> pipe:[510876]
l-wx------ 1 root root 64 Nov 19 10:56 29 -> pipe:[510876]
lr-x------ 1 root root 64 Nov 19 06:54 3 -> /dev/urandom
lrwx------ 1 root root 64 Nov 19 10:56 30 -> socket:[527569]
lrwx------ 1 root root 64 Nov 19 10:56 31 -> socket:[527570]
lrwx------ 1 root root 64 Nov 19 10:56 32 -> socket:[528123]
lrwx------ 1 root root 64 Nov 19 10:56 33 -> socket:[528124]
lrwx------ 1 root root 64 Nov 19 10:56 34 -> socket:[530711]
lrwx------ 1 root root 64 Nov 19 10:56 35 -> socket:[530712]
lrwx------ 1 root root 64 Nov 19 10:56 36 -> socket:[533366]
lrwx------ 1 root root 64 Nov 19 10:56 37 -> socket:[533367]
lrwx------ 1 root root 64 Nov 19 10:56 38 -> socket:[535390]
lrwx------ 1 root root 64 Nov 19 10:56 39 -> socket:[531056]
lrwx------ 1 root root 64 Nov 19 06:54 4 -> socket:[11726]
lrwx------ 1 root root 64 Nov 19 10:56 40 -> socket:[531057]
lrwx------ 1 root root 64 Nov 19 10:56 41 -> socket:[535391]
lrwx------ 1 root root 64 Nov 19 10:56 42 -> socket:[537751]
lrwx------ 1 root root 64 Nov 19 10:56 43 -> socket:[533468]
lrwx------ 1 root root 64 Nov 19 10:56 44 -> socket:[531154]
lrwx------ 1 root root 64 Nov 19 10:56 45 -> socket:[531155]
lrwx------ 1 root root 64 Nov 19 10:56 46 -> socket:[533469]
lrwx------ 1 root root 64 Nov 19 10:56 47 -> socket:[536172]
lrwx------ 1 root root 64 Nov 19 10:56 48 -> socket:[536173]
lrwx------ 1 root root 64 Nov 19 10:56 49 -> socket:[537877]
l-wx------ 1 root root 64 Nov 19 06:54 5 ->
/var/log/asterisk/messages
lrwx------ 1 root root 64 Nov 19 10:56 50 -> socket:[537752]
lrwx------ 1 root root 64 Nov 19 10:56 51 -> socket:[539817]
lrwx------ 1 root root 64 Nov 19 10:56 52 -> socket:[537878]
lrwx------ 1 root root 64 Nov 19 10:56 53 -> socket:[539818]
lrwx------ 1 root root 64 Nov 19 10:56 54 -> socket:[541781]
lrwx------ 1 root root 64 Nov 19 10:56 55 -> socket:[541782]
lrwx------ 1 root root 64 Nov 19 10:56 56 -> socket:[543462]
lrwx------ 1 root root 64 Nov 19 10:56 57 -> socket:[545171]
lrwx------ 1 root root 64 Nov 19 10:56 58 -> socket:[537432]
lrwx------ 1 root root 64 Nov 19 10:56 59 -> socket:[537433]
l-wx------ 1 root root 64 Nov 19 06:54 6 ->
/var/log/asterisk/debug.log
lrwx------ 1 root root 64 Nov 19 10:56 60 -> socket:[545172]
lrwx------ 1 root root 64 Nov 19 10:56 61 ->
anon_inode:[timerfd]
lrwx------ 1 root root 64 Nov 19 10:56 62 -> socket:[541196]
lrwx------ 1 root root 64 Nov 19 10:56 63 -> socket:[538319]
lrwx------ 1 root root 64 Nov 19 10:56 64 -> socket:[538320]
lrwx------ 1 root root 64 Nov 19 10:56 65 -> socket:[474586]
lrwx------ 1 root root 64 Nov 19 10:56 66 -> socket:[541197]
lrwx------ 1 root root 64 Nov 19 10:56 67 -> socket:[542437]
lrwx------ 1 root root 64 Nov 19 10:56 68 -> socket:[542438]
lr-x------ 1 root root 64 Nov 19 10:56 69 -> pipe:[545174]
lrwx------ 1 root root 64 Nov 19 06:54 7 ->
/var/lib/asterisk/astdb
lrwx------ 1 root root 64 Nov 19 10:56 70 -> socket:[543463]
l-wx------ 1 root root 64 Nov 19 10:56 71 -> pipe:[545174]
lrwx------ 1 root root 64 Nov 19 10:56 76 -> socket:[543659]
lrwx------ 1 root root 64 Nov 19 10:56 77 -> socket:[543660]
lrwx------ 1 root root 64 Nov 19 10:56 78 ->
anon_inode:[timerfd]
lr-x------ 1 root root 64 Nov 19 10:56 79 -> pipe:[543662]
lrwx------ 1 root root 64 Nov 19 06:54 8 -> anon_inode:[timerfd]
l-wx------ 1 root root 64 Nov 19 10:56 80 -> pipe:[543662]
lrwx------ 1 root root 64 Nov 19 06:54 9 -> socket:[12052]
[asterisksnom: ~]# cat /proc/net/netlink
sk Eth Pid Groups Rmem Wmem Dump Locks Drops
Inode
ffff8800bac17000 0 0 00000000 0 0 0 2 0
3
ffff8800b5ef8000 4 0 00000000 0 0 0 2 0
6201
ffff8800b71cf000 10 0 00000000 0 0 0 2 0
5455
ffff8800b7176000 11 0 00000000 0 0 0 2 0
12
ffff8800b1169000 15 4294962899 00000000 0 0 0 2 0
7979
ffff8800b16cf000 15 441 00000001 0 0 0 2 0
1542
ffff8800b1168800 15 4294962900 00000000 0 0 0 2 0
7978
ffff8800b7088800 15 0 00000000 0 0 0 2 0
5
ffff8800b71c9800 16 0 00000000 0 0 0 2 0
15
ffff8800b16ca000 16 2362 00000002 0 0 0 2 0
12313
Stefan
Am 19.11.2015 um 10:49 schrieb Stefan Priebe - Profihost AG:
>
> Am 19.11.2015 um 10:44 schrieb Florian Weimer:
>> On 11/18/2015 10:36 PM, Stefan Priebe wrote:
>>
>>>> please try to get a backtrace with debugging information. It is likely
>>>> that this is the make_request/__check_pf functionality in glibc, but it
>>>> would be nice to get some certainty.
>>>
>>> sorry here it is. What I'm wondering is why is there ipv6 stuff? I don't
>>> have ipv6 except for link local.
>>
>> glibc needs to know if the system has global unicast addresses if it
>> receives AAAA records.
>>
>> It's curious that net.ipv6.conf.all.disable_ipv6=1 makes the problem go
>> away. Even with that setting, the kernel seems to send two Netlink
>> responses. So either this is enough to narrow the window for the race
>> so that no longer triggers, or there is a genuine kernel issue with
>> supplying the requested IPv6 Netlink response.
>
> No idea it also goes away by downgrading to 3.18 again.
>
> Stefan
>
>>> Could it be this one?
>>>
>>> https://bugzilla.redhat.com/show_bug.cgi?id=505105#c79
>>
>> No, that's on the DNS/UDP side, not in the Netlink code.
>>
>> Florian
>>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Hannes Frederic Sowa <hannes@stressinduktion.org> |
|---|---|
| Date | 2015-11-19 12:50 +0100 |
| Message-ID | <qwvSV-iU-1@gated-at.bofh.it> |
| In reply to | #1272975 |
On Thu, Nov 19, 2015, at 10:56, Stefan Priebe - Profihost AG wrote: > OK it had a livelock again. It just took more time. > > So here is the data: Thanks, I couldn't reproduce it so far with simple threaded resolver loop on your kernel. :/ Your data is useless if you don't also provide the file descriptor which you are blocking on right now. ;) Thanks, Hannes -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Stefan Priebe - Profihost AG <s.priebe@profihost.ag> |
|---|---|
| Date | 2015-11-19 12:50 +0100 |
| Message-ID | <qwvSV-iU-7@gated-at.bofh.it> |
| In reply to | #1273065 |
Am 19.11.2015 um 12:41 schrieb Hannes Frederic Sowa: > On Thu, Nov 19, 2015, at 10:56, Stefan Priebe - Profihost AG wrote: >> OK it had a livelock again. It just took more time. >> >> So here is the data: > > Thanks, I couldn't reproduce it so far with simple threaded resolver > loop on your kernel. :/ > > Your data is useless if you don't also provide the file descriptor which > you are blocking on right now. ;) > > Thanks, > Hannes ah sorry. So we need the gdb backtrace with the rcvmsg and then the fd list + netlink list? Stefan -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
Page 1 of 2 [1] 2 Next page →
Back to top | Article view | linux.kernel
csiph-web