Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1732079 > unrolled thread

BUG: Sporadic crashes with current Linus tree

Back to article view | Back to linux.kernel

Contents

  BUG: Sporadic crashes with current Linus tree Thomas Gleixner <tglx@linutronix.de> - 2017-09-14 09:40 +0200
    Re: BUG: Sporadic crashes with current Linus tree Andy Lutomirski <luto@kernel.org> - 2017-09-14 17:30 +0200
      Re: BUG: Sporadic crashes with current Linus tree Thomas Gleixner <tglx@linutronix.de> - 2017-09-14 18:10 +0200
        Re: BUG: Sporadic crashes with current Linus tree Andy Lutomirski <luto@kernel.org> - 2017-09-14 18:50 +0200
          Re: BUG: Sporadic crashes with current Linus tree Thomas Gleixner <tglx@linutronix.de> - 2017-09-15 09:10 +0200

#1732079 — BUG: Sporadic crashes with current Linus tree

Hi!

I've seen the following crash sporadically with commit 46c1e79fee:

Have not seen that with 3882a734c19b, though I saw the PCID warnings on
that machine.

I have no idea how to reproduce so bisecting is pretty much pointless. Any
idea what to do?

Thanks,

	tglx


BUG: unable to handle kernel paging request at ffffffffffffffd8
IP: __memmove+0x24/0x1a0
PGD 1e17c0c067 P4D 1e17c0c067 PUD 1e17c0e067 PMD 0
Oops: 0002 [#1] SMP
CPU: 43 PID: 1958 Comm: rsyslogd Tainted: G        W       4.13.0+ #184
task: ffff9eb5e8780080 task.stack: ffffb648a08cc000
RIP: 0010:__memmove+0x24/0x1a0
RSP: 0018:ffffb648a08cfef0 EFLAGS: 00252097
RAX: ffffffffffffffd8 RBX: ffffffffffffff50 RCX: 0000000000000028
RDX: 0000000000000028 RSI: ffffb648a08cffd8 RDI: ffffffffffffffd8
RBP: ffffb648a08cff10 R08: ffffb648a08d0000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000000
R13: ffffb648a08cff20 R14: 00007fffef2ed9a8 R15: 0000000000000000
FS:  00007f05281eff80(0000) GS:ffff9eb5ff5c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffffffffffd8 CR3: 000000302857c002 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
? fixup_bad_iret+0x3c/0x60
  error_entry+0xb4/0xc0
? general_protection+0xc/0x30
Code: 90 90 90 90 90 90 90 48 89 f8 48 83 fa 20 0f 82 03 01 00 00 48 39 fe 7d 0f 49 89 f0 49 01 d0 49 39 f8 0f 8f 9f 00 00 00 48 89 d1 <f3> a4 c3 48 81 fa a8 02 00 00 72 05 40 38 fe 74 3b 48 83 ea 20
RIP: __memmove+0x24/0x1a0 RSP: ffffb648a08cfef0
CR2: ffffffffffffffd8
[ end trace 72d77f3e20344f94 ]---

[toc] | [next] | [standalone]

#1732353

On Thu, Sep 14, 2017 at 12:38 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> Hi!
>
> I've seen the following crash sporadically with commit 46c1e79fee:
>
> Have not seen that with 3882a734c19b, though I saw the PCID warnings on
> that machine.
>
> I have no idea how to reproduce so bisecting is pretty much pointless. Any
> idea what to do?

Does tools/testing/selftests/x86/sigreturn_64 reproduce it?

>
> Thanks,
>
>         tglx
>
>
> BUG: unable to handle kernel paging request at ffffffffffffffd8
> IP: __memmove+0x24/0x1a0
> PGD 1e17c0c067 P4D 1e17c0c067 PUD 1e17c0e067 PMD 0
> Oops: 0002 [#1] SMP
> CPU: 43 PID: 1958 Comm: rsyslogd Tainted: G        W       4.13.0+ #184
> task: ffff9eb5e8780080 task.stack: ffffb648a08cc000
> RIP: 0010:__memmove+0x24/0x1a0
> RSP: 0018:ffffb648a08cfef0 EFLAGS: 00252097
> RAX: ffffffffffffffd8 RBX: ffffffffffffff50 RCX: 0000000000000028
> RDX: 0000000000000028 RSI: ffffb648a08cffd8 RDI: ffffffffffffffd8
> RBP: ffffb648a08cff10 R08: ffffb648a08d0000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000000
> R13: ffffb648a08cff20 R14: 00007fffef2ed9a8 R15: 0000000000000000
> FS:  00007f05281eff80(0000) GS:ffff9eb5ff5c0000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: ffffffffffffffd8 CR3: 000000302857c002 CR4: 00000000001606e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
> ? fixup_bad_iret+0x3c/0x60
>   error_entry+0xb4/0xc0
> ? general_protection+0xc/0x30
> Code: 90 90 90 90 90 90 90 48 89 f8 48 83 fa 20 0f 82 03 01 00 00 48 39 fe 7d 0f 49 89 f0 49 01 d0 49 39 f8 0f 8f 9f 00 00 00 48 89 d1 <f3> a4 c3 48 81 fa a8 02 00 00 72 05 40 38 fe 74 3b 48 83 ea 20
> RIP: __memmove+0x24/0x1a0 RSP: ffffb648a08cfef0
> CR2: ffffffffffffffd8

Ugh, weird.  It kind of looks like current->thread.sp0 == NULL.  I
have a patch series that changes a bunch of that code in my git tree,
but that's definitely not in Linus' tree.

[toc] | [prev] | [next] | [standalone]

#1732413

On Thu, 14 Sep 2017, Andy Lutomirski wrote:
> On Thu, Sep 14, 2017 at 12:38 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > Hi!
> >
> > I've seen the following crash sporadically with commit 46c1e79fee:
> >
> > Have not seen that with 3882a734c19b, though I saw the PCID warnings on
> > that machine.
> >
> > I have no idea how to reproduce so bisecting is pretty much pointless. Any
> > idea what to do?
> 
> Does tools/testing/selftests/x86/sigreturn_64 reproduce it?

Will try tomorrow once I figured out how to compile that stuff. Invoking a
simple make in that directory fails.

Built it manually and when I run it tells: stack16 is too high

> Ugh, weird.  It kind of looks like current->thread.sp0 == NULL.  I
> have a patch series that changes a bunch of that code in my git tree,
> but that's definitely not in Linus' tree.

Right. The stupid thing is that the machine did not throw up all day
neither idle nor loaded. Still the same kernel which barfed tonight several
times.

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1732452

[Multipart message — attachments visible in raw view] — view raw

On Thu, Sep 14, 2017 at 9:00 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> On Thu, 14 Sep 2017, Andy Lutomirski wrote:
>> On Thu, Sep 14, 2017 at 12:38 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>> > Hi!
>> >
>> > I've seen the following crash sporadically with commit 46c1e79fee:
>> >
>> > Have not seen that with 3882a734c19b, though I saw the PCID warnings on
>> > that machine.
>> >
>> > I have no idea how to reproduce so bisecting is pretty much pointless. Any
>> > idea what to do?
>>
>> Does tools/testing/selftests/x86/sigreturn_64 reproduce it?
>
> Will try tomorrow once I figured out how to compile that stuff. Invoking a
> simple make in that directory fails.

What's the error?  It works for me.

>
> Built it manually and when I run it tells: stack16 is too high
>
>> Ugh, weird.  It kind of looks like current->thread.sp0 == NULL.  I
>> have a patch series that changes a bunch of that code in my git tree,
>> but that's definitely not in Linus' tree.
>
> Right. The stupid thing is that the machine did not throw up all day
> neither idle nor loaded. Still the same kernel which barfed tonight several
> times.

This is weird.  The crashing process is rsyslogd, which should have
been running for a long time and shouldn't have any strange state.  I
wonder if this is some kind of memory corruption.  There would have to
be corruption of thread_struct *and* some kind of issue causing IRET
to fail, though.

The attached patch could plausibly give some useful hint.

>
> Thanks,
>
>         tglx
>
>
>
>

[toc] | [prev] | [next] | [standalone]

#1732734

On Thu, 14 Sep 2017, Andy Lutomirski wrote:
> On Thu, Sep 14, 2017 at 9:00 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > On Thu, 14 Sep 2017, Andy Lutomirski wrote:
> >> On Thu, Sep 14, 2017 at 12:38 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> >> > Hi!
> >> >
> >> > I've seen the following crash sporadically with commit 46c1e79fee:
> >> >
> >> > Have not seen that with 3882a734c19b, though I saw the PCID warnings on
> >> > that machine.
> >> >
> >> > I have no idea how to reproduce so bisecting is pretty much pointless. Any
> >> > idea what to do?
> >>
> >> Does tools/testing/selftests/x86/sigreturn_64 reproduce it?
> >
> > Will try tomorrow once I figured out how to compile that stuff. Invoking a
> > simple make in that directory fails.
> 
> What's the error?  It works for me.

gcc -m64 -o /home/tglx/work/kernel/linus/linux/tools/testing/selftests/x86/sysret_ss_attrs_64 -O2 -g -std=gnu99 -pthread -Wall  sysret_ss_attrs.c thunks.S -lrt -ldl
/usr/bin/ld: /tmp/cco4vSkU.o: relocation R_X86_64_32S against `.text' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status

> >
> > Built it manually and when I run it tells: stack16 is too high
> >
> >> Ugh, weird.  It kind of looks like current->thread.sp0 == NULL.  I
> >> have a patch series that changes a bunch of that code in my git tree,
> >> but that's definitely not in Linus' tree.
> >
> > Right. The stupid thing is that the machine did not throw up all day
> > neither idle nor loaded. Still the same kernel which barfed tonight several
> > times.
> 
> This is weird.  The crashing process is rsyslogd, which should have
> been running for a long time and shouldn't have any strange state.  I
> wonder if this is some kind of memory corruption.  There would have to
> be corruption of thread_struct *and* some kind of issue causing IRET
> to fail, though.
> 
> The attached patch could plausibly give some useful hint.

I'll put it on that machine and hope it will reproduce. Didn't die since
yesterday moring ....

Thanks,

	tglx

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web