Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1232097 > unrolled thread

Re: Multiple potential races on vma->vm_flags

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Multiple potential races on vma->vm_flags Oleg Nesterov <oleg@redhat.com> - 2015-09-24 15:20 +0200
    Re: Multiple potential races on vma->vm_flags Sasha Levin <sasha.levin@oracle.com> - 2015-09-24 18:30 +0200
      Re: Multiple potential races on vma->vm_flags Oleg Nesterov <oleg@redhat.com> - 2015-09-24 19:30 +0200
        Re: Multiple potential races on vma->vm_flags Andrey Ryabinin <ryabinin.a.a@gmail.com> - 2015-09-24 21:00 +0200
          Re: Multiple potential races on vma->vm_flags Sasha Levin <sasha.levin@oracle.com> - 2015-09-24 21:10 +0200
          Re: Multiple potential races on vma->vm_flags Oleg Nesterov <oleg@redhat.com> - 2015-09-25 14:50 +0200

#1232097 — Re: Multiple potential races on vma->vm_flags

On 09/15, Sasha Levin wrote:
>
> I've modified my tests to stress the exit path of processes with many vmas,
> and hit the following NULL ptr deref (not sure if it's related to the original issue):

I am shy to ask. Looks like I am the only stupid one who needs
more info...

> [1181047.935563] kasan: GPF could be caused by NULL-ptr deref or user memory accessgeneral protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC KASAN

Well, I know absolutely nothing about kasan, to the point I can't even
unserstand where does this message come from. grep didn't help. But this
doesn't matter...

> [1181047.937223] Modules linked in:
> [1181047.937772] CPU: 4 PID: 21912 Comm: trinity-c341 Not tainted 4.3.0-rc1-next-20150914-sasha-00043-geddd763-dirty #2554
> [1181047.939387] task: ffff8804195c8000 ti: ffff880433f00000 task.ti: ffff880433f00000
> [1181047.940533] RIP: unmap_vmas (mm/memory.c:1337)

I do not know which tree/branch do you use. In Linus's tree mm/memory.c:1337 is

	struct mm_struct *mm = vma->vm_mm;

but this doesn't match the asm below,

>    0:   08 80 3c 02 00 0f       or     %al,0xf00023c(%rax)
>    6:   85 22                   test   %esp,(%rdx)
>    8:   01 00                   add    %eax,(%rax)
>    a:   00 48 8b                add    %cl,-0x75(%rax)
>    d:   43                      rex.XB
>    e:   40                      rex
>    f:   48 8d b8 c8 04 00 00    lea    0x4c8(%rax),%rdi
>   16:   48 89 45 d0             mov    %rax,-0x30(%rbp)
>   1a:   48 b8 00 00 00 00 00    movabs $0xdffffc0000000000,%rax
>   21:   fc ff df
>   24:   48 89 fa                mov    %rdi,%rdx
>   27:   48 c1 ea 03             shr    $0x3,%rdx
>   2b:*  80 3c 02 00             cmpb   $0x0,(%rdx,%rax,1)               <-- trapping instruction
>   2f:   0f 85 ee 00 00 00       jne    0x123
>   35:   48 8b 45 d0             mov    -0x30(%rbp),%rax
>   39:   48 83 b8 c8 04 00 00    cmpq   $0x0,0x4c8(%rax)
>   40:   00

And I do not see anything similar in "objdump -d". So could you at least
show mm/memory.c:1337 in your tree?

Hmm. movabs $0xdffffc0000000000,%rax above looks suspicious, this looks
like kasan_mem_to_shadow(). So perhaps this code was generated by kasan?
(I can't check, my gcc is very old). Or what?

Any chance you can tell us where exactly we hit NULL-deref in unmap_vmas?

Oleg.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1232241

On 09/24/2015 09:11 AM, Oleg Nesterov wrote:
> On 09/15, Sasha Levin wrote:
>>
>> I've modified my tests to stress the exit path of processes with many vmas,
>> and hit the following NULL ptr deref (not sure if it's related to the original issue):
> 
> I am shy to ask. Looks like I am the only stupid one who needs
> more info...
> 
>> [1181047.935563] kasan: GPF could be caused by NULL-ptr deref or user memory accessgeneral protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC KASAN
> 
> Well, I know absolutely nothing about kasan, to the point I can't even
> unserstand where does this message come from. grep didn't help. But this
> doesn't matter...

The reason behind this message is that NULL ptr derefs when using kasan are
manifested as GFPs. This is because in order to validate an access to a given
memory address kasan would check (shadow_base + (mem_offset >> 3)), so in the case of
a NULL it would try to access shadow_base + 0, which would GFP.

>> [1181047.937223] Modules linked in:
>> [1181047.937772] CPU: 4 PID: 21912 Comm: trinity-c341 Not tainted 4.3.0-rc1-next-20150914-sasha-00043-geddd763-dirty #2554
>> [1181047.939387] task: ffff8804195c8000 ti: ffff880433f00000 task.ti: ffff880433f00000
>> [1181047.940533] RIP: unmap_vmas (mm/memory.c:1337)
> 
> I do not know which tree/branch do you use. In Linus's tree mm/memory.c:1337 is

I'm running -next + Kirill's THP patchset.

> 	struct mm_struct *mm = vma->vm_mm;

void unmap_vmas(struct mmu_gather *tlb,
                struct vm_area_struct *vma, unsigned long start_addr,
                unsigned long end_addr)
{
        struct mm_struct *mm = vma->vm_mm;

        mmu_notifier_invalidate_range_start(mm, start_addr, end_addr);
        for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next)
                unmap_single_vma(tlb, vma, start_addr, end_addr, NULL); <--- this
        mmu_notifier_invalidate_range_end(mm, start_addr, end_addr);
}

> but this doesn't match the asm below,
> 
>>    0:   08 80 3c 02 00 0f       or     %al,0xf00023c(%rax)
>>    6:   85 22                   test   %esp,(%rdx)
>>    8:   01 00                   add    %eax,(%rax)
>>    a:   00 48 8b                add    %cl,-0x75(%rax)
>>    d:   43                      rex.XB
>>    e:   40                      rex
>>    f:   48 8d b8 c8 04 00 00    lea    0x4c8(%rax),%rdi
>>   16:   48 89 45 d0             mov    %rax,-0x30(%rbp)
>>   1a:   48 b8 00 00 00 00 00    movabs $0xdffffc0000000000,%rax
>>   21:   fc ff df
>>   24:   48 89 fa                mov    %rdi,%rdx
>>   27:   48 c1 ea 03             shr    $0x3,%rdx
>>   2b:*  80 3c 02 00             cmpb   $0x0,(%rdx,%rax,1)               <-- trapping instruction
>>   2f:   0f 85 ee 00 00 00       jne    0x123
>>   35:   48 8b 45 d0             mov    -0x30(%rbp),%rax
>>   39:   48 83 b8 c8 04 00 00    cmpq   $0x0,0x4c8(%rax)
>>   40:   00
> 
> And I do not see anything similar in "objdump -d". So could you at least
> show mm/memory.c:1337 in your tree?
> 
> Hmm. movabs $0xdffffc0000000000,%rax above looks suspicious, this looks
> like kasan_mem_to_shadow(). So perhaps this code was generated by kasan?
> (I can't check, my gcc is very old). Or what?

This is indeed kasan code. 0xdffffc0000000000 is the shadow base, and you see
kasan trying to access shadow base + (ptr >> 3), which is why we get GFP.

Looking at the assembly, the address we were trying to access was:

 RDI: 00000000000004c8

> Any chance you can tell us where exactly we hit NULL-deref in unmap_vmas?

I hope the information above helped, please let me know if it didn't and you
need anything else.


Thanks,
Sasha

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1232282

On 09/24, Sasha Levin wrote:
>
> On 09/24/2015 09:11 AM, Oleg Nesterov wrote:
> >
> > Well, I know absolutely nothing about kasan, to the point I can't even
> > unserstand where does this message come from. grep didn't help. But this
> > doesn't matter...
>
> The reason behind this message is that NULL ptr derefs when using kasan are
> manifested as GFPs. This is because in order to validate an access to a given
> memory address kasan would check (shadow_base + (mem_offset >> 3)), so in the case of
> a NULL it would try to access shadow_base + 0, which would GFP.

OK, so this just means the kernele derefs the NULL pointer,

> I'm running -next + Kirill's THP patchset.
>
> > 	struct mm_struct *mm = vma->vm_mm;
>
> void unmap_vmas(struct mmu_gather *tlb,
>                 struct vm_area_struct *vma, unsigned long start_addr,
>                 unsigned long end_addr)
> {
>         struct mm_struct *mm = vma->vm_mm;
>
>         mmu_notifier_invalidate_range_start(mm, start_addr, end_addr);
>         for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next)
>                 unmap_single_vma(tlb, vma, start_addr, end_addr, NULL); <--- this
>         mmu_notifier_invalidate_range_end(mm, start_addr, end_addr);
> }

And I do not see any dereference at this line,

> >>    0:   08 80 3c 02 00 0f       or     %al,0xf00023c(%rax)
> >>    6:   85 22                   test   %esp,(%rdx)
> >>    8:   01 00                   add    %eax,(%rax)
> >>    a:   00 48 8b                add    %cl,-0x75(%rax)
> >>    d:   43                      rex.XB
> >>    e:   40                      rex
> >>    f:   48 8d b8 c8 04 00 00    lea    0x4c8(%rax),%rdi
> >>   16:   48 89 45 d0             mov    %rax,-0x30(%rbp)
> >>   1a:   48 b8 00 00 00 00 00    movabs $0xdffffc0000000000,%rax
> >>   21:   fc ff df
> >>   24:   48 89 fa                mov    %rdi,%rdx
> >>   27:   48 c1 ea 03             shr    $0x3,%rdx
> >>   2b:*  80 3c 02 00             cmpb   $0x0,(%rdx,%rax,1)               <-- trapping instruction
> >>   2f:   0f 85 ee 00 00 00       jne    0x123
> >>   35:   48 8b 45 d0             mov    -0x30(%rbp),%rax
> >>   39:   48 83 b8 c8 04 00 00    cmpq   $0x0,0x4c8(%rax)
> >>   40:   00
> >
> > And I do not see anything similar in "objdump -d". So could you at least
> > show mm/memory.c:1337 in your tree?
> >
> > Hmm. movabs $0xdffffc0000000000,%rax above looks suspicious, this looks
> > like kasan_mem_to_shadow(). So perhaps this code was generated by kasan?
> > (I can't check, my gcc is very old). Or what?
>
> This is indeed kasan code. 0xdffffc0000000000 is the shadow base, and you see
> kasan trying to access shadow base + (ptr >> 3), which is why we get GFP.

and thus this asm can't help, right?

So how can we figure out where exactly the kernel hits NULL ? And what
exactly it tries to dereference?

> I hope the information above helped, please let me know if it didn't and you
> need anything else.

Thanks a lot, it does help, but I am still confused.


Looks like, "function+offset" is more useful than the line numbers,
at least we could look at mm/memory.s.

Oleg.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1232335

2015-09-24 20:26 GMT+03:00 Oleg Nesterov <oleg@redhat.com>:
> On 09/24, Sasha Levin wrote:
>>
>> On 09/24/2015 09:11 AM, Oleg Nesterov wrote:
>> >
>> > Well, I know absolutely nothing about kasan, to the point I can't even
>> > unserstand where does this message come from. grep didn't help. But this
>> > doesn't matter...
>>
>> The reason behind this message is that NULL ptr derefs when using kasan are
>> manifested as GFPs. This is because in order to validate an access to a given
>> memory address kasan would check (shadow_base + (mem_offset >> 3)), so in the case of
>> a NULL it would try to access shadow_base + 0, which would GFP.
>
> OK, so this just means the kernele derefs the NULL pointer,
>
>> I'm running -next + Kirill's THP patchset.
>>
>> >     struct mm_struct *mm = vma->vm_mm;
>>
>> void unmap_vmas(struct mmu_gather *tlb,
>>                 struct vm_area_struct *vma, unsigned long start_addr,
>>                 unsigned long end_addr)
>> {
>>         struct mm_struct *mm = vma->vm_mm;
>>
>>         mmu_notifier_invalidate_range_start(mm, start_addr, end_addr);
>>         for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next)
>>                 unmap_single_vma(tlb, vma, start_addr, end_addr, NULL); <--- this
>>         mmu_notifier_invalidate_range_end(mm, start_addr, end_addr);
>> }
>
> And I do not see any dereference at this line,
>

I noticed, that addr2line sometimes doesn't work reliably on
compiler-instrumented code.
I've seen couple times that it points to the next line of code.


>> >>    0:   08 80 3c 02 00 0f       or     %al,0xf00023c(%rax)
>> >>    6:   85 22                   test   %esp,(%rdx)
>> >>    8:   01 00                   add    %eax,(%rax)
>> >>    a:   00 48 8b                add    %cl,-0x75(%rax)
>> >>    d:   43                      rex.XB
>> >>    e:   40                      rex
>> >>    f:   48 8d b8 c8 04 00 00    lea    0x4c8(%rax),%rdi
>> >>   16:   48 89 45 d0             mov    %rax,-0x30(%rbp)
>> >>   1a:   48 b8 00 00 00 00 00    movabs $0xdffffc0000000000,%rax
>> >>   21:   fc ff df
>> >>   24:   48 89 fa                mov    %rdi,%rdx
>> >>   27:   48 c1 ea 03             shr    $0x3,%rdx
>> >>   2b:*  80 3c 02 00             cmpb   $0x0,(%rdx,%rax,1)               <-- trapping instruction
>> >>   2f:   0f 85 ee 00 00 00       jne    0x123
>> >>   35:   48 8b 45 d0             mov    -0x30(%rbp),%rax
>> >>   39:   48 83 b8 c8 04 00 00    cmpq   $0x0,0x4c8(%rax)
>> >>   40:   00
>> >
>> > And I do not see anything similar in "objdump -d". So could you at least
>> > show mm/memory.c:1337 in your tree?
>> >
>> > Hmm. movabs $0xdffffc0000000000,%rax above looks suspicious, this looks
>> > like kasan_mem_to_shadow(). So perhaps this code was generated by kasan?
>> > (I can't check, my gcc is very old). Or what?
>>
>> This is indeed kasan code. 0xdffffc0000000000 is the shadow base, and you see
>> kasan trying to access shadow base + (ptr >> 3), which is why we get GFP.
>
> and thus this asm can't help, right?
>

I think it can.

> So how can we figure out where exactly the kernel hits NULL ? And what
> exactly it tries to dereference?

So we tried to dereference 0x4c8.  That 0x4c8 is probably offset in some struct.
The only big struct here is mm_struct.
So I think that we tried to derefernce null mm, and this asm:
         > cmpq   $0x0,0x4c8(%rax)

is likely from inlined mm_has_notifiers():
    static inline int mm_has_notifiers(struct mm_struct *mm)
    {
             return unlikely(mm->mmu_notifier_mm);
    }


Sasha, could you confirm that in your kernel mmu_notifier_mm field has
0x4c8 offset?
I would use gdb for that:
gdb vmlinux
(gdb) p/x &(((struct mm_struct*)0)->mmu_notifier_mm)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1232340

On 09/24/2015 02:52 PM, Andrey Ryabinin wrote:
> Sasha, could you confirm that in your kernel mmu_notifier_mm field has
> 0x4c8 offset?
> I would use gdb for that:
> gdb vmlinux
> (gdb) p/x &(((struct mm_struct*)0)->mmu_notifier_mm)

(gdb) p/x &(((struct mm_struct*)0)->mmu_notifier_mm)
$1 = 0x4c8


Thanks,
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1232770

On 09/24, Andrey Ryabinin wrote:
>
> 2015-09-24 20:26 GMT+03:00 Oleg Nesterov <oleg@redhat.com>:
> > On 09/24, Sasha Levin wrote:
> >>
> >> void unmap_vmas(struct mmu_gather *tlb,
> >>                 struct vm_area_struct *vma, unsigned long start_addr,
> >>                 unsigned long end_addr)
> >> {
> >>         struct mm_struct *mm = vma->vm_mm;
> >>
> >>         mmu_notifier_invalidate_range_start(mm, start_addr, end_addr);
> >>         for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next)
> >>                 unmap_single_vma(tlb, vma, start_addr, end_addr, NULL); <--- this
> >>         mmu_notifier_invalidate_range_end(mm, start_addr, end_addr);
> >> }
> >
> > And I do not see any dereference at this line,
> >
>
> I noticed, that addr2line sometimes doesn't work reliably on
> compiler-instrumented code.
> I've seen couple times that it points to the next line of code.

Yes, I know that we can't trust it. That is why I think (at least in
this particular case) function+offset would be more helpful. And we
need more asm probably.

> >> >>    0:   08 80 3c 02 00 0f       or     %al,0xf00023c(%rax)
> >> >>    6:   85 22                   test   %esp,(%rdx)
> >> >>    8:   01 00                   add    %eax,(%rax)
> >> >>    a:   00 48 8b                add    %cl,-0x75(%rax)
> >> >>    d:   43                      rex.XB
> >> >>    e:   40                      rex
> >> >>    f:   48 8d b8 c8 04 00 00    lea    0x4c8(%rax),%rdi
> >> >>   16:   48 89 45 d0             mov    %rax,-0x30(%rbp)
> >> >>   1a:   48 b8 00 00 00 00 00    movabs $0xdffffc0000000000,%rax
> >> >>   21:   fc ff df
> >> >>   24:   48 89 fa                mov    %rdi,%rdx
> >> >>   27:   48 c1 ea 03             shr    $0x3,%rdx
> >> >>   2b:*  80 3c 02 00             cmpb   $0x0,(%rdx,%rax,1)               <-- trapping instruction
> >> >>   2f:   0f 85 ee 00 00 00       jne    0x123
> >> >>   35:   48 8b 45 d0             mov    -0x30(%rbp),%rax
> >> >>   39:   48 83 b8 c8 04 00 00    cmpq   $0x0,0x4c8(%rax)
> >> >>   40:   00
> >> >
> >> > And I do not see anything similar in "objdump -d". So could you at least
> >> > show mm/memory.c:1337 in your tree?
> >> >
> >> > Hmm. movabs $0xdffffc0000000000,%rax above looks suspicious, this looks
> >> > like kasan_mem_to_shadow(). So perhaps this code was generated by kasan?
> >> > (I can't check, my gcc is very old). Or what?
> >>
> >> This is indeed kasan code. 0xdffffc0000000000 is the shadow base, and you see
> >> kasan trying to access shadow base + (ptr >> 3), which is why we get GFP.
> >
> > and thus this asm can't help, right?
> >
>
> I think it can.
>
> > So how can we figure out where exactly the kernel hits NULL ? And what
> > exactly it tries to dereference?
>
> So we tried to dereference 0x4c8.  That 0x4c8 is probably offset in some struct.
> The only big struct here is mm_struct.
> So I think that we tried to derefernce null mm, and this asm:
>          > cmpq   $0x0,0x4c8(%rax)
>
> is likely from inlined mm_has_notifiers():
>     static inline int mm_has_notifiers(struct mm_struct *mm)
>     {
>              return unlikely(mm->mmu_notifier_mm);
>     }

Looks reasonable... Thanks.

I was going to say that this is impossible because the caller should have
crashed if ->mm == NULL. But unmap_vmas() uses mm = vma->vm_mm, so it looks
like this vma or mm->mmap was corrupted...

Oleg.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web