Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1483832 > unrolled thread

perf: perf_fuzzer lockup in perf_cgroup_attach

Started byVince Weaver <vincent.weaver@maine.edu>
First post2016-09-15 04:50 +0200
Last post2016-09-15 14:50 +0200
Articles 11 on this page of 31 — 7 participants

Back to article view | Back to linux.kernel

Contents

  perf: perf_fuzzer lockup in perf_cgroup_attach Vince Weaver <vincent.weaver@maine.edu> - 2016-09-15 04:50 +0200
    Re: perf: perf_fuzzer lockup in perf_cgroup_attach Stephane Eranian <eranian@google.com> - 2016-09-15 07:40 +0200
      Re: perf: perf_fuzzer lockup in perf_cgroup_attach Peter Zijlstra <peterz@infradead.org> - 2016-09-15 09:30 +0200
        Re: perf: perf_fuzzer lockup in perf_cgroup_attach Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-15 14:20 +0200
          [PATCH] scripts: add script for translating stack dump function  offsets Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-16 16:50 +0200
            Re: [PATCH] scripts: add script for translating stack dump function offsets Linus Torvalds <torvalds@linux-foundation.org> - 2016-09-16 20:20 +0200
              Re: [PATCH] scripts: add script for translating stack dump function  offsets Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-16 21:20 +0200
                Re: [PATCH] scripts: add script for translating stack dump function offsets Linus Torvalds <torvalds@linux-foundation.org> - 2016-09-16 21:30 +0200
                  [PATCH v2] scripts: add script for translating stack dump function  offsets Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-16 23:30 +0200
                    Re: [PATCH v2] scripts: add script for translating stack dump  function offsets Linus Torvalds <torvalds@linux-foundation.org> - 2016-09-17 02:10 +0200
                      Re: [PATCH v2] scripts: add script for translating stack dump  function offsets Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-17 02:50 +0200
                        Re: [PATCH v2] scripts: add script for translating stack dump  function offsets Linus Torvalds <torvalds@linux-foundation.org> - 2016-09-17 04:00 +0200
                          Re: [PATCH v2] scripts: add script for translating stack dump  function offsets Linus Torvalds <torvalds@linux-foundation.org> - 2016-09-17 04:40 +0200
                          Re: [PATCH v2] scripts: add script for translating stack dump  function offsets Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-17 20:50 +0200
                          [PATCH v3] scripts: add script for translating stack dump function Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-19 18:00 +0200
                            Re: [PATCH v3] scripts: add script for translating stack dump function Linus Torvalds <torvalds@linux-foundation.org> - 2016-09-19 21:00 +0200
                              Re: [PATCH v3] scripts: add script for translating stack dump function Linus Torvalds <torvalds@linux-foundation.org> - 2016-09-19 21:20 +0200
                                Re: [PATCH v3] scripts: add script for translating stack dump  function Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-19 21:30 +0200
                                Re: [PATCH v3] scripts: add script for translating stack dump  function Rabin Vincent <rabin@rab.in> - 2016-09-19 22:10 +0200
                                  Re: [PATCH v3] scripts: add script for translating stack dump function Linus Torvalds <torvalds@linux-foundation.org> - 2016-09-19 22:30 +0200
                                    Re: [PATCH v3] scripts: add script for translating stack dump  function Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-19 23:00 +0200
                                      Re: [PATCH v3] scripts: add script for translating stack dump  function Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-19 23:10 +0200
                                      Re: [PATCH v3] scripts: add script for translating stack dump function Linus Torvalds <torvalds@linux-foundation.org> - 2016-09-19 23:10 +0200
                    Re: [PATCH v2] scripts: add script for translating stack dump  function offsets Peter Zijlstra <peterz@infradead.org> - 2016-09-17 03:30 +0200
                    Re: [PATCH v2] scripts: add script for translating stack dump  function offsets Rabin Vincent <rabin@rab.in> - 2016-09-17 10:20 +0200
                      Re: [PATCH v2] scripts: add script for translating stack dump  function offsets Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-17 20:50 +0200
                Re: [PATCH] scripts: add script for translating stack dump function offsets Vegard Nossum <vegard.nossum@gmail.com> - 2016-09-17 11:20 +0200
                  Re: [PATCH] scripts: add script for translating stack dump function  offsets Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-17 21:00 +0200
      Re: perf: perf_fuzzer lockup in perf_cgroup_attach Vince Weaver <vincent.weaver@maine.edu> - 2016-09-15 14:50 +0200
    Re: perf: perf_fuzzer lockup in perf_cgroup_attach Peter Zijlstra <peterz@infradead.org> - 2016-09-15 09:20 +0200
      Re: perf: perf_fuzzer lockup in perf_cgroup_attach Vince Weaver <vincent.weaver@maine.edu> - 2016-09-15 14:50 +0200

Page 2 of 2 — ← Prev page 1 [2]

#1486850 — Re: [PATCH v3] scripts: add script for translating stack dump function

FromJosh Poimboeuf <jpoimboe@redhat.com>
Date2016-09-19 23:00 +0200
SubjectRe: [PATCH v3] scripts: add script for translating stack dump function
Message-ID<sjdPr-1U6-1@gated-at.bofh.it>
In reply to#1486840
On Mon, Sep 19, 2016 at 01:24:03PM -0700, Linus Torvalds wrote:
> On Mon, Sep 19, 2016 at 1:00 PM, Rabin Vincent <rabin@rab.in> wrote:
> >
> > Note that addr2line has localized strings, so the regex with the " at "
> > won't match for everyone unless you invoke addr2line with LANG=C.
> 
> Ok, I'll make it match just on the space instead.
> 
> > __write_once_size på /home/rabinv/dev/linux/include/linux/compiler.h:248
> 
> That's an odd localization choice.
> 
> "på"? Wouldn't "i" (or perhaps "vid") be a better choice?
> 
> Anyway, this works for me in the Swedish locale too. Look ok?
> 
>    -       addr2line -fpie $objfile $hexaddr | sed "s;$dir_prefix;;"
>    +       addr2line -fpie $objfile $hexaddr |
>    +               sed "s; $dir_prefix\(\./\)*; ;"
> 
> because I certainly hope there is always a space there.

No luck.  The Japanese translation uses an empty string:

  $ grep -A1 '" at "' binutils/po/ja.po
  msgid " at "
  msgstr ""

-- 
Josh

[toc] | [prev] | [next] | [standalone]

#1486855 — Re: [PATCH v3] scripts: add script for translating stack dump function

FromJosh Poimboeuf <jpoimboe@redhat.com>
Date2016-09-19 23:10 +0200
SubjectRe: [PATCH v3] scripts: add script for translating stack dump function
Message-ID<sjdZ8-2dG-21@gated-at.bofh.it>
In reply to#1486850
On Mon, Sep 19, 2016 at 02:02:18PM -0700, Linus Torvalds wrote:
> On Mon, Sep 19, 2016 at 1:56 PM, Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> > On Mon, Sep 19, 2016 at 01:24:03PM -0700, Linus Torvalds wrote:
> >>
> >> because I certainly hope there is always a space there.
> >
> > No luck.  The Japanese translation uses an empty string:
> 
> Heh.
> 
> Ok, we don't want to *just* have the "./" pattern ever be replaced,
> because for all I know there could be a directory name ending with "."
> in there somewhere.
> 
> But I guess we could do it this way instead:
> 
>   --- a/scripts/faddr2line
>   +++ b/scripts/faddr2line
>   @@ -79,7 +79,7 @@ find_dir_prefix() {
>                   return
>           fi
> 
>   -       DIR_PREFIX=$prefix
>   +       DIR_PREFIX="$prefix\(\./\)*"
>           return 0
>    }
> 
> and thus just make it part of the auto-generated pattern string instead.

Actually, false alarm.  The empty translation string seems to mean "use
the original string":

  $ export LANG=ja_JP.utf8
  $ ./scripts/faddr2line ~/k/vmlinux type_show+0x10/45
  type_show+0x10/0x2d:
  type_show at drivers/video/backlight/backlight.c:213

So your previous patch which checks for a space looks good after all :-)

-- 
Josh

[toc] | [prev] | [next] | [standalone]

#1486857 — Re: [PATCH v3] scripts: add script for translating stack dump function

FromLinus Torvalds <torvalds@linux-foundation.org>
Date2016-09-19 23:10 +0200
SubjectRe: [PATCH v3] scripts: add script for translating stack dump function
Message-ID<sjdZ8-2dG-23@gated-at.bofh.it>
In reply to#1486850
On Mon, Sep 19, 2016 at 1:56 PM, Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> On Mon, Sep 19, 2016 at 01:24:03PM -0700, Linus Torvalds wrote:
>>
>> because I certainly hope there is always a space there.
>
> No luck.  The Japanese translation uses an empty string:

Heh.

Ok, we don't want to *just* have the "./" pattern ever be replaced,
because for all I know there could be a directory name ending with "."
in there somewhere.

But I guess we could do it this way instead:

  --- a/scripts/faddr2line
  +++ b/scripts/faddr2line
  @@ -79,7 +79,7 @@ find_dir_prefix() {
                  return
          fi

  -       DIR_PREFIX=$prefix
  +       DIR_PREFIX="$prefix\(\./\)*"
          return 0
   }

and thus just make it part of the auto-generated pattern string instead.

             Linus

[toc] | [prev] | [next] | [standalone]

#1485451 — Re: [PATCH v2] scripts: add script for translating stack dump function offsets

FromPeter Zijlstra <peterz@infradead.org>
Date2016-09-17 03:30 +0200
SubjectRe: [PATCH v2] scripts: add script for translating stack dump function offsets
Message-ID<sicC5-4fC-1@gated-at.bofh.it>
In reply to#1485388
On Fri, Sep 16, 2016 at 04:26:56PM -0500, Josh Poimboeuf wrote:
> Ok, how about this.  If this looks ok, would you be willing to apply it?

Looks good, and yes without --inline addr2line would be tons less useful
;-) Thanks for cooking this up.

[toc] | [prev] | [next] | [standalone]

#1485497 — Re: [PATCH v2] scripts: add script for translating stack dump function offsets

FromRabin Vincent <rabin@rab.in>
Date2016-09-17 10:20 +0200
SubjectRe: [PATCH v2] scripts: add script for translating stack dump function offsets
Message-ID<sij0R-7B-3@gated-at.bofh.it>
In reply to#1485388
On Fri, Sep 16, 2016 at 04:26:56PM -0500, Josh Poimboeuf wrote:
> +	addr2line -ie $objfile $hexaddr

Could you pass in -f and -p too to addr2line?

Before:

 $ scripts/faddr2line ~/dev/kvm2/vmlinux free_reserved_area+0x90
 /home/rabin/dev/linux/include/linux/compiler.h:222
 /home/rabin/dev/linux/include/linux/page-flags.h:149
 /home/rabin/dev/linux/include/linux/page-flags.h:154
 /home/rabin/dev/linux/include/linux/page-flags.h:287
 /home/rabin/dev/linux/include/linux/mm.h:1778
 /home/rabin/dev/linux/include/linux/mm.h:1785
 /home/rabin/dev/linux/mm/page_alloc.c:6599

After:

 $ scripts/faddr2line ~/dev/kvm2/vmlinux free_reserved_area+0x90
 __read_once_size at /home/rabin/dev/linux/include/linux/compiler.h:222
  (inlined by) PageTail at /home/rabin/dev/linux/include/linux/page-flags.h:149
  (inlined by) PageCompound at /home/rabin/dev/linux/include/linux/page-flags.h:154
  (inlined by) ClearPageReserved at /home/rabin/dev/linux/include/linux/page-flags.h:287
  (inlined by) __free_reserved_page at /home/rabin/dev/linux/include/linux/mm.h:1778
  (inlined by) free_reserved_page at /home/rabin/dev/linux/include/linux/mm.h:1785
  (inlined by) free_reserved_area at /home/rabin/dev/linux/mm/page_alloc.c:6599

[toc] | [prev] | [next] | [standalone]

#1485587 — Re: [PATCH v2] scripts: add script for translating stack dump function offsets

FromJosh Poimboeuf <jpoimboe@redhat.com>
Date2016-09-17 20:50 +0200
SubjectRe: [PATCH v2] scripts: add script for translating stack dump function offsets
Message-ID<sisQx-64f-1@gated-at.bofh.it>
In reply to#1485497
On Sat, Sep 17, 2016 at 10:15:45AM +0200, Rabin Vincent wrote:
> On Fri, Sep 16, 2016 at 04:26:56PM -0500, Josh Poimboeuf wrote:
> > +	addr2line -ie $objfile $hexaddr
> 
> Could you pass in -f and -p too to addr2line?
> 
> Before:
> 
>  $ scripts/faddr2line ~/dev/kvm2/vmlinux free_reserved_area+0x90
>  /home/rabin/dev/linux/include/linux/compiler.h:222
>  /home/rabin/dev/linux/include/linux/page-flags.h:149
>  /home/rabin/dev/linux/include/linux/page-flags.h:154
>  /home/rabin/dev/linux/include/linux/page-flags.h:287
>  /home/rabin/dev/linux/include/linux/mm.h:1778
>  /home/rabin/dev/linux/include/linux/mm.h:1785
>  /home/rabin/dev/linux/mm/page_alloc.c:6599
> 
> After:
> 
>  $ scripts/faddr2line ~/dev/kvm2/vmlinux free_reserved_area+0x90
>  __read_once_size at /home/rabin/dev/linux/include/linux/compiler.h:222
>   (inlined by) PageTail at /home/rabin/dev/linux/include/linux/page-flags.h:149
>   (inlined by) PageCompound at /home/rabin/dev/linux/include/linux/page-flags.h:154
>   (inlined by) ClearPageReserved at /home/rabin/dev/linux/include/linux/page-flags.h:287
>   (inlined by) __free_reserved_page at /home/rabin/dev/linux/include/linux/mm.h:1778
>   (inlined by) free_reserved_page at /home/rabin/dev/linux/include/linux/mm.h:1785
>   (inlined by) free_reserved_area at /home/rabin/dev/linux/mm/page_alloc.c:6599

Yeah, I think that's much better.

-- 
Josh

[toc] | [prev] | [next] | [standalone]

#1485510 — Re: [PATCH] scripts: add script for translating stack dump function offsets

FromVegard Nossum <vegard.nossum@gmail.com>
Date2016-09-17 11:20 +0200
SubjectRe: [PATCH] scripts: add script for translating stack dump function offsets
Message-ID<sijWW-GT-21@gated-at.bofh.it>
In reply to#1485260
On 16 September 2016 at 21:17, Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> On Fri, Sep 16, 2016 at 11:12:10AM -0700, Linus Torvalds wrote:
>> Side note: I find addr2line almost completely useless in many cases
>> not because of address space randomization, but because of how complex
>> the inlining often is. I just had something where I decided to use
>> addr2line and it just pointed me to the __read_once_size_nocheck()
>> line in <linux/compiler.h>. That was not very useful.
>>
>> I ended up actually looking at the instructions *around* it, to find
>> where that one instruction had been inlined from.
>>
>> So I'm wondering if this kind of helper script could be extended to
>> have that "look around it" thing to help.
>
> I think that issue is solved by addr2line's '--inline' option, which the
> script uses:

Another small gotcha is that stack trace addresses are _return
addresses_, not callsites. So you'll sometimes want to pass 'addr - 1'
instead of just addr, as the next address (the return address) may
belong to a completely unrelated deeply inlined function.


Vegard

[toc] | [prev] | [next] | [standalone]

#1485590 — Re: [PATCH] scripts: add script for translating stack dump function offsets

FromJosh Poimboeuf <jpoimboe@redhat.com>
Date2016-09-17 21:00 +0200
SubjectRe: [PATCH] scripts: add script for translating stack dump function offsets
Message-ID<sit0d-67s-3@gated-at.bofh.it>
In reply to#1485510
On Sat, Sep 17, 2016 at 11:11:44AM +0200, Vegard Nossum wrote:
> On 16 September 2016 at 21:17, Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> > On Fri, Sep 16, 2016 at 11:12:10AM -0700, Linus Torvalds wrote:
> >> Side note: I find addr2line almost completely useless in many cases
> >> not because of address space randomization, but because of how complex
> >> the inlining often is. I just had something where I decided to use
> >> addr2line and it just pointed me to the __read_once_size_nocheck()
> >> line in <linux/compiler.h>. That was not very useful.
> >>
> >> I ended up actually looking at the instructions *around* it, to find
> >> where that one instruction had been inlined from.
> >>
> >> So I'm wondering if this kind of helper script could be extended to
> >> have that "look around it" thing to help.
> >
> > I think that issue is solved by addr2line's '--inline' option, which the
> > script uses:
> 
> Another small gotcha is that stack trace addresses are _return
> addresses_, not callsites. So you'll sometimes want to pass 'addr - 1'
> instead of just addr, as the next address (the return address) may
> belong to a completely unrelated deeply inlined function.

Hm, good point.  In that case, should it *always* subtract 1?  (Except
when the offset is already 0, of course.)

-- 
Josh

[toc] | [prev] | [next] | [standalone]

#1484132

FromVince Weaver <vincent.weaver@maine.edu>
Date2016-09-15 14:50 +0200
Message-ID<shEh4-75Z-31@gated-at.bofh.it>
In reply to#1483856
On Wed, 14 Sep 2016, Stephane Eranian wrote:

> I would think there is a way to disable KASLR for this kind of testing!

yes, it's just I hadn't realized I had it enabled until I couldn't figure 
out why addr2line wasn't working.

> Which of your fuzzer scripts are you using? fast_repro99.sh?

yes.  I should probably give that script a more meaningful name, but by 
now I think it's too late for that.

I also am running with paranoid set to "0"

> 
> > The best I can tell things are getting wedged somehow in
> > perf_cgroup_switch() while interrupts are disabled.  Interrupts are never
> > getting re-enabled, causing the RCU and NMI watchdogs to trigger (and more
> > alarming things like the SATA bus resetting).
> >
> How do you get to perf_cgroup_switch() from the traces you provide below?

It was my best guess after trying to trace through the code.  It could
in theory be anywhere but it definitely seems like it is happening after
perf_cgroup_attach at some point when interrupts are disabled (possibly
ftrace too?)

It's difficult because some code gets inlined and then it's jumping to a 
function pointer.  And then mid-debug the system finally had enough and 
started making threads hang forever while I was getting close.

I've rebooted the machine I'll see if I can replicate.

Vince

[toc] | [prev] | [next] | [standalone]

#1483891

FromPeter Zijlstra <peterz@infradead.org>
Date2016-09-15 09:20 +0200
Message-ID<shz7H-3We-7@gated-at.bofh.it>
In reply to#1483832
On Wed, Sep 14, 2016 at 10:43:29PM -0400, Vince Weaver wrote:
> 
> so the skylake that was fuzzing finally is mostly locked up.
> 
> Really hard to tell what's going, especially as KASLR made looking up the 
> addresses a big pain.
> 
> The best I can tell things are getting wedged somehow in 
> perf_cgroup_switch() while interrupts are disabled.  Interrupts are never 
> getting re-enabled, causing the RCU and NMI watchdogs to trigger (and more 
> alarming things like the SATA bus resetting).

How do you go about using cgroups? Do you set them up yourself, does the
fuzzer do so?

[toc] | [prev] | [next] | [standalone]

#1484121

FromVince Weaver <vincent.weaver@maine.edu>
Date2016-09-15 14:50 +0200
Message-ID<shEh3-75Z-1@gated-at.bofh.it>
In reply to#1483891
On Thu, 15 Sep 2016, Peter Zijlstra wrote:

> On Wed, Sep 14, 2016 at 10:43:29PM -0400, Vince Weaver wrote:
> > 
> > so the skylake that was fuzzing finally is mostly locked up.
> > 
> > Really hard to tell what's going, especially as KASLR made looking up the 
> > addresses a big pain.
> > 
> > The best I can tell things are getting wedged somehow in 
> > perf_cgroup_switch() while interrupts are disabled.  Interrupts are never 
> > getting re-enabled, causing the RCU and NMI watchdogs to trigger (and more 
> > alarming things like the SATA bus resetting).
> 
> How do you go about using cgroups? Do you set them up yourself, does the
> fuzzer do so?

that's an interesting thing, I don't think the fuzzer actually sets up any 
cgroups.  It will try creating events with PERF_FLAG_PID_CGROUP but it 
will just use a random number as the cgroup fd.  

I know at one time I meant to do more interesting things with cgroups
but if I recall it wasn't really possible without running as root and I 
usually don't fuzz as root.

Vince

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web