Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1405709 > unrolled thread

x86: A process doesn't stop on hw breakpoints sometimes

Back to article view | Back to linux.kernel

Contents

  x86: A process doesn't stop on hw breakpoints sometimes Andrei Vagin <avagin@gmail.com> - 2016-05-24 01:10 +0200
    Re: x86: A process doesn't stop on hw breakpoints sometimes Andrei Vagin <avagin@gmail.com> - 2016-05-24 03:30 +0200
      Re: x86: A process doesn't stop on hw breakpoints sometimes Andrei Vagin <avagin@gmail.com> - 2016-05-24 07:40 +0200
    Re: x86: A process doesn't stop on hw breakpoints sometimes Andy Lutomirski <luto@amacapital.net> - 2016-05-24 03:40 +0200
      Re: x86: A process doesn't stop on hw breakpoints sometimes Oleg Nesterov <oleg@redhat.com> - 2016-05-25 01:40 +0200

#1405709 — x86: A process doesn't stop on hw breakpoints sometimes

[Multipart message — attachments visible in raw view] — view raw

Hi,

We use breakpoints on CRIU to stop a processes before calling
rt_sigreturn and we found that sometimes a process runs through a
break-point without stopping on it.

https://github.com/xemul/criu/issues/162


A small reproducer is attached. It forks a child, stops it, sets a
breakpoint, executes a child, waits when it stops on the breakpoint. I
execute it a few times concurrently and wait a few minutes.

https://asciinema.org/a/006l3u5v82ubbkfy9fto07agd

I know that it can be reproduced on:
AMD A10 Micro-6700T
Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz
Intel(R) Core(TM) i7-4600U CPU @ 2.10GHz

so It doesn't look like a bug in a processor.

Thanks,
Andrew

[toc] | [next] | [standalone]

#1405760

On Mon, May 23, 2016 at 4:05 PM, Andrei Vagin <avagin@gmail.com> wrote:
> Hi,
>
> We use breakpoints on CRIU to stop a processes before calling
> rt_sigreturn and we found that sometimes a process runs through a
> break-point without stopping on it.
>
> https://github.com/xemul/criu/issues/162
>
>
> A small reproducer is attached. It forks a child, stops it, sets a
> breakpoint, executes a child, waits when it stops on the breakpoint. I
> execute it a few times concurrently and wait a few minutes.
>
> https://asciinema.org/a/006l3u5v82ubbkfy9fto07agd

I reproduced this issue on 4.4.9-300.fc23.x86_64

>
> I know that it can be reproduced on:
> AMD A10 Micro-6700T
> Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz
> Intel(R) Core(TM) i7-4600U CPU @ 2.10GHz
>
> so It doesn't look like a bug in a processor.
>
> Thanks,
> Andrew

[toc] | [prev] | [next] | [standalone]

#1405851

On Mon, May 23, 2016 at 6:28 PM, Andrei Vagin <avagin@gmail.com> wrote:
> On Mon, May 23, 2016 at 4:05 PM, Andrei Vagin <avagin@gmail.com> wrote:
>> Hi,
>>
>> We use breakpoints on CRIU to stop a processes before calling
>> rt_sigreturn and we found that sometimes a process runs through a
>> break-point without stopping on it.
>>
>> https://github.com/xemul/criu/issues/162
>>
>>
>> A small reproducer is attached. It forks a child, stops it, sets a
>> breakpoint, executes a child, waits when it stops on the breakpoint. I
>> execute it a few times concurrently and wait a few minutes.
>>
>> https://asciinema.org/a/006l3u5v82ubbkfy9fto07agd
>
> I reproduced this issue on 4.4.9-300.fc23.x86_64

Oops. I can't reproduce this issue on 4.6 and 4.5. Looks like it was
fixed between 4.4 and 4.5. Sorry for the noise.

>
>>
>> I know that it can be reproduced on:
>> AMD A10 Micro-6700T
>> Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz
>> Intel(R) Core(TM) i7-4600U CPU @ 2.10GHz
>>
>> so It doesn't look like a bug in a processor.
>>
>> Thanks,
>> Andrew

[toc] | [prev] | [next] | [standalone]

#1405766

On Mon, May 23, 2016 at 4:05 PM, Andrei Vagin <avagin@gmail.com> wrote:
> Hi,
>
> We use breakpoints on CRIU to stop a processes before calling
> rt_sigreturn and we found that sometimes a process runs through a
> break-point without stopping on it.
>
> https://github.com/xemul/criu/issues/162
>
>
> A small reproducer is attached. It forks a child, stops it, sets a
> breakpoint, executes a child, waits when it stops on the breakpoint. I
> execute it a few times concurrently and wait a few minutes.
>
> https://asciinema.org/a/006l3u5v82ubbkfy9fto07agd
>
> I know that it can be reproduced on:
> AMD A10 Micro-6700T
> Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz
> Intel(R) Core(TM) i7-4600U CPU @ 2.10GHz
>
> so It doesn't look like a bug in a processor.

I'm guessing you're either hitting a subtle bug in the mess that is
breakpoint handling or you're hitting a bug in perf's context switch
code.

Given that the breakpoint gets missed many times in a row, this is
presumably either a bug in breakpoint programming (i.e. the thing
isn't actually set in dr0/dr7) or a bug in the bp state tracking.  If
it were a bug in RF flag handling, I'd expect it to skip once and trip
the second time through.

All that being said, I stared at the code for a while and I don't see
the bug.  I can trigger this quite rarely on a VM, and it's not fun to
debug :(

--Andy

[toc] | [prev] | [next] | [standalone]

#1406514

On 05/23, Andy Lutomirski wrote:
>
> I'm guessing you're either hitting a subtle bug in the mess that is
> breakpoint handling or you're hitting a bug in perf's context switch
> code.

yes, same feeling...

> Given that the breakpoint gets missed many times in a row,

yes, the child specially tries to hit the same bp again and again,

> this is
> presumably either a bug in breakpoint programming (i.e. the thing
> isn't actually set in dr0/dr7) or a bug in the bp state tracking.

or some buf in perf_sched_in(). In fact this is what I think now, but
I can be wrong.

> If
> it were a bug in RF flag handling, I'd expect it to skip once and trip
> the second time through.

Exactly.

It would be nice to ensure that this problem has actually gone, and how.

So, Andrei, if you have any motivation, we can continue. The next step
needs a simple kernel patch or kernel module which allows to read dr0/dr7
and print these registers in the "fail" loop.

Oleg.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web