Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1651908 > unrolled thread

Re: [x86/mm] e2a7dcce31: kernel_BUG_at_arch/x86/mm/tlb.c

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [x86/mm] e2a7dcce31: kernel_BUG_at_arch/x86/mm/tlb.c Andy Lutomirski <luto@kernel.org> - 2017-05-27 18:10 +0200
    Re: [x86/mm] e2a7dcce31: kernel_BUG_at_arch/x86/mm/tlb.c Andy Lutomirski <luto@kernel.org> - 2017-05-27 19:00 +0200
      Re: [x86/mm] e2a7dcce31: kernel_BUG_at_arch/x86/mm/tlb.c Arjan van de Ven <arjan@linux.intel.com> - 2017-05-31 06:10 +0200
        Re: [x86/mm] e2a7dcce31: kernel_BUG_at_arch/x86/mm/tlb.c Rik van Riel <riel@redhat.com> - 2017-05-31 14:50 +0200
          Re: [x86/mm] e2a7dcce31: kernel_BUG_at_arch/x86/mm/tlb.c Andy Lutomirski <luto@kernel.org> - 2017-05-31 16:00 +0200
    Re: [x86/mm] e2a7dcce31: kernel_BUG_at_arch/x86/mm/tlb.c Ingo Molnar <mingo@kernel.org> - 2017-05-28 11:00 +0200
    Re: [x86/mm] e2a7dcce31: kernel_BUG_at_arch/x86/mm/tlb.c Rik van Riel <riel@redhat.com> - 2017-05-28 16:20 +0200
      Re: [x86/mm] e2a7dcce31: kernel_BUG_at_arch/x86/mm/tlb.c Andy Lutomirski <luto@kernel.org> - 2017-05-28 18:00 +0200

#1651908 — Re: [x86/mm] e2a7dcce31: kernel_BUG_at_arch/x86/mm/tlb.c

On Sat, May 27, 2017 at 6:31 AM, kernel test robot
<xiaolong.ye@intel.com> wrote:
>
> FYI, we noticed the following commit:
>
> commit: e2a7dcce31f10bd7471b4245a6d1f2de344e7adf ("x86/mm: Rework lazy TLB to track the actual loaded mm")
> https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git x86/tlbflush_cleanup

Ugh, there's an unpleasant interaction between this patch and
intel_idle.  I suspect that the intel_idle code in question is either
wrong or pointless, but I want to investigate further.  Ingo, can you
hold off on applying this patch?

>
> in testcase: will-it-scale
> with following parameters:
>
>         test: readseek2
>         cpufreq_governor: performance
>
> test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
> test-url: https://github.com/antonblanchard/will-it-scale
>
>
> on test machine: 32 threads Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz with 64G memory
>
> caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
>
>
> +------------------------------------------+------------+------------+
> |                                          | ed3e5f1969 | e2a7dcce31 |
> +------------------------------------------+------------+------------+
> | boot_successes                           | 11         | 1          |
> | boot_failures                            | 0          | 4          |
> | kernel_BUG_at_arch/x86/mm/tlb.c          | 0          | 4          |
> | invalid_opcode:#[##]                     | 0          | 4          |
> | Kernel_panic-not_syncing:Fatal_exception | 0          | 4          |
> +------------------------------------------+------------+------------+
>
>
>
> [   22.777203] kernel BUG at arch/x86/mm/tlb.c:39!
> [   22.784095] invalid opcode: 0000 [#1] SMP
> [   22.788850] Modules linked in: ip_tables
> [   22.793503] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.12.0-rc2-00061-ge2a7dcc #1
> [   22.802448] Hardware name: Intel Corporation S2600CP/S2600CP, BIOS SE5C600.86B.99.99.x036.091920111209 09/19/2011
> [   22.814399] task: ffff88081bb725c0 task.stack: ffffc90006358000
> [   22.821303] RIP: 0010:leave_mm+0x39/0x40
> [   22.825950] RSP: 0018:ffffc9000635be48 EFLAGS: 00010046
> [   22.832065] RAX: 0000000000000001 RBX: ffffffff81fa8ba0 RCX: 000000000000001f
> [   22.840312] RDX: ffffffff81f1f9a0 RSI: ffffffff81fa8ba0 RDI: 0000000000000002
> [   22.848568] RBP: ffffc9000635be70 R08: 0000000000000760 R09: 0000000000000008
> [   22.856823] R10: ffffc9000635be50 R11: 0000000000000760 R12: 0000000000000004
> [   22.865069] R13: 0000000000000005 R14: 0000000000000030 R15: ffffffff81fa8d80
> [   22.873325] FS:  0000000000000000(0000) GS:ffff88081f280000(0000) knlGS:0000000000000000
> [   22.882838] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   22.889527] CR2: 000055635b17b0b8 CR3: 0000000001e09000 CR4: 00000000000406e0
> [   22.897780] Call Trace:
> [   22.900782]  ? intel_idle+0x109/0x110
> [   22.905159]  cpuidle_enter_state+0xf2/0x2d0
> [   22.910118]  cpuidle_enter+0x17/0x20
> [   22.914388]  call_cpuidle+0x23/0x40
> [   22.918551]  do_idle+0x189/0x1e0
> [   22.922431]  cpu_startup_entry+0x1d/0x20
> [   22.927091]  start_secondary+0x107/0x130
> [   22.931750]  secondary_startup_64+0x9f/0x9f
> [   22.936692] Code: 8b 05 04 bd fa 7e 83 f8 01 74 20 48 81 fa a0 f9 f1 81 74 15 55 31 d2 48 c7 c6 a0 f9 f1 81 31 ff 48 89 e5 e8 9a ff ff ff 5d f3 c3 <0f> 0b 0f 1f 44 00 00 66 66 66 66 90 55 48 8b 05 37 f5 02 01 48
> [   22.958497] RIP: leave_mm+0x39/0x40 RSP: ffffc9000635be48
> [   22.964859] ---[ end trace 1426870dcae851dd ]---
>
>
> To reproduce:
>
>         git clone https://github.com/01org/lkp-tests.git
>         cd lkp-tests
>         bin/lkp install job.yaml  # job file is attached in this email
>         bin/lkp run     job.yaml
>
>
>
> Thanks,
> Kernel Test Robot

[toc] | [next] | [standalone]

#1651937

On Sat, May 27, 2017 at 9:00 AM, Andy Lutomirski <luto@kernel.org> wrote:
> On Sat, May 27, 2017 at 6:31 AM, kernel test robot
> <xiaolong.ye@intel.com> wrote:
>>
>> FYI, we noticed the following commit:
>>
>> commit: e2a7dcce31f10bd7471b4245a6d1f2de344e7adf ("x86/mm: Rework lazy TLB to track the actual loaded mm")
>> https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git x86/tlbflush_cleanup
>
> Ugh, there's an unpleasant interaction between this patch and
> intel_idle.  I suspect that the intel_idle code in question is either
> wrong or pointless, but I want to investigate further.  Ingo, can you
> hold off on applying this patch?

I think this is what's going on: intel_idle has an optimization and
sometimes calls leave_mm().  This is a rather expensive way of working
around x86 Linux's fairly weak lazy mm handling.  It also abuses the
whole switch_mm state machine.  In particular, there's no guarantee
that the mm is actually lazy at the time.  The old code didn't care,
but the new code can oops.

The short-term fix is to just reorder the code in leave_mm() to avoid the OOPS.

I think the long-term fix is to improve laziness and delete the
leave_mm() call from intel_idle entirely.  Doing that will be quite
simple on top of the PCID.  PCID needs to reliably track when the TLB
is stale without needing IPIs for every invalidation, and laziness
should be able to reuse the same mechanism.  Once we do this, I think
we may even be able to remove leave_mm() altogether.  (Rik should like
this, since it will let us suppress host TLB flush IPIs being sent to
CPUs in guest mode as well.)

--Andy

[toc] | [prev] | [next] | [standalone]

#1653816

On 5/27/2017 9:56 AM, Andy Lutomirski wrote:
> On Sat, May 27, 2017 at 9:00 AM, Andy Lutomirski <luto@kernel.org> wrote:
>> On Sat, May 27, 2017 at 6:31 AM, kernel test robot
>> <xiaolong.ye@intel.com> wrote:
>>>
>>> FYI, we noticed the following commit:
>>>
>>> commit: e2a7dcce31f10bd7471b4245a6d1f2de344e7adf ("x86/mm: Rework lazy TLB to track the actual loaded mm")
>>> https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git x86/tlbflush_cleanup
>>
>> Ugh, there's an unpleasant interaction between this patch and
>> intel_idle.  I suspect that the intel_idle code in question is either
>> wrong or pointless, but I want to investigate further.  Ingo, can you
>> hold off on applying this patch?
>
> I think this is what's going on: intel_idle has an optimization and
> sometimes calls leave_mm().  This is a rather expensive way of working
> around x86 Linux's fairly weak lazy mm handling.  It also abuses the
> whole switch_mm state machine.  In particular, there's no guarantee
> that the mm is actually lazy at the time.  The old code didn't care,
> but the new code can oops.
>
> The short-term fix is to just reorder the code in leave_mm() to avoid the OOPS.

fwiw the reason the code is in intel_idle is to avoid tlb flush IPIs to idle cpus,
once the cpu goes into a deep enough idle state.  In the current linux code,
that is done by no longer having the old TLB live on the CPU, by switching to the neutral
kernel-only set of tlbs.

If your proposed changes do that (avoid the IPI/wakeup), great!
(if not, there should be a way to do that)

[toc] | [prev] | [next] | [standalone]

#1654189

On Tue, 2017-05-30 at 21:05 -0700, Arjan van de Ven wrote:
> On 5/27/2017 9:56 AM, Andy Lutomirski wrote:
> > On Sat, May 27, 2017 at 9:00 AM, Andy Lutomirski <luto@kernel.org>
> > wrote:
> > > On Sat, May 27, 2017 at 6:31 AM, kernel test robot
> > > <xiaolong.ye@intel.com> wrote:
> > > > 
> > > > FYI, we noticed the following commit:
> > > > 
> > > > commit: e2a7dcce31f10bd7471b4245a6d1f2de344e7adf ("x86/mm:
> > > > Rework lazy TLB to track the actual loaded mm")
> > > > https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git
> > > > x86/tlbflush_cleanup
> > > 
> > > Ugh, there's an unpleasant interaction between this patch and
> > > intel_idle.  I suspect that the intel_idle code in question is
> > > either
> > > wrong or pointless, but I want to investigate further.  Ingo, can
> > > you
> > > hold off on applying this patch?
> > 
> > I think this is what's going on: intel_idle has an optimization and
> > sometimes calls leave_mm().  This is a rather expensive way of
> > working
> > around x86 Linux's fairly weak lazy mm handling.  It also abuses
> > the
> > whole switch_mm state machine.  In particular, there's no guarantee
> > that the mm is actually lazy at the time.  The old code didn't
> > care,
> > but the new code can oops.
> > 
> > The short-term fix is to just reorder the code in leave_mm() to
> > avoid the OOPS.
> 
> fwiw the reason the code is in intel_idle is to avoid tlb flush IPIs
> to idle cpus,
> once the cpu goes into a deep enough idle state.  In the current
> linux code,
> that is done by no longer having the old TLB live on the CPU, by
> switching to the neutral
> kernel-only set of tlbs.
> 
> If your proposed changes do that (avoid the IPI/wakeup), great!
> (if not, there should be a way to do that)

My patch moves the atomic write from the intel idle
path into the tlb invalidation path, and gets rid of
the IPI.

Shouldn't be too hard to get that on top of Andy's
patches, once those have settled.

https://patchwork.kernel.org/patch/9307541/

[toc] | [prev] | [next] | [standalone]

#1654263

On Wed, May 31, 2017 at 5:44 AM, Rik van Riel <riel@redhat.com> wrote:
> On Tue, 2017-05-30 at 21:05 -0700, Arjan van de Ven wrote:
>> On 5/27/2017 9:56 AM, Andy Lutomirski wrote:
>> > On Sat, May 27, 2017 at 9:00 AM, Andy Lutomirski <luto@kernel.org>
>> > wrote:
>> > > On Sat, May 27, 2017 at 6:31 AM, kernel test robot
>> > > <xiaolong.ye@intel.com> wrote:
>> > > >
>> > > > FYI, we noticed the following commit:
>> > > >
>> > > > commit: e2a7dcce31f10bd7471b4245a6d1f2de344e7adf ("x86/mm:
>> > > > Rework lazy TLB to track the actual loaded mm")
>> > > > https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git
>> > > > x86/tlbflush_cleanup
>> > >
>> > > Ugh, there's an unpleasant interaction between this patch and
>> > > intel_idle.  I suspect that the intel_idle code in question is
>> > > either
>> > > wrong or pointless, but I want to investigate further.  Ingo, can
>> > > you
>> > > hold off on applying this patch?
>> >
>> > I think this is what's going on: intel_idle has an optimization and
>> > sometimes calls leave_mm().  This is a rather expensive way of
>> > working
>> > around x86 Linux's fairly weak lazy mm handling.  It also abuses
>> > the
>> > whole switch_mm state machine.  In particular, there's no guarantee
>> > that the mm is actually lazy at the time.  The old code didn't
>> > care,
>> > but the new code can oops.
>> >
>> > The short-term fix is to just reorder the code in leave_mm() to
>> > avoid the OOPS.
>>
>> fwiw the reason the code is in intel_idle is to avoid tlb flush IPIs
>> to idle cpus,
>> once the cpu goes into a deep enough idle state.  In the current
>> linux code,
>> that is done by no longer having the old TLB live on the CPU, by
>> switching to the neutral
>> kernel-only set of tlbs.
>>
>> If your proposed changes do that (avoid the IPI/wakeup), great!
>> (if not, there should be a way to do that)
>
> My patch moves the atomic write from the intel idle
> path into the tlb invalidation path, and gets rid of
> the IPI.
>
> Shouldn't be too hard to get that on top of Andy's
> patches, once those have settled.
>
> https://patchwork.kernel.org/patch/9307541/
>

I may beat you to it -- I'm trying out a total rewrite of lazy mode on
top of my series.

[toc] | [prev] | [next] | [standalone]

#1652060

* Andy Lutomirski <luto@kernel.org> wrote:

> On Sat, May 27, 2017 at 6:31 AM, kernel test robot
> <xiaolong.ye@intel.com> wrote:
> >
> > FYI, we noticed the following commit:
> >
> > commit: e2a7dcce31f10bd7471b4245a6d1f2de344e7adf ("x86/mm: Rework lazy TLB to track the actual loaded mm")
> > https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git x86/tlbflush_cleanup
> 
> Ugh, there's an unpleasant interaction between this patch and
> intel_idle.  I suspect that the intel_idle code in question is either
> wrong or pointless, but I want to investigate further.  Ingo, can you
> hold off on applying this patch?

Yeah, will wait for the next iteration of your series.

Thanks,

	Ingo

[toc] | [prev] | [next] | [standalone]

#1652135

On Sat, 2017-05-27 at 09:00 -0700, Andy Lutomirski wrote:
> On Sat, May 27, 2017 at 6:31 AM, kernel test robot
> <xiaolong.ye@intel.com> wrote:
> > 
> > FYI, we noticed the following commit:
> > 
> > commit: e2a7dcce31f10bd7471b4245a6d1f2de344e7adf ("x86/mm: Rework
> > lazy TLB to track the actual loaded mm")
> > https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git
> > x86/tlbflush_cleanup
> 
> Ugh, there's an unpleasant interaction between this patch and
> intel_idle.  I suspect that the intel_idle code in question is either
> wrong or pointless,

It would be pointless if my "make lazy TLB mode even lazier"
patch from last year had been applied.

https://patchwork.kernel.org/patch/9307541/

I guess I'll go re-send that and investigate the intel_idle
code in addition to that.

[toc] | [prev] | [next] | [standalone]

#1652168

On Sun, May 28, 2017 at 7:11 AM, Rik van Riel <riel@redhat.com> wrote:
> On Sat, 2017-05-27 at 09:00 -0700, Andy Lutomirski wrote:
>> On Sat, May 27, 2017 at 6:31 AM, kernel test robot
>> <xiaolong.ye@intel.com> wrote:
>> >
>> > FYI, we noticed the following commit:
>> >
>> > commit: e2a7dcce31f10bd7471b4245a6d1f2de344e7adf ("x86/mm: Rework
>> > lazy TLB to track the actual loaded mm")
>> > https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git
>> > x86/tlbflush_cleanup
>>
>> Ugh, there's an unpleasant interaction between this patch and
>> intel_idle.  I suspect that the intel_idle code in question is either
>> wrong or pointless,
>
> It would be pointless if my "make lazy TLB mode even lazier"
> patch from last year had been applied.
>
> https://patchwork.kernel.org/patch/9307541/
>
> I guess I'll go re-send that and investigate the intel_idle
> code in addition to that.

You might want to wait a bit until I'm done churning the code.  I
suspect that increased TLB laziness will look a bit different on top.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web