Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1594131 > unrolled thread

Re: [GIT pull] CPU hotplug updates for 4.9

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [GIT pull] CPU hotplug updates for 4.9 Thomas Gleixner <tglx@linutronix.de> - 2017-03-07 12:20 +0100
    Re: [GIT pull] CPU hotplug updates for 4.9 Thomas Gleixner <tglx@linutronix.de> - 2017-03-08 10:50 +0100
      Re: [GIT pull] CPU hotplug updates for 4.9 Thomas Gleixner <tglx@linutronix.de> - 2017-03-09 11:30 +0100
        Re: [GIT pull] CPU hotplug updates for 4.9 Thomas Gleixner <tglx@linutronix.de> - 2017-03-09 18:50 +0100
          Re: [GIT pull] CPU hotplug updates for 4.9 Ingo Molnar <mingo@kernel.org> - 2017-03-10 08:30 +0100

#1594131 — Re: [GIT pull] CPU hotplug updates for 4.9

On Tue, 7 Mar 2017, Bart Van Assche wrote:
> On Mon, 2016-10-03 at 19:37 +0200, Thomas Gleixner wrote:
> > Yet another batch of cpu hotplug core updates and conversions:
> 
> Hello Thomas,
> 
> Although I'm not sure this behavior has been introduced by the changes in this
> pull request, since I started testing v4.11-rc[01] I ran several times into a
> cpuhp_issue_call() hang:

...

> INFO: task systemd-udevd:837 blocked for more than 480 seconds.
>       Tainted: G          I     4.11.0-rc1-dbg+ #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> systemd-udevd   D    0   837    542 0x00000104
> Call Trace:
>  __schedule+0x302/0xc30
>  schedule+0x38/0x90
>  schedule_timeout+0x255/0x490
>  wait_for_completion+0x103/0x170
>  cpuhp_issue_call+0xb9/0xe0
>  __cpuhp_setup_state+0xf6/0x180
>  pkg_temp_thermal_init+0x76/0x1000 [x86_pkg_temp_thermal]
>  do_one_initcall+0x3e/0x170
>  do_init_module+0x5a/0x1ed
>  load_module+0x2339/0x2a40
>  SYSC_finit_module+0xbc/0xf0
>  SyS_finit_module+0x9/0x10
>  do_syscall_64+0x57/0x140
>  entry_SYSCALL64_slow_path+0x25/0x25
> 
> Kernel v4.10 runs fine on the same system. From the dmidecode output:

That's odd. There was no change related to the hotplug stuff post 4.10
neither in the core nor in that particular driver.

I have no immediate clue what to look for aside of asking you whether you
could do a bisect between 4.10 and 4.11-rc1.

Thanks,

	tglx

[toc] | [next] | [standalone]

#1594989

Bart,

On Tue, 7 Mar 2017, Bart Van Assche wrote:
> On Tue, 2017-03-07 at 10:30 +0100, Thomas Gleixner wrote:
> > That's odd. There was no change related to the hotplug stuff post 4.10
> > neither in the core nor in that particular driver.
> > 
> > I have no immediate clue what to look for aside of asking you whether you
> > could do a bisect between 4.10 and 4.11-rc1.
> 
> I will keep trying to find the commit that introduced this behavior. The
> bisect I ran this morning yielded a commit that only changes ARM code which
> does not make sense since I ran the bisect on an x86 system. I guess that
> means that during one of the bisect steps I did not wait long enough to see
> whether or not the hang occurs and hence that I provided incorrect input to
> the bisect process.

Before you proceed with bisecting, could you try Linus head first,
especially commit:

fa3aa7a54fe6 ("jiffies: Revert bogus conversion of NSEC_PER_SEC to TICK_NSEC")

which fixes: 93825f2ec736 ("jiffies: Reuse TICK_NSEC instead of NSEC_PER_JIFFY")

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1595888

Bart,

On Wed, 8 Mar 2017, Bart Van Assche wrote:

> Sorry but the cpuhp_issue_call() still occurs with Linus' latest tree
> (commit ec3b93ae0bf4 / Merge branch 'x86-urgent-for-linus' of
> git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip).
> 
> This morning I discovered that the hang only occurs if I/O AT support is
> enabled in the BIOS. I had enabled I/O AT support in the BIOS on February
> 23rd. With I/O AT support enabled the following warning appears during boot:

That's interesting, but I have no idea how that's related.

> WARNING: CPU: 8 PID: 596 at drivers/dma/ioat/dca.c:342 ioat_dca_init+0x185/0x190 [ioatdma]
> ioatdma 0000:00:04.0: APICID_TAG_MAP set incorrectly by BIOS, disabling DCA

..

> In contrast with previous tests this morning I have been able to reproduce
> this hang with kernel v4.10. So it's not a kernel v4.10 regression. But the
> hang did not occur in a test with kernel v4.9.7. I assume this means that
> the regression got introduced between the v4.9 and v4.10 kernels.

Is it always the x86_pkg_thermal init which locks up?

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1596258

On Thu, 9 Mar 2017, Bart Van Assche wrote:
> On Thu, 2017-03-09 at 11:22 +0100, Thomas Gleixner wrote:
> > > In contrast with previous tests this morning I have been able to reproduce
> > > this hang with kernel v4.10. So it's not a kernel v4.10 regression. But the
> > > hang did not occur in a test with kernel v4.9.7. I assume this means that
> > > the regression got introduced between the v4.9 and v4.10 kernels.
> > 
> > Is it always the x86_pkg_thermal init which locks up?
> 
> Hello Thomas,
> 
> Apparently not. Here are a few other call traces that appeared in the system
> log:
> 
> INFO: task systemd-udevd:748 blocked for more than 480 seconds.
>       Tainted: G          IO    4.11.0-rc1-dbg+ #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> systemd-udevd   D    0   748    518 0x00000104
> Call Trace:
>  __schedule+0x302/0xc30
>  schedule+0x38/0x90
>  schedule_timeout+0x255/0x490
>  wait_for_completion+0x103/0x170
>  cpuhp_issue_call+0xb9/0xe0
>  __cpuhp_setup_state+0xf6/0x180
>  coretemp_init+0x8d/0x1000 [coretemp]
>  do_one_initcall+0x3e/0x170
>  do_init_module+0x5a/0x1ed
>  load_module+0x2339/0x2a40
>  SYSC_finit_module+0xbc/0xf0
>  SyS_finit_module+0x9/0x10
>  do_syscall_64+0x57/0x140
>  entry_SYSCALL64_slow_path+0x25/0x25
> Showing all locks held in the system:
> 2 locks held by khungtaskd/91:
>  #0:  (rcu_read_lock){......}, at: [<ffffffff8111a6f0>] watchdog+0xa0/0x5d0
>  #1:  (tasklist_lock){.+.?..}, at: [<ffffffff810bf36d>] debug_show_all_locks+0x3d/0x1a0
> 1 lock held by systemd-udevd/748:
>  #0:  (cpu_hotplug.dep_map){++++++}, at: [<ffffffff8106712d>] get_online_cpus+0x2d/0x80

Ok, so it's random. Now it would be interesting what the rest of the system
does when this happens. I still have no idea why that IOAT setting has any
influence.

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1596693

* Bart Van Assche <Bart.VanAssche@sandisk.com> wrote:

> On Thu, 2017-03-09 at 18:43 +0100, Thomas Gleixner wrote:
> > Ok, so it's random. Now it would be interesting what the rest of the system
> > does when this happens. I still have no idea why that IOAT setting has any
> > influence.
> 
> Hello Thomas,
> 
> The cpuhp_issue_call() hang happens during boot and before I get the chance to log
> in with ssh. Can you be more specific about what information you are looking for?
> 
> Anyway, the result of a new bisect I ran is as follows:
> * good: 631ddaba5905 ("Merge branches 'pm-sleep' and 'powercap'").
> * good: 80f1b3dea9d4 ("Merge branch 'device-properties'").
> * bad:  a67485d4bf97 ("Merge tag 'acpi-4.10-rc1' of
>   git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm").
> 
> Does this make sense to you?

The bisection information is really useful, if the failure mode is deterministic.

It would be nice to have more bisection points, the above range is about 5,000 
commits.

Thanks,

	Ingo

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web