Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1490070 > unrolled thread

Re: thousands of kworker processes with 4.7.x and 4.8-rc*

Started byTomasz Chmielewski <mangoo@wpkg.org>
First post2016-09-23 15:30 +0200
Last post2016-09-27 06:00 +0200
Articles 8 — 3 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: thousands of kworker processes with 4.7.x and 4.8-rc* Tomasz Chmielewski <mangoo@wpkg.org> - 2016-09-23 15:30 +0200
    Re: thousands of kworker processes with 4.7.x and 4.8-rc* Mike Galbraith <umgwanakikbuti@gmail.com> - 2016-09-23 16:20 +0200
      Re: thousands of kworker processes with 4.7.x and 4.8-rc* Tomasz Chmielewski <mangoo@wpkg.org> - 2016-09-23 18:20 +0200
        Re: thousands of kworker processes with 4.7.x and 4.8-rc* Tomasz Chmielewski <mangoo@wpkg.org> - 2016-09-25 14:50 +0200
          Re: thousands of kworker processes with 4.7.x and 4.8-rc* Mike Galbraith <umgwanakikbuti@gmail.com> - 2016-09-25 19:30 +0200
            Re: thousands of kworker processes with 4.7.x and 4.8-rc* Mike Galbraith <umgwanakikbuti@gmail.com> - 2016-09-25 20:30 +0200
          Re: thousands of kworker processes with 4.7.x and 4.8-rc* Nikolay Borisov <n.borisov.lkml@gmail.com> - 2016-09-25 21:10 +0200
            Re: thousands of kworker processes with 4.7.x and 4.8-rc* Tomasz Chmielewski <mangoo@wpkg.org> - 2016-09-27 06:00 +0200

#1490070 — Re: thousands of kworker processes with 4.7.x and 4.8-rc*

FromTomasz Chmielewski <mangoo@wpkg.org>
Date2016-09-23 15:30 +0200
SubjectRe: thousands of kworker processes with 4.7.x and 4.8-rc*
Message-ID<skyIa-4aO-17@gated-at.bofh.it>
On 2016-09-19 16:08, Tomasz Chmielewski wrote:
> On several servers running 4.7.x and 4.8-rc6/7 kernels I'm seeing
> thousands of kworker processes.
> # ps auxf|grep -c kworker
> 2104
> Load average goes into hundreds on a pretty much idle server (biggest
> CPU and RAM consumers are probably SSHD with one user logged in and
> rsyslog writing ~1 line per minute):
> # uptime
>  06:58:56 up 26 min,  1 user,  load average: 146.11, 215.46, 105.70
> # uptime
>  06:59:48 up 26 min,  1 user,  load average: 305.20, 240.84, 120.25
> Sometimes seeing lots of them in "D" state:
> root     19474  0.0  0.0      0     0 ?        D    06:54   0:00  \_
> [kworker/0:208]
> root     19475  0.0  0.0      0     0 ?        D    06:54   0:00  \_
> [kworker/0:209]


I did some experiments to see when the problem first appeared. Thousands 
of kworker processes start to show up in 4.7.0-rc5.

kernel version | kworker count after boot
-------------------------------------------
4.6.3 	        37
4.6.4 	        47
4.6.5 	        46
4.6.6 	        49
4.6.7 	        49
4.7.0-rc1 	46
4.7.0-rc2 	49
4.7.0-rc3	45
4.7.0-rc4	47
4.7.0-rc5	1592
4.7.0-rc6	1714
4.7.0-rc7 	1955
4.7.0 	        2088
4.7.1 	        1222
4.7.2 	        1699
4.7.3           1446
4.7.4 	        1781
4.8-rc1 	(not tested)
4.8-rc2 	2012
4.8-rc3 	1696
4.8-rc4 	1210
4.8-rc5 	1890
4.8-rc6 	1657
4.8-rc7 	1647


Tomasz Chmielewski
https://lxadm.com

[toc] | [next] | [standalone]

#1490153

FromMike Galbraith <umgwanakikbuti@gmail.com>
Date2016-09-23 16:20 +0200
Message-ID<skzuy-4Kq-9@gated-at.bofh.it>
In reply to#1490070
On Fri, 2016-09-23 at 22:23 +0900, Tomasz Chmielewski wrote:
> On 2016-09-19 16:08, Tomasz Chmielewski wrote:
> > On several servers running 4.7.x and 4.8-rc6/7 kernels I'm seeing
> > thousands of kworker processes.
> > # ps auxf|grep -c kworker
> > 2104
> > Load average goes into hundreds on a pretty much idle server (biggest
> > CPU and RAM consumers are probably SSHD with one user logged in and
> > rsyslog writing ~1 line per minute):
> > # uptime
> >  06:58:56 up 26 min,  1 user,  load average: 146.11, 215.46, 105.70
> > # uptime
> >  06:59:48 up 26 min,  1 user,  load average: 305.20, 240.84, 120.25
> > Sometimes seeing lots of them in "D" state:
> > root     19474  0.0  0.0      0     0 ?        D    06:54   0:00  \_
> > [kworker/0:208]
> > root     19475  0.0  0.0      0     0 ?        D    06:54   0:00  \_
> > [kworker/0:209]
> 
> 
> I did some experiments to see when the problem first appeared. Thousands 
> of kworker processes start to show up in 4.7.0-rc5.
> 
> kernel version | kworker count after boot
> -------------------------------------------
> 4.6.3 > 	>         37
> 4.6.4 > 	>         47
> 4.6.5 > 	>         46
> 4.6.6 > 	>         49
> 4.6.7 > 	>         49
> 4.7.0-rc1 > 	> 46
> 4.7.0-rc2 > 	> 49
> 4.7.0-rc3> 	> 45
> 4.7.0-rc4> 	> 47
> 4.7.0-rc5> 	> 1592

Best bet would be to use 'git bisect' to locate the exact commit that
caused this, and post the bisection result along with your config.

AFAIK, nobody else is seeing this, is the kernel virgin source?

	-Mike

[toc] | [prev] | [next] | [standalone]

#1490300

FromTomasz Chmielewski <mangoo@wpkg.org>
Date2016-09-23 18:20 +0200
Message-ID<skBmF-69k-3@gated-at.bofh.it>
In reply to#1490153
On 2016-09-23 23:10, Mike Galbraith wrote:

>> I did some experiments to see when the problem first appeared. 
>> Thousands
>> of kworker processes start to show up in 4.7.0-rc5.
>> 
>> kernel version | kworker count after boot
>> -------------------------------------------
>> 4.6.3 > 	>         37
>> 4.6.4 > 	>         47
>> 4.6.5 > 	>         46
>> 4.6.6 > 	>         49
>> 4.6.7 > 	>         49
>> 4.7.0-rc1 > 	> 46
>> 4.7.0-rc2 > 	> 49
>> 4.7.0-rc3> 	> 45
>> 4.7.0-rc4> 	> 47
>> 4.7.0-rc5> 	> 1592
> 
> Best bet would be to use 'git bisect' to locate the exact commit that
> caused this, and post the bisection result along with your config.
> 
> AFAIK, nobody else is seeing this, is the kernel virgin source?

Yes, it's a kernel.org kernel.

I found some similar reports, though without much more info:

https://github.com/zfsonlinux/zfs/issues/5036 - kernel 4.7.2, initially 
attributed to ZFS on Linux, but then reproduced without ZFS

https://github.com/systemd/systemd/issues/4069 - kernel 4.7.2


I'll try to bisect.



Tomasz Chmielewski
https://lxadm.com

[toc] | [prev] | [next] | [standalone]

#1490885

FromTomasz Chmielewski <mangoo@wpkg.org>
Date2016-09-25 14:50 +0200
Message-ID<slh2x-6EB-1@gated-at.bofh.it>
In reply to#1490300
On 2016-09-25 18:29, Tomasz Chmielewski wrote:

>> I'll try to bisect.
> 
> OK, not a kernel regression, but some config change caused it.
> However, I'm not able to locate which change exactly.
> 
> I'm attaching two configs which I've tried with 4.7.3 - one results in
> thousands of kworkers, and the other doesn't. Also included a diff
> between them.
> 
> Any obvious changes I should try?

The problem is the allocator.

-CONFIG_SLUB=y
+CONFIG_SLAB=y


With SLUB, I'm getting a handful of kworker processes, as expected.

With SLAB, I'm getting thousands of kworker processes.


Not sure if that's expected behaviour or not.


Tomasz Chmielewski
https://lxadm.com

[toc] | [prev] | [next] | [standalone]

#1490937

FromMike Galbraith <umgwanakikbuti@gmail.com>
Date2016-09-25 19:30 +0200
Message-ID<sllpv-10M-1@gated-at.bofh.it>
In reply to#1490885
On Sun, 2016-09-25 at 21:40 +0900, Tomasz Chmielewski wrote:

> The problem is the allocator.
> 
> -CONFIG_SLUB=y
> +CONFIG_SLAB=y
> 
> 
> With SLUB, I'm getting a handful of kworker processes, as expected.
> 
> With SLAB, I'm getting thousands of kworker processes.
> 
> 
> Not sure if that's expected behaviour or not.

I seriously doubt 1500+ kworkers piling up is expected.

4.7.0-rc4       47
4.7.0-rc5       1592

Presuming you didn't switch to SLAB and/or change userspace all around
while testing, that would still indicate a kernel regression lurking
between rc4 and rc5.

	-Mike

[toc] | [prev] | [next] | [standalone]

#1490946

FromMike Galbraith <umgwanakikbuti@gmail.com>
Date2016-09-25 20:30 +0200
Message-ID<slmlA-1C2-7@gated-at.bofh.it>
In reply to#1490937
On Sun, 2016-09-25 at 19:21 +0200, Mike Galbraith wrote:
> On Sun, 2016-09-25 at 21:40 +0900, Tomasz Chmielewski wrote:
> 
> > The problem is the allocator.
> > 
> > -CONFIG_SLUB=y
> > +CONFIG_SLAB=y
> > 
> > 
> > With SLUB, I'm getting a handful of kworker processes, as expected.
> > 
> > With SLAB, I'm getting thousands of kworker processes.
> > 
> > 
> > Not sure if that's expected behaviour or not.
> 
> I seriously doubt 1500+ kworkers piling up is expected.
> 
> 4.7.0-rc4       47
> 4.7.0-rc5       1592
> 
> Presuming you didn't switch to SLAB and/or change userspace all around
> while testing, that would still indicate a kernel regression lurking
> between rc4 and rc5.

Nevermind, seems config change was how you met the thing.

FWIW, I turned on SLAB with a distro config here, and see no such
behavior.  If I could reproduce, I'd take the busted config back to
older kernels.

	-Mike

[toc] | [prev] | [next] | [standalone]

#1490955

FromNikolay Borisov <n.borisov.lkml@gmail.com>
Date2016-09-25 21:10 +0200
Message-ID<slmYh-240-1@gated-at.bofh.it>
In reply to#1490885

On 25.09.2016 15:40, Tomasz Chmielewski wrote:
> On 2016-09-25 18:29, Tomasz Chmielewski wrote:
> 
>>> I'll try to bisect.
>>
>> OK, not a kernel regression, but some config change caused it.
>> However, I'm not able to locate which change exactly.
>>
>> I'm attaching two configs which I've tried with 4.7.3 - one results in
>> thousands of kworkers, and the other doesn't. Also included a diff
>> between them.
>>
>> Any obvious changes I should try?
> 
> The problem is the allocator.
> 
> -CONFIG_SLUB=y
> +CONFIG_SLAB=y
> 
> 
> With SLUB, I'm getting a handful of kworker processes, as expected.
> 
> With SLAB, I'm getting thousands of kworker processes.
> 
> 
> Not sure if that's expected behaviour or not.


Why don't you sample the stacks of some of those kworker processes to
see if they are all executing a parituclar piece of work. That might
help you narrow down where they originate from. Cat multiple
/proc/$kworker-pid/stack files and see if a pattern emerges.

Regards,
Nikolay

> 
> 
> Tomasz Chmielewski
> https://lxadm.com
> 
>

[toc] | [prev] | [next] | [standalone]

#1491673

FromTomasz Chmielewski <mangoo@wpkg.org>
Date2016-09-27 06:00 +0200
Message-ID<slRIJ-4r1-7@gated-at.bofh.it>
In reply to#1490955
On 2016-09-26 04:07, Nikolay Borisov wrote:

>> Not sure if that's expected behaviour or not.
> 
> 
> Why don't you sample the stacks of some of those kworker processes to
> see if they are all executing a parituclar piece of work. That might
> help you narrow down where they originate from. Cat multiple
> /proc/$kworker-pid/stack files and see if a pattern emerges.

FYI, it was reproduced and bisected here (scroll to the bottom):

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1626564


Tomasz Chmielewski
https://lxadm.com

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web