Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1349922 > unrolled thread

4.4.3: OOPS when running "stress-ng --sock 5"

Back to article view | Back to linux.kernel

Contents

  4.4.3: OOPS when running "stress-ng --sock 5" Holger Schurig <holgerschurig@gmail.com> - 2016-03-04 08:50 +0100
    Re: 4.4.3: OOPS when running "stress-ng --sock 5" Holger Schurig <holgerschurig@gmail.com> - 2016-03-07 10:20 +0100
    Re: 4.4.3: OOPS when running "stress-ng --sock 5" Holger Schurig <holgerschurig@gmail.com> - 2016-03-07 11:10 +0100
    Re: 4.4.3: OOPS when running "stress-ng --sock 5" Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> - 2016-03-07 12:10 +0100

#1349922 — 4.4.3: OOPS when running "stress-ng --sock 5"

Hi,

on my system I can reproduce reliably a kernel OOPS when I run stress-ng
("apt-get install stress-ng"). Any help on how to track this down would
be appreciated, networking code is outside of my comfort zone (I'm just
a dilettante at device drivers ...).

It takes only a minute or two to get the OOPS:

root@ptxc:~# stress-ng --sock 5
stress-ng: info: [361] dispatching hogs: 0 I/O-Sync, 0 CPU, 0 VM-mmap, 0 HDD-Write, 0 Fork, 0 Context-switch, 0 Pipe, 0 Cache, 5 Socket, 0 Yield, 0 Fallocate, 0 Flock, 0 Affinity, 0 Timer, 0 Dentry, 0 Urandom, 0 Float, 0 Int, 0 Semaphore, 0 Open, 0 SigQueue, 0 Poll
Unable to handle kernel NULL pointer dereference at virtual address 00000104
pgd = ee0d8000
[00000104] *pgd=3e17c831, *pte=00000000, *ppte=00000000
Internal error: Oops: 817 [#1] SMP ARM
Modules linked in: bnep smsc95xx usbnet mii usbhid imx_sdma flexcan btusb btrtl btbcm btintel bluetooth
CPU: 2 PID: 362 Comm: stress-ng-socke Not tainted 4.4.3 #1
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
task: eeb30a00 ti: eea0a000 task.ti: eea0a000
PC is at __rmqueue+0x74/0x308
LR is at 0x3
pc : [<c00972fc>]    lr : [<00000003>]    psr: 60030093
sp : eea0bc08  ip : 00000200  fp : eea0bc54
r10: efd80b14  r9 : 00000008  r8 : 00000000
r7 : 00000003  r6 : 00000000  r5 : c050bff8  r4 : 00000100
r3 : c05ce36c  r2 : 0000006c  r1 : 00000200  r0 : 00000100
Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
Control: 10c5387d  Table: 3e0d804a  DAC: 00000051
Process stress-ng-socke (pid: 362, stack limit = 0xeea0a210)
Stack: (0xeea0bc08 to 0xeea0c000)
bc00:                   0000ffff c05ca780 ed93dd80 ed93dd80 eea0bc5c c05ce280
bc20: c03d5838 c03d3b00 c05b04f8 eea0bd5c c050bff8 c050bfe4 c050bfe4 ed93de38
bc40: 00000008 c05ce280 eea0bcec eea0bc58 c0097cb4 c0097294 00000141 0002c26d
bc60: c03d59c4 00000018 c05ced00 c05b0100 ffffacd4 c0439bb0 0000000a c05d19c0
bc80: 00000000 c05b0080 00000100 c05ce490 c05ced08 c05ce3a8 c05cee15 00000128
bca0: 00000141 020252c0 00000000 fffffff8 00000000 eea0bd5c 60030013 00000003
bcc0: eea0bcf4 020052c0 00000003 c05ced00 0000ffcb ed93de38 eea0be84 00000000
bce0: eea0bda4 eea0bcf0 c0098084 c009759c c006caf8 80100010 0fcfc2fc 40030013
bd00: eea0bd24 ed93dd80 ed93dd80 00040000 ed999e00 ed93dd80 eea0bd8c eea0bd28
bd20: c03ee130 c03ebcac 00000002 ef001c00 00000000 024102c0 00000000 000346db
bd40: c05b0100 00000000 00000002 ed93e114 00000005 00000000 00000000 c05ced00
bd60: 00000000 c05ce280 00000000 00000000 00000000 00000000 eea0be84 eeb30eb4
bd80: 024000c0 000005d0 0000ffcb ed93de38 eea0be84 00000000 eea0bdbc eea0bda8
bda0: c0389650 c0097fb8 ed93dd80 ed93dd80 eea0bdd4 eea0bdc0 c03896c8 c03895ec
bdc0: ed999e00 ed93dd80 eea0be4c eea0bdd8 c03e14d4 c03896b8 0000ffcb 00000014
bde0: 000014bf 00000001 eeb30eb4 00000001 00000001 00000000 eea0a018 00000000
be00: eeb30eb4 00000001 0000ffcb 00000560 c0434ca8 0000ffcb 7fffffff 7fffffff
be20: ed958000 ed93dd80 00000000 00000000 00000000 eea6c000 eea0a000 00000000
be40: eea0be6c eea0be50 c0407cbc c03e131c ee98c1a0 ed93dd80 eea0beec 00000000
be60: eea0be7c eea0be70 c0385784 c0407c34 eea0bed4 eea0be80 c0385820 c0385774
be80: c00e6220 00000000 00000000 00000001 00000560 000005d0 eea0bee4 00000001
bea0: 00000000 00000000 00000000 eea0bf00 00000000 eea6c000 eea0bf80 00000000
bec0: 00000000 c000fae4 eea0bf3c eea0bed8 c00c9b2c c03857a0 00000b30 00000004
bee0: eea0bf1c bea359bc 00000b30 00000001 00000000 00000b30 eea0bee4 00000001
bf00: eea6c000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
bf20: eea6c000 00000b30 bea359bc eea0bf80 eea0bf4c eea0bf40 c00c9b84 c00c9ab0
bf40: eea0bf7c eea0bf50 c00ca330 c00c9b5c 00000000 00000000 eea0bf7c eea6c000
bf60: eea6c000 00000b30 bea359bc c000fae4 eea0bfa4 eea0bf80 c00cac0c c00ca2a4
bf80: 00000000 00000000 00000004 0002a1e8 b6f6f140 00000004 00000000 eea0bfa8
bfa0: c000f920 c00cabcc 00000004 0002a1e8 00000004 bea359bc 00000b30 bea379bc
bfc0: 00000004 0002a1e8 b6f6f140 00000004 00000b30 0000016f 0002a1f0 00000003
bfe0: 00000000 bea358f4 00014a57 b6eaa4d6 40030030 00000004 00000000 00000000
Backtrace: 
[<c0097288>] (__rmqueue) from [<c0097cb4>] (get_page_from_freelist+0x724/0x914)
 r10:c05ce280 r9:00000008 r8:ed93de38 r7:c050bfe4 r6:c050bfe4 r5:c050bff8
 r4:eea0bd5c
[<c0097590>] (get_page_from_freelist) from [<c0098084>] (__alloc_pages_nodemask+0xd8/0x898)
 r10:00000000 r9:eea0be84 r8:ed93de38 r7:0000ffcb r6:c05ced00 r5:00000003
 r4:020052c0
[<c0097fac>] (__alloc_pages_nodemask) from [<c0389650>] (skb_page_frag_refill+0x70/0xcc)
 r10:00000000 r9:eea0be84 r8:ed93de38 r7:0000ffcb r6:000005d0 r5:024000c0
 r4:eeb30eb4
[<c03895e0>] (skb_page_frag_refill) from [<c03896c8>] (sk_page_frag_refill+0x1c/0x74)
 r5:ed93dd80 r4:ed93dd80
[<c03896ac>] (sk_page_frag_refill) from [<c03e14d4>] (tcp_sendmsg+0x1c4/0xa58)
 r5:ed93dd80 r4:ed999e00
[<c03e1310>] (tcp_sendmsg) from [<c0407cbc>] (inet_sendmsg+0x94/0xc8)
 r10:00000000 r9:eea0a000 r8:eea6c000 r7:00000000 r6:00000000 r5:00000000
 r4:ed93dd80
[<c0407c28>] (inet_sendmsg) from [<c0385784>] (sock_sendmsg+0x1c/0x2c)
 r5:00000000 r4:eea0beec
[<c0385768>] (sock_sendmsg) from [<c0385820>] (sock_write_iter+0x8c/0xc0)
[<c0385794>] (sock_write_iter) from [<c00c9b2c>] (new_sync_write+0x88/0xac)
 r8:c000fae4 r7:00000000 r6:00000000 r5:eea0bf80 r4:eea6c000
[<c00c9aa4>] (new_sync_write) from [<c00c9b84>] (__vfs_write+0x34/0x40)
 r7:eea0bf80 r6:bea359bc r5:00000b30 r4:eea6c000
[<c00c9b50>] (__vfs_write) from [<c00ca330>] (vfs_write+0x98/0x16c)
[<c00ca298>] (vfs_write) from [<c00cac0c>] (SyS_write+0x4c/0xa8)
 r8:c000fae4 r7:bea359bc r6:00000b30 r5:eea6c000 r4:eea6c000
[<c00cabc0>] (SyS_write) from [<c000f920>] (ret_fast_syscall+0x0/0x3c)
 r7:00000004 r6:b6f6f140 r5:0002a1e8 r4:00000004
Code: e3a04c01 e157000e e1a02102 e3a0cc02 (e5801004) 
---[ end trace 5eaad8c38456d9bc ]---

[toc] | [next] | [standalone]

#1351418

I compared my config with imx_v6_v7_defconfig which didn't segfault.

After I turned on CONFIG_SWAP, my segfault vanished.

I did turn off CONFIG_SWAP because my device only has SD-Card and eMMC.
So I never intended to create a swap partition. And thought "why compile
it in the kernel when I never use it?".

But it seems the kernel is instable with this setting.

So we have a potential denial-of-service in kernels compiled without
CONFIG_SWAP, don't we? At least when it comes to skb handling.

Other memory tests never showed anything weird, and my system is running
X11 with some Qt applications as well as Java applications since about a
year without trouble. During all this time without CONFIG_SWAP.

[toc] | [prev] | [next] | [standalone]

#1351485

I have rejoiced prematurely, it just now took way longer I hit the
segfault. Previously 1m or at max 2m was enough.

root@ptxc:~# stress-ng --sock 20
stress-ng: info: [359] dispatching hogs: 0 I/O-Sync, 0 CPU, 0 VM-mmap, 0 HDD-Write, 0 Fork, 0 Context-switch, 0 Pipe, 0 Cache, 20 Socket, 0 Yield, 0 Fallocate, 0 Flock, 0 Affinity, 0 Timer, 0 Dentry, 0 Urandom, 0 Float, 0 Int, 0 Semaphore, 0 Open, 0 SigQueue, 0 Poll
[   42.253392] random: nonblocking pool is initialized
[  567.649965] Unable to handle kernel NULL pointer dereference at virtual address 00000104
[  567.658087] pgd = ee11c000
[  567.660797] [00000104] *pgd=3eaf4831, *pte=00000000, *ppte=00000000
[  567.667112] Internal error: Oops: 817 [#1] SMP ARM
[  567.671904] Modules linked in: bnep btusb btrtl btbcm btintel bluetooth smsc95xx usbnet usbhid mii imx_sdma flexcan
[  567.682514] CPU: 1 PID: 383 Comm: stress-ng-socke Not tainted 4.4.4PTXC #3
[  567.689390] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[  567.695920] task: ed9f9e00 ti: eeaf0000 task.ti: eeaf0000
[  567.701333] PC is at __rmqueue+0x74/0x308
[  567.705346] LR is at 0x3
[  567.707882] pc : [<c00973cc>]    lr : [<00000003>]    psr: 60030093
[  567.707882] sp : eeaf1c00  ip : 00000200  fp : eeaf1c4c
[  567.719359] r10: efd5f514  r9 : 00000008  r8 : 00000000
[  567.724585] r7 : 00000003  r6 : 00000000  r5 : c051343c  r4 : 00000100
[  567.731113] r3 : c05d6e2c  r2 : 0000006c  r1 : 00000200  r0 : 00000100
[  567.737643] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
[  567.744866] Control: 10c5387d  Table: 3e11c04a  DAC: 00000051
[  567.750612] Process stress-ng-socke (pid: 383, stack limit = 0xeeaf0210)
[  567.757314] Stack: (0xeeaf1c00 to 0xeeaf2000)
[  567.761677] 1c00: 0000ffff c05d3200 ed976880 ed976880 eeaf1c54 c05d6d40 c03dc720 c03da9e8
[  567.769859] 1c20: 20030013 eeaf1d54 c051343c c0513428 c0513428 6104de4b 00000008 c05d6d40
[  567.778040] 1c40: eeaf1ce4 eeaf1c50 c0097d84 c0097364 00000141 0002a602 ef7bc2c0 00000018
[  567.786220] 1c60: c05d77c0 c05d77c0 00000000 c05d6d40 00000000 00000000 00000000 00000000
[  567.794401] 1c80: 00000100 c05d6f50 c05d77c8 c05d6e68 c05d78d5 00000128 00000141 020252c0
[  567.802582] 1ca0: 00000000 fffffff8 00000000 eeaf1d54 60030013 00000003 00000000 020052c0
[  567.810762] 1cc0: 00000003 c05d77c0 0000ffcb 6104de4b eeaf1e84 00000000 eeaf1d9c eeaf1ce8
[  567.818942] 1ce0: c0098154 c009766c c006cb38 00100010 60ecb9db 40030013 eeaf1d1c ed976880
[  567.827123] 1d00: ed976880 00040000 eacecc00 ed976880 eeaf1d84 eeaf1d20 c03f4d44 c03f2c30
[  567.835304] 1d20: 00000002 ef001c00 00000000 024102c0 00000000 000346db c05b8100 00000000
[  567.843484] 1d40: 00000002 ed976c14 00000002 00000000 00000000 c05d77c0 00000000 c05d6d40
[  567.851664] 1d60: 00000000 00000000 00000000 00000000 eeaf1e84 ed9fa2b4 024000c0 00000fb0
[  567.859845] 1d80: 0000ffcb 6104de4b eeaf1e84 00000000 eeaf1db4 eeaf1da0 c03901d4 c0098088
[  567.868025] 1da0: ed976880 ed976880 eeaf1dcc eeaf1db8 c039024c c0390170 eaf60600 ed976880
[  567.876206] 1dc0: eeaf1e4c eeaf1dd0 c03e83fc c039023c 0000ffcb 00000001 23c09b2d 000017c8
[  567.884386] 1de0: 00000001 eeaf1e8c c05b8bb4 eeaf0000 00000001 00000000 ed9fa2b4 00000000
[  567.892566] 1e00: 00000001 ed976938 0000ffcb 00000c90 c004c6d8 0000ffcb 7fffffff c00473e8
[  567.900747] 1e20: ed983c80 ed976880 00000000 00000000 00000000 eea03780 eeaf0000 00000000
[  567.908927] 1e40: eeaf1e6c eeaf1e50 c040ec5c c03e8228 00000000 00000000 eeaf1eec 00000000
[  567.917107] 1e60: eeaf1e7c eeaf1e70 c038c308 c040ebd4 eeaf1ed4 eeaf1e80 c038c3a4 c038c2f8
[  567.925287] 1e80: c0050004 00000000 00000000 00000001 00000c90 00000fb0 eeaf1ee4 00000001
[  567.933467] 1ea0: 00000000 00000000 00000000 eeaf1f00 afb50401 eea03780 eeaf1f80 00000000
[  567.941647] 1ec0: 00000000 c000fae4 eeaf1f3c eeaf1ed8 c00cfe84 c038c324 00001c40 ef0a4000
[  567.949828] 1ee0: eeaf1f14 bef469bc 00001c40 00000001 00000000 00001c40 eeaf1ee4 00000001
[  567.958008] 1f00: eea03780 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  567.966189] 1f20: eea03780 00001c40 bef469bc eeaf1f80 eeaf1f4c eeaf1f40 c00cfedc c00cfe08
[  567.974369] 1f40: eeaf1f7c eeaf1f50 c00d0688 c00cfeb4 00000000 00000000 eeaf1f7c eea03780
[  567.982550] 1f60: eea03780 00001c40 bef469bc c000fae4 eeaf1fa4 eeaf1f80 c00d0f64 c00d05fc
[  567.990730] 1f80: 00000000 00000000 00000004 0002a1e8 b6f94598 00000004 00000000 eeaf1fa8
[  567.998910] 1fa0: c000f920 c00d0f24 00000004 0002a1e8 00000004 bef469bc 00001c40 bef489bc
[  568.007091] 1fc0: 00000004 0002a1e8 b6f94598 00000004 00001c40 0000018d 0002a1f0 00000003
[  568.015271] 1fe0: 00000000 bef468f4 00014a57 b6ecf4d6 40030030 00000004 00000000 00000000
[  568.023447] Backtrace: 
[  568.025916] [<c0097358>] (__rmqueue) from [<c0097d84>] (get_page_from_freelist+0x724/0x914)
[  568.034267]  r10:c05d6d40 r9:00000008 r8:6104de4b r7:c0513428 r6:c0513428 r5:c051343c
[  568.042164]  r4:eeaf1d54
[  568.044716] [<c0097660>] (get_page_from_freelist) from [<c0098154>] (__alloc_pages_nodemask+0xd8/0x898)
[  568.054108]  r10:00000000 r9:eeaf1e84 r8:6104de4b r7:0000ffcb r6:c05d77c0 r5:00000003
[  568.062004]  r4:020052c0
[  568.064566] [<c009807c>] (__alloc_pages_nodemask) from [<c03901d4>] (skb_page_frag_refill+0x70/0xcc)
[  568.073697]  r10:00000000 r9:eeaf1e84 r8:6104de4b r7:0000ffcb r6:00000fb0 r5:024000c0
[  568.081593]  r4:ed9fa2b4
[  568.084145] [<c0390164>] (skb_page_frag_refill) from [<c039024c>] (sk_page_frag_refill+0x1c/0x74)
[  568.093016]  r5:ed976880 r4:ed976880
[  568.096627] [<c0390230>] (sk_page_frag_refill) from [<c03e83fc>] (tcp_sendmsg+0x1e0/0xa68)
[  568.104890]  r5:ed976880 r4:eaf60600
[  568.108507] [<c03e821c>] (tcp_sendmsg) from [<c040ec5c>] (inet_sendmsg+0x94/0xc8)
[  568.115989]  r10:00000000 r9:eeaf0000 r8:eea03780 r7:00000000 r6:00000000 r5:00000000
[  568.123884]  r4:ed976880
[  568.126436] [<c040ebc8>] (inet_sendmsg) from [<c038c308>] (sock_sendmsg+0x1c/0x2c)
[  568.134004]  r5:00000000 r4:eeaf1eec
[  568.137610] [<c038c2ec>] (sock_sendmsg) from [<c038c3a4>] (sock_write_iter+0x8c/0xc0)
[  568.145448] [<c038c318>] (sock_write_iter) from [<c00cfe84>] (new_sync_write+0x88/0xac)
[  568.153450]  r8:c000fae4 r7:00000000 r6:00000000 r5:eeaf1f80 r4:eea03780
[  568.160221] [<c00cfdfc>] (new_sync_write) from [<c00cfedc>] (__vfs_write+0x34/0x40)
[  568.167877]  r7:eeaf1f80 r6:bef469bc r5:00001c40 r4:eea03780
[  568.173592] [<c00cfea8>] (__vfs_write) from [<c00d0688>] (vfs_write+0x98/0x16c)
[  568.180905] [<c00d05f0>] (vfs_write) from [<c00d0f64>] (SyS_write+0x4c/0xa8)
[  568.187953]  r8:c000fae4 r7:bef469bc r6:00001c40 r5:eea03780 r4:eea03780
[  568.194730] [<c00d0f18>] (SyS_write) from [<c000f920>] (ret_fast_syscall+0x0/0x3c)
[  568.202299]  r7:00000004 r6:b6f94598 r5:0002a1e8 r4:00000004
[  568.208014] Code: e3a04c01 e157000e e1a02102 e3a0cc02 (e5801004) 
[  568.214113] ---[ end trace a72ad5170492b3b2 ]---

[toc] | [prev] | [next] | [standalone]

#1351540

Holger Schurig wrote:
> So I did an "arm-linux-gnueabihf-objdump -Sgd linux/vmlinux", not sure
> if that helps:
> 
> c00972ec <__rmqueue>:
>  * Do the hard work of removing an element from the buddy allocator.
>  * Call me with the zone->lock already held.
>  */
> static struct page *__rmqueue(struct zone *zone, unsigned int order,
>                                 int migratetype, gfp_t gfp_flags)
> {
> c00972ec:       e1a0c00d        mov     ip, sp
> c00972f0:       e92ddff0        push    {r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc}
> c00972f4:       e24cb004        sub     fp, ip, #4
> c00972f8:       e24dd024        sub     sp, sp, #36     ; 0x24
>         unsigned int current_order;
>         struct free_area *area;
>         struct page *page;
> 
>         /* Find a page of the appropriate size in the preferred list */
>         for (current_order = order; current_order < MAX_ORDER; ++current_order) {
> c00972fc:       e351000a        cmp     r1, #10
>  * Do the hard work of removing an element from the buddy allocator.
>  * Call me with the zone->lock already held.
>  */
> 
I tried on x86_64 but I could not reproduce it.
Thus, we need to examine this problem using your environment.

I didn't notice that c00972ec is __rmqueue+0x0.
Actual line number to examine is c0097360 ("pc" register) which is __rmqueue+0x74.
Please show us line number and assembly code around c0097360.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web