Groups > linux.kernel > #1571518 > unrolled thread

keys: GPF in request_key

Started by	Dmitry Vyukov <dvyukov@google.com>
First post	2017-02-01 13:30 +0100
Last post	2017-02-01 18:20 +0100
Articles	12 — 2 participants

Back to article view | Back to linux.kernel

  keys: GPF in request_key Dmitry Vyukov <dvyukov@google.com> - 2017-02-01 13:30 +0100
    Re: keys: GPF in request_key David Howells <dhowells@redhat.com> - 2017-02-01 14:20 +0100
      Re: keys: GPF in request_key Dmitry Vyukov <dvyukov@google.com> - 2017-02-01 14:30 +0100
    Re: keys: GPF in request_key David Howells <dhowells@redhat.com> - 2017-02-01 14:30 +0100
    Re: keys: GPF in request_key David Howells <dhowells@redhat.com> - 2017-02-01 14:50 +0100
      Re: keys: GPF in request_key Dmitry Vyukov <dvyukov@google.com> - 2017-02-01 15:10 +0100
        Re: keys: GPF in request_key David Howells <dhowells@redhat.com> - 2017-02-01 16:00 +0100
          Re: keys: GPF in request_key Dmitry Vyukov <dvyukov@google.com> - 2017-02-01 16:30 +0100
            Re: keys: GPF in request_key David Howells <dhowells@redhat.com> - 2017-02-01 16:50 +0100
              Re: keys: GPF in request_key Dmitry Vyukov <dvyukov@google.com> - 2017-02-01 17:00 +0100
                Re: keys: GPF in request_key David Howells <dhowells@redhat.com> - 2017-02-01 18:00 +0100
                  Re: keys: GPF in request_key Dmitry Vyukov <dvyukov@google.com> - 2017-02-01 18:20 +0100

#1571518 — keys: GPF in request_key

From	Dmitry Vyukov <dvyukov@google.com>
Date	2017-02-01 13:30 +0100
Subject	keys: GPF in request_key
Message-ID	<t62cW-5Vf-33@gated-at.bofh.it>

Hello,

I am seeing the following crashes in request_key while running
syzkaller fuzzer. This is observed on upstream commits
566cf877a1fcb6d6dc0126b076aad062054c2637,
f9a42e0d58cf0fe3d902e63d4582f2ea4cd2bb8b and
a2ca3d617944417e9dd5f09fc8a4549cda115f4f. Unfortunately this is not
reproducible (probably due to global nature of keys).

BUG: unable to handle kernel paging request at fffffbfff9453f4d
IP: __key_link_begin+0x35/0x2d0 security/keys/keyring.c:1107
PGD 21fff4067
PUD 21fff3067
PMD 0
Oops: 0000 [#1] SMP KASAN
Dumping ftrace buffer:
   (ftrace buffer empty)
Modules linked in:
CPU: 0 PID: 3682 Comm: syz-executor5 Not tainted 4.10.0-rc6+ #30
Hardware name: Google Google Compute Engine/Google Compute Engine,
BIOS Google 01/01/2011
task: ffff8801ca97c740 task.stack: ffff8801c1160000
RIP: 0010:__key_link_begin+0x35/0x2d0 security/keys/keyring.c:1107
RSP: 0018:ffff8801c1167b88 EFLAGS: 00010a06
RAX: dffffc0000000000 RBX: ffff8801d772bac0 RCX: ffffc90004787000
RDX: 1ffffffff9453f4d RSI: ffffffff8209b8df RDI: ffffffffca29fa68
RBP: ffff8801c1167bb8 R08: ffff8801dbe1cfa0 R09: 0000000000000001
R10: ffffe8ffffc32628 R11: 1ffff1003b7c39f8 R12: ffffffffca29fa58
R13: ffff8801d772bac0 R14: dffffc0000000000 R15: ffff8801c1167bf8
FS:  00007f4ae1cdb700(0000) GS:ffff8801dbe00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: fffffbfff9453f4d CR3: 00000001db964000 CR4: 00000000001406f0
DR0: 0000000020000000 DR1: 0000000020001000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
Call Trace:
 key_link+0x90/0x230 security/keys/keyring.c:1258
 request_key_and_link+0x2d8/0x1150 security/keys/request_key.c:549
 SYSC_request_key security/keys/keyctl.c:213 [inline]
 SyS_request_key+0x1a8/0x370 security/keys/keyctl.c:158
 entry_SYSCALL_64_fastpath+0x1f/0xc2
RIP: 0033:0x44fac9
RSP: 002b:00007f4ae1cdab58 EFLAGS: 00000212 ORIG_RAX: 00000000000000f9
RAX: ffffffffffffffda RBX: 0000000020ebe000 RCX: 000000000044fac9
RDX: 0000000020b1b000 RSI: 0000000020ebe000 RDI: 00000000209b8ffb
RBP: 00000000209b8ffb R08: 0000000000000000 R09: 0000000000000000
R10: fffffffffffffffb R11: 0000000000000212 R12: 0000000000708000
R13: 00000000ffffffff R14: 00000000202f7000 R15: 0000000000000000
Code: 41 54 49 89 f4 53 49 89 d7 48 89 fb 48 83 ec 08 e8 d1 50 67 ff
49 8d 7c 24 10 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <80>
3c 02 00 0f 85 35 02 00 00 49 83 7c 24 10 00 0f 84 bb 01 00
RIP: __key_link_begin+0x35/0x2d0 security/keys/keyring.c:1107 RSP:
ffff8801c1167b88
CR2: fffffbfff9453f4d
---[ end trace ccc5ddd8687a3f43 ]---


general protection fault: 0000 [#1] SMP KASAN
Dumping ftrace buffer:
   (ftrace buffer empty)
Modules linked in:
CPU: 1 PID: 19539 Comm: syz-executor7 Not tainted 4.10.0-rc6+ #32
Hardware name: Google Google Compute Engine/Google Compute Engine,
BIOS Google 01/01/2011
task: ffff88017e7fa200 task.stack: ffff8801d1db8000
RIP: 0010:__key_link_begin+0x35/0x2d0 security/keys/keyring.c:1107
RSP: 0018:ffff8801d1dbfb88 EFLAGS: 00010202
RAX: dffffc0000000000 RBX: ffff8801d76f0ac0 RCX: ffffc900012be000
RDX: 000000000fa33dad RSI: ffffffff8209b8bf RDI: 000000007d19ed68
RBP: ffff8801d1dbfbb8 R08: ffff8801dbf1cfa0 R09: 0000000000000001
R10: ffffe8ffffd55c18 R11: 1ffff1003b7e39f8 R12: 000000007d19ed58
R13: ffff8801d76f0ac0 R14: dffffc0000000000 R15: ffff8801d1dbfbf8
FS:  00007f6815118700(0000) GS:ffff8801dbf00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020b22000 CR3: 00000001c6dc1000 CR4: 00000000001406e0
DR0: 0000000020000000 DR1: 0000000020000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
Call Trace:
 key_link+0x90/0x230 security/keys/keyring.c:1258
 request_key_and_link+0x2d8/0x1150 security/keys/request_key.c:549
 SYSC_request_key security/keys/keyctl.c:213 [inline]
 SyS_request_key+0x1a8/0x370 security/keys/keyctl.c:158
 entry_SYSCALL_64_fastpath+0x1f/0xc2
RIP: 0033:0x44fac9
RSP: 002b:00007f6815117b58 EFLAGS: 00000212 ORIG_RAX: 00000000000000f9
RAX: ffffffffffffffda RBX: 0000000020ebe000 RCX: 000000000044fac9
RDX: 0000000020b1b000 RSI: 0000000020ebe000 RDI: 00000000209b8ffb
RBP: 00000000209b8ffb R08: 0000000000000000 R09: 0000000000000000
R10: fffffffffffffffb R11: 0000000000000212 R12: 0000000000708000
R13: 0000000080000001 R14: 00000000004c2a80 R15: 0000000000000000
Code: 41 54 49 89 f4 53 49 89 d7 48 89 fb 48 83 ec 08 e8 d1 50 67 ff
49 8d 7c 24 10 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <80>
3c 02 00 0f 85 35 02 00 00 49 83 7c 24 10 00 0f 84 bb 01 00
RIP: __key_link_begin+0x35/0x2d0 security/keys/keyring.c:1107 RSP:
ffff8801d1dbfb88
---[ end trace 75fd8b40181652fb ]---


BUG: unable to handle kernel paging request at fffffbfffae46d45
IP: __key_link_begin+0x35/0x2d0 security/keys/keyring.c:1107
PGD 21fff4067
PUD 21fff3067
PMD 0
Oops: 0000 [#1] SMP KASAN
Dumping ftrace buffer:
   (ftrace buffer empty)
Modules linked in:
CPU: 0 PID: 16792 Comm: syz-executor1 Not tainted 4.10.0-rc6 #29
Hardware name: Google Google Compute Engine/Google Compute Engine,
BIOS Google 01/01/2011
task: ffff8801c92f2480 task.stack: ffff8801d1e50000
RIP: 0010:__key_link_begin+0x35/0x2d0 security/keys/keyring.c:1107
RSP: 0018:ffff8801d1e57840 EFLAGS: 00010a02
RAX: dffffc0000000000 RBX: ffff8801d770c940 RCX: ffffc90001f07000
RDX: 1ffffffffae46d45 RSI: ffffffff8209b8df RDI: ffffffffd7236a28
RBP: ffff8801d1e57870 R08: ffff8801dbe1cfa0 R09: 0000000000000001
R10: ffffe8ffffc38e88 R11: 1ffff1003b7c39f8 R12: ffffffffd7236a18
R13: ffff8801d770c940 R14: dffffc0000000000 R15: ffff8801d1e578b0
FS:  00007f7f98ebe700(0000) GS:ffff8801dbe00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: fffffbfffae46d45 CR3: 00000001c97f7000 CR4: 00000000001426f0
Call Trace:
 key_link+0x90/0x230 security/keys/keyring.c:1258
 request_key_and_link+0x2d8/0x1150 security/keys/request_key.c:549
 SYSC_request_key security/keys/keyctl.c:213 [inline]
 SyS_request_key+0x1a8/0x370 security/keys/keyctl.c:158
 do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:280
 entry_SYSCALL64_slow_path+0x25/0x25
RIP: 0033:0x44fa69
RSP: 002b:00007f7f98ebdb58 EFLAGS: 00000212 ORIG_RAX: 00000000000000f9
RAX: ffffffffffffffda RBX: 0000000000708000 RCX: 000000000044fa69
RDX: 0000000020b1b000 RSI: 0000000020ebe000 RDI: 00000000209b8ffb
RBP: 000000000000034e R08: 0000000000000000 R09: 0000000000000000
R10: fffffffffffffffb R11: 0000000000000212 R12: 00000000209b8ffb
R13: 0000000020ebe000 R14: 0000000020b1b000 R15: 0000000000000000
Code: 41 54 49 89 f4 53 49 89 d7 48 89 fb 48 83 ec 08 e8 d1 50 67 ff
49 8d 7c 24 10 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <80>
3c 02 00 0f 85 35 02 00 00 49 83 7c 24 10 00 0f 84 bb 01 00
RIP: __key_link_begin+0x35/0x2d0 security/keys/keyring.c:1107 RSP:
ffff8801d1e57840
CR2: fffffbfffae46d45
---[ end trace 70f503a26e161643 ]---


The line causes the crash is:
    BUG_ON(index_key->desc_len == 0);

The addresses that the line tried to access are:

RDI: ffffffffca29fa68
RDI: ffffffffd7236a28
RDI: 000000007d19ed68

The first two point to modules range and the last one is somewhere is
userspace (if not KASAN, it could actually succeed). Looking at these
values I can suggest that it is a random int32 sign-extended to
pointer.

[toc] | [next] | [standalone]

#1571548

From	David Howells <dhowells@redhat.com>
Date	2017-02-01 14:20 +0100
Message-ID	<t62Zk-6rB-15@gated-at.bofh.it>
In reply to	#1571518

Dmitry Vyukov <dvyukov@google.com> wrote:

> Code: 41 54 49 89 f4 53 49 89 d7 48 89 fb 48 83 ec 08 e8 d1 50 67 ff
> 49 8d 7c 24 10 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <80>
> 3c 02 00 0f 85 35 02 00 00 49 83 7c 24 10 00 0f 84 bb 01 00

This disassembles to:

   0:   41 54                   push   %r12
   2:   49 89 f4                mov    %rsi,%r12
   5:   53                      push   %rbx
   6:   49 89 d7                mov    %rdx,%r15
   9:   48 89 fb                mov    %rdi,%rbx
   c:   48 83 ec 08             sub    $0x8,%rsp
  10:   e8 d1 50 67 ff          callq  0xffffffffff6750e6
  15:   49 8d 7c 24 10          lea    0x10(%r12),%rdi
  1a:   48 b8 00 00 00 00 00    movabs $0xdffffc0000000000,%rax
  21:   fc ff df 
  24:   48 89 fa                mov    %rdi,%rdx
  27:   48 c1 ea 03             shr    $0x3,%rdx
  2b:*  80 3c 02 00             cmpb   $0x0,(%rdx,%rax,1)               <-- trapping instruction
  2f:   0f 85 35 02 00 00       jne    0x26a
  35:   49 83 7c 24 10 00       cmpq   $0x0,0x10(%r12)
  3b:   0f                      .byte 0xf
  3c:   84                      .byte 0x84
  3d:   bb                      .byte 0xbb
  3e:   01 00                   add    %eax,(%rax)

I can see that RAX got loaded from the instruction at 0x1a, but the code
doesn't look very much like what I get out of the compiler (your compiled
function is also at least double the size of what I get, presumably due to
kasan?).

Can you disassemble __key_link_begin() for me and send me your config?

In particular, 0xdffffc0000000000 looks very weird.  Is this code validating
the pointer in R12?

David

[toc] | [prev] | [next] | [standalone]

#1571562

From	Dmitry Vyukov <dvyukov@google.com>
Date	2017-02-01 14:30 +0100
Message-ID	<t6391-6uR-27@gated-at.bofh.it>
In reply to	#1571548

On Wed, Feb 1, 2017 at 2:11 PM, David Howells <dhowells@redhat.com> wrote:
> Dmitry Vyukov <dvyukov@google.com> wrote:
>
>> Code: 41 54 49 89 f4 53 49 89 d7 48 89 fb 48 83 ec 08 e8 d1 50 67 ff
>> 49 8d 7c 24 10 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <80>
>> 3c 02 00 0f 85 35 02 00 00 49 83 7c 24 10 00 0f 84 bb 01 00
>
> This disassembles to:
>
>    0:   41 54                   push   %r12
>    2:   49 89 f4                mov    %rsi,%r12
>    5:   53                      push   %rbx
>    6:   49 89 d7                mov    %rdx,%r15
>    9:   48 89 fb                mov    %rdi,%rbx
>    c:   48 83 ec 08             sub    $0x8,%rsp
>   10:   e8 d1 50 67 ff          callq  0xffffffffff6750e6
>   15:   49 8d 7c 24 10          lea    0x10(%r12),%rdi
>   1a:   48 b8 00 00 00 00 00    movabs $0xdffffc0000000000,%rax
>   21:   fc ff df
>   24:   48 89 fa                mov    %rdi,%rdx
>   27:   48 c1 ea 03             shr    $0x3,%rdx
>   2b:*  80 3c 02 00             cmpb   $0x0,(%rdx,%rax,1)               <-- trapping instruction
>   2f:   0f 85 35 02 00 00       jne    0x26a
>   35:   49 83 7c 24 10 00       cmpq   $0x0,0x10(%r12)
>   3b:   0f                      .byte 0xf
>   3c:   84                      .byte 0x84
>   3d:   bb                      .byte 0xbb
>   3e:   01 00                   add    %eax,(%rax)
>
> I can see that RAX got loaded from the instruction at 0x1a, but the code
> doesn't look very much like what I get out of the compiler (your compiled
> function is also at least double the size of what I get, presumably due to
> kasan?).
>
> Can you disassemble __key_link_begin() for me and send me your config?
>
> In particular, 0xdffffc0000000000 looks very weird.  Is this code validating
> the pointer in R12?


Here is the disasm:
https://gist.githubusercontent.com/dvyukov/c06f742adba0ee162dfb1ece75b5d8ab/raw/14aba22492090e09ea36bddd5db344fdcfc25065/gistfile1.txt

Yes, this is with KASAN+KCOV so size increase is expected.

Before each memory access to *p KASAN inserts:

if (*(char*)(0xdffffc0000000000 + p/8)) __kasan_report();
... *p .... // original memory accesses

You can see it here:

ffffffff8203ea84: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff8203ea8b: fc ff df
ffffffff8203ea8e: 48 89 fa             mov    %rdi,%rdx
ffffffff8203ea91: 48 c1 ea 03           shr    $0x3,%rdx
ffffffff8203ea95: 80 3c 02 00           cmpb   $0x0,(%rdx,%rax,1) //
<<< ACCESS that crashes

For valid kernel addresses this never crashes.
In this case the original addresses that code tried to accesses is in
%RDI. And they look bogus.

[toc] | [prev] | [next] | [standalone]

#1571559

From	David Howells <dhowells@redhat.com>
Date	2017-02-01 14:30 +0100
Message-ID	<t6390-6uR-19@gated-at.bofh.it>
In reply to	#1571518

Dmitry Vyukov <dvyukov@google.com> wrote:

> The line causes the crash is:
>     BUG_ON(index_key->desc_len == 0);
> 
> The addresses that the line tried to access are:
>
> RDI: ffffffffca29fa68
> RDI: ffffffffd7236a28
> RDI: 000000007d19ed68

These are all calculated from R12:

	lea    0x10(%r12),%rdi
	...
	mov    %rdi,%rdx
	shr    $0x3,%rdx
	cmpb   $0x0,(%rdx,%rax,1)

R12 would appear to be index_key - which would correlate with 0x10 being the
offset of desc_len therein - but why muck around with the pointer rather than
just 

In my compiler output,

	BUG_ON(index_key->desc_len == 0);

is turned into:

   0xffffffff8131f023 <+0>:     cmpq   $0x0,0x10(%rsi)
   0xffffffff8131f028 <+5>:     jne    0xffffffff8131f02c <__key_link_begin+9>
   0xffffffff8131f02a <+7>:     ud2    

What compiler are you using?

David

[toc] | [prev] | [next] | [standalone]

#1571572

From	David Howells <dhowells@redhat.com>
Date	2017-02-01 14:50 +0100
Message-ID	<t63sm-6Bo-17@gated-at.bofh.it>
In reply to	#1571518

Dmitry Vyukov <dvyukov@google.com> wrote:

>  request_key_and_link+0x2d8/0x1150 security/keys/request_key.c:549

Can you disassemble this function for me?  There are several possible paths
and without the argument to the syscall and whether there's a key that was
matched, it's hard to say which path is being taken - but this might help
determine that.

David

[toc] | [prev] | [next] | [standalone]

#1571581

From	Dmitry Vyukov <dvyukov@google.com>
Date	2017-02-01 15:10 +0100
Message-ID	<t63LI-6WZ-15@gated-at.bofh.it>
In reply to	#1571572

On Wed, Feb 1, 2017 at 2:48 PM, David Howells <dhowells@redhat.com> wrote:
> Dmitry Vyukov <dvyukov@google.com> wrote:
>
>>  request_key_and_link+0x2d8/0x1150 security/keys/request_key.c:549
>
> Can you disassemble this function for me?  There are several possible paths
> and without the argument to the syscall and whether there's a key that was
> matched, it's hard to say which path is being taken - but this might help
> determine that.

Here it is:
https://gist.githubusercontent.com/dvyukov/65efc41d00ef0033f9374853b9265c71/raw/9d8540dfb199b81f3d3534ec4cc6da378d07f5b2/gistfile1.txt

I actually know what were the arguments to the syscall. Since it
happened in a user process context, I know what syzkaller program it
was running at the time of the crash. It's just they are not
reproducible. Here are the 3 programs, and they are almost equivalent
as far as I can see. It's in syzkaller format, but I hope you can
decipher it, it's just syscall names, address and data in hex:
https://gist.githubusercontent.com/dvyukov/19bd59ffa286a74b49ca2371b69d4c5c/raw/004eaaa58a4ca775c008591fbb94eae78f92ef86/gistfile1.txt

[toc] | [prev] | [next] | [standalone]

#1571625

From	David Howells <dhowells@redhat.com>
Date	2017-02-01 16:00 +0100
Message-ID	<t64y6-7eg-11@gated-at.bofh.it>
In reply to	#1571581

Dmitry Vyukov <dvyukov@google.com> wrote:

> > Can you disassemble this function for me?  There are several possible paths
> > and without the argument to the syscall and whether there's a key that was
> > matched, it's hard to say which path is being taken - but this might help
> > determine that.
> 
> Here it is:
> https://gist.githubusercontent.com/dvyukov/65efc41d00ef0033f9374853b9265c71/raw/9d8540dfb199b81f3d3534ec4cc6da378d07f5b2/gistfile1.txt

Okay, it's called from here:

ffffffff820490fd:	lea    -0x1b0(%rbp),%rdi
ffffffff82049104:	callq  ffffffff82047800 <construct_get_dest_keyring>
ffffffff82049109:	mov    -0x1b0(%rbp),%rdi
ffffffff82049110:	mov    %r15,%rsi
ffffffff82049113:	callq  ffffffff8203efd0 <key_link>  <---
ffffffff82049118:	mov    -0x1b0(%rbp),%rdi
ffffffff8204911f:	mov    %eax,%r14d
ffffffff82049122:	callq  ffffffff82037ab0 <key_put>

Which should correspond to this:

	key_ref = search_process_keyrings(&ctx);

	if (!IS_ERR(key_ref)) {
		key = key_ref_to_ptr(key_ref);
		if (dest_keyring) {
			construct_get_dest_keyring(&dest_keyring);
			ret = key_link(dest_keyring, key);  <---
			key_put(dest_keyring);
			if (ret < 0) {
				key_put(key);
				key = ERR_PTR(ret);
				goto error_free;
			}
		}

which means that the search was successful, the requested key already exists
and there was a destination keyring nominated by userspace.  The first
conditional clause of construct_get_dest_keyring() must've been true:

	struct key *dest_keyring = *_dest_keyring
	...
	if (dest_keyring) {
		/* the caller supplied one */
		key_get(dest_keyring);
	} else {

because it matches the containing if-condition in the calling function.

> I actually know what were the arguments to the syscall. Since it
> happened in a user process context, I know what syzkaller program it
> was running at the time of the crash. It's just they are not
> reproducible. Here are the 3 programs, and they are almost equivalent
> as far as I can see. It's in syzkaller format, but I hope you can
> decipher it, it's just syscall names, address and data in hex:
> https://gist.githubusercontent.com/dvyukov/19bd59ffa286a74b49ca2371b69d4c5c/raw/004eaaa58a4ca775c008591fbb94eae78f92ef86/gistfile1.txt

add_key(&(0x7f0000d02000)="6465616400", ...

What does the "6465616400" represent?  A string containing only numeric
characters or are these 2-digit hex codes and the string is actually "dead"?

David

[toc] | [prev] | [next] | [standalone]

#1571666

From	Dmitry Vyukov <dvyukov@google.com>
Date	2017-02-01 16:30 +0100
Message-ID	<t6518-7DU-9@gated-at.bofh.it>
In reply to	#1571625

On Wed, Feb 1, 2017 at 3:54 PM, David Howells <dhowells@redhat.com> wrote:
> Dmitry Vyukov <dvyukov@google.com> wrote:
>
>> > Can you disassemble this function for me?  There are several possible paths
>> > and without the argument to the syscall and whether there's a key that was
>> > matched, it's hard to say which path is being taken - but this might help
>> > determine that.
>>
>> Here it is:
>> https://gist.githubusercontent.com/dvyukov/65efc41d00ef0033f9374853b9265c71/raw/9d8540dfb199b81f3d3534ec4cc6da378d07f5b2/gistfile1.txt
>
> Okay, it's called from here:
>
> ffffffff820490fd:       lea    -0x1b0(%rbp),%rdi
> ffffffff82049104:       callq  ffffffff82047800 <construct_get_dest_keyring>
> ffffffff82049109:       mov    -0x1b0(%rbp),%rdi
> ffffffff82049110:       mov    %r15,%rsi
> ffffffff82049113:       callq  ffffffff8203efd0 <key_link>  <---
> ffffffff82049118:       mov    -0x1b0(%rbp),%rdi
> ffffffff8204911f:       mov    %eax,%r14d
> ffffffff82049122:       callq  ffffffff82037ab0 <key_put>
>
> Which should correspond to this:
>
>         key_ref = search_process_keyrings(&ctx);
>
>         if (!IS_ERR(key_ref)) {
>                 key = key_ref_to_ptr(key_ref);
>                 if (dest_keyring) {
>                         construct_get_dest_keyring(&dest_keyring);
>                         ret = key_link(dest_keyring, key);  <---
>                         key_put(dest_keyring);
>                         if (ret < 0) {
>                                 key_put(key);
>                                 key = ERR_PTR(ret);
>                                 goto error_free;
>                         }
>                 }
>
> which means that the search was successful, the requested key already exists
> and there was a destination keyring nominated by userspace.  The first
> conditional clause of construct_get_dest_keyring() must've been true:
>
>         struct key *dest_keyring = *_dest_keyring
>         ...
>         if (dest_keyring) {
>                 /* the caller supplied one */
>                 key_get(dest_keyring);
>         } else {
>
> because it matches the containing if-condition in the calling function.
>
>> I actually know what were the arguments to the syscall. Since it
>> happened in a user process context, I know what syzkaller program it
>> was running at the time of the crash. It's just they are not
>> reproducible. Here are the 3 programs, and they are almost equivalent
>> as far as I can see. It's in syzkaller format, but I hope you can
>> decipher it, it's just syscall names, address and data in hex:
>> https://gist.githubusercontent.com/dvyukov/19bd59ffa286a74b49ca2371b69d4c5c/raw/004eaaa58a4ca775c008591fbb94eae78f92ef86/gistfile1.txt
>
> add_key(&(0x7f0000d02000)="6465616400", ...
>
> What does the "6465616400" represent?  A string containing only numeric
> characters or are these 2-digit hex codes and the string is actually "dead"?

Yes, it is hex. The actual data passed to the syscall is "dead".

[toc] | [prev] | [next] | [standalone]

#1571671

From	David Howells <dhowells@redhat.com>
Date	2017-02-01 16:50 +0100
Message-ID	<t65kt-7KL-3@gated-at.bofh.it>
In reply to	#1571666

Dmitry Vyukov <dvyukov@google.com> wrote:

> >> I actually know what were the arguments to the syscall.

The program resolves to the attached C file.  I can't see how you can tell
which of the six request_key() calls triggered.

Also, what OS are running this on?  If you have PAM set up properly, none of
the request_key() calls should find a match since the key being added is
stored in the user_session keyring, but there should have been a session
keyring set up by PAM which would override that.

David
---
#include <sys/types.h>
#include <keyutils.h>

static const char type_1[] = "dead";
static const char type_2[] = "user";
static const char type_3[] = "\272eld";
static const char type_4[] = ".request_key_auth";
static const char type_5[] = "syzkaller";
static const char desc_1[] = "syz";
static const char desc_2[] = "syz\1";
static const char callout_1[] = "/dev/vcs#";
static const char callout_2[] = ")}]";

static const char payload_1[] = {
	0x44, 0xeb, 0x63, 0xab, 0xd2, 0xdb, 0x6f, 0xd3, 0x4e, 0x9d, 0xfa, 0x7d, 0x35, 0x53, 0x90,
	0x35, 0xf8, 0x8d, 0x7c, 0xbe, 0x94, 0x91, 0xcf, 0xb5, 0xb5, 0xe8, 0x93, 0x47, 0x32, 0x3e,
	0x1f, 0x0f, 0x50, 0xd8, 0x07, 0x9f, 0x65, 0xf8, 0x9f, 0x4e, 0x2c, 0xa8, 0x58, 0xdb, 0xe4,
	0x6c, 0x35, 0x7f, 0x96, 0x34, 0x89, 0x93, 0xc0, 0xb3, 0xea, 0xa0, 0x14, 0x18, 0x8f, 0xee,
	0xb6, 0xae, 0x8b, 0x40, 0xd0, 0x36, 0x95, 0x34, 0x10, 0x17, 0xeb, 0xbe, 0x4d, 0xd6, 0x44,
	0xf4, 0xcd, 0x78, 0x01, 0x63, 0x8a, 0x9e, 0xa9, 0xbf, 0x06, 0x0c, 0x32, 0xc7, 0x4a, 0x63,
	0x34, 0x30, 0x5f, 0x57, 0x7a, 0xe7, 0xde, 0x87, 0x7e, 0x83, 0x4d, 0x70, 0x99, 0x23, 0x5d,
	0x59, 0xd0, 0xa7, 0xd2, 0x3d, 0x02, 0x0f, 0x76, 0xe7, 0x1d, 0x7f, 0xe2, 0x3a, 0x54, 0xce,
	0x0d, 0x81, 0x3f, 0xdd, 0xba, 0x81, 0xe0, 0x73, 0xd5, 0xf1, 0x80, 0x21, 0x6a, 0x34, 0xb0,
	0x31, 0x42, 0xdb, 0x05, 0x59, 0xd3, 0xea, 0x2b, 0x81, 0x1e, 0xc2, 0xb8, 0x3f, 0x95, 0xac,
	0x02, 0x14, 0xb0, 0xf7, 0xe9, 0x8c, 0x74, 0xc0, 0x83, 0xf1, 0xe3, 0x65, 0x5b, 0x59, 0x40,
	0x1a, 0xe4, 0xf7, 0xe1, 0x7f, 0x84, 0x48, 0x9e, 0xb0, 0x93, 0x12, 0x58, 0xa8, 0xfa, 0xef,
	0xcc, 0x8e, 0xe0, 0x0a, 0x39, 0xef, 0x8e, 0xa6, 0xdd, 0x52, 0x3d, 0xa4, 0x39, 0x37
};
static const char payload_2[] = {
	0x2f, 0xaa, 0x95, 0x92, 0x7f, 0x7f, 0xb2, 0x2c, 0x7d, 0x2d, 0xfe, 0x14, 0x02, 0xe2, 0xeb,
	0x70, 0x66, 0x5d, 0xea, 0x11, 0x2f, 0xc6, 0x83, 0xb0, 0x72, 0x9a, 0xfe, 0xc7, 0xf4, 0x50,
	0x67, 0x8e, 0xad, 0xdc, 0xd6, 0xf7, 0x3c, 0x66, 0x11, 0x1d, 0x85, 0xf3, 0x48, 0x1d, 0xaf,
	0x7b, 0xaa, 0x82, 0xd7, 0x00, 0x5c, 0x4f, 0x80, 0x4d, 0xc9, 0x51, 0xe2, 0x41, 0xd6, 0x43,
	0x2d, 0x3f, 0x6f, 0x3c, 0x17, 0xe0, 0xea, 0xdb, 0xa2, 0x95, 0x0d, 0x2b, 0xe3, 0xea, 0x58,
	0xcb, 0x81, 0x98, 0xb4, 0xe7, 0xf4, 0xda, 0xd5, 0x74, 0x96, 0x56, 0x6c, 0x85, 0x6c, 0xb9,
	0xd1, 0xdc, 0x6a, 0x18, 0xbe, 0xe6, 0x5d, 0xd2, 0xb3, 0xce, 0x08, 0xcb, 0xe6, 0xad, 0xb9,
	0x36, 0x39, 0x9e, 0x3a, 0x1e, 0xb2, 0xf4, 0x7f, 0xa1, 0xb7, 0x98, 0x05, 0xaf, 0x18, 0x93,
	0xa2, 0xda, 0xa1, 0xfe, 0x71, 0x9b, 0xe8, 0x7d, 0x0a, 0x4a, 0xb3, 0x91, 0x09, 0xe4, 0xa8,
	0xee, 0x41, 0x7b, 0x3b, 0xee, 0x25, 0x53, 0xb9, 0x81, 0xef, 0xb6, 0x53, 0x09, 0x14, 0x9e,
	0x68, 0xf0, 0x75, 0x03, 0xa9, 0x2e, 0xf5, 0xc4, 0x7a, 0x44, 0xb2, 0x54, 0x86, 0xbc, 0x08,
	0xfa, 0x16, 0xbc, 0xd3, 0x5f, 0xe9, 0xdd, 0xd7, 0x34, 0x98, 0x8e, 0xe0, 0xa3, 0xd0, 0x2b,
	0x8b, 0x4a, 0xec, 0xf4, 0x23, 0x3c
};

int main()
{
	key_serial_t r24, r25, r26;
	
	add_key(type_1, desc_1, payload_1, sizeof(payload_1), 0xffffffffffffffff);
	r25 = request_key(type_2, desc_2, callout_1, 0xfffffffffffffffb);
	add_key(type_5, desc_2, 0x0, 0x0, 0xfffffffffffffffb);
	request_key(type_4, desc_1, callout_2, 0xfffffffffffffffb);
	r26 = add_key(type_2, desc_2, payload_2, sizeof(payload_2), 0xfffffffffffffffb);
	keyctl_unlink(r25, r26);


	add_key(type_3, desc_1, payload_1, sizeof(payload_1) - 1, 0xffffffffffffffff);
	r24 = request_key(type_2, desc_2, callout_1, 0xfffffffffffffffb);
	add_key(type_5, desc_2, 0x0, 0x0, 0xfffffffffffffffb);
	request_key(type_4, desc_1, callout_2, 0xfffffffffffffffb);
	r25 = add_key(type_2, desc_2, payload_2, sizeof(payload_2), 0xfffffffffffffffb);
	keyctl_unlink(r24, r25);


	add_key(type_1, desc_1, payload_1, sizeof(payload_1) - 1, 0xffffffffffffffff);
	r25 = request_key(type_2, desc_2, callout_1, 0xfffffffffffffffb);
	add_key(type_5, desc_2, 0x0, 0x0, 0xfffffffffffffffb);
	request_key(type_4, desc_1, callout_2, 0xfffffffffffffffb);
	r26 = add_key(type_2, desc_2, payload_2, sizeof(payload_2), 0xfffffffffffffffb);
	keyctl_unlink(r25, r26);
	return 0;
}

[toc] | [prev] | [next] | [standalone]

#1571681

From	Dmitry Vyukov <dvyukov@google.com>
Date	2017-02-01 17:00 +0100
Message-ID	<t65ua-7Of-17@gated-at.bofh.it>
In reply to	#1571671

On Wed, Feb 1, 2017 at 4:42 PM, David Howells <dhowells@redhat.com> wrote:
> Dmitry Vyukov <dvyukov@google.com> wrote:
>
>> >> I actually know what were the arguments to the syscall.
>
> The program resolves to the attached C file.  I can't see how you can tell
> which of the six request_key() calls triggered.

These are 3 separate programs that provoked the 3 different crashes
that I posted in the first email. So we need to guess one of the _two_
request_key() calls in each program. But I don't know which of the two
calls triggered the crash.


> Also, what OS are running this on?  If you have PAM set up properly, none of
> the request_key() calls should find a match since the key being added is
> stored in the user_session keyring, but there should have been a session
> keyring set up by PAM which would override that.

The OS is debian/wheezy created with:
$ debootstrap --include=openssh-server,curl,tar,time,strace,sudo,less,psmisc
wheezy wheezy

I did not do any additional setup. I don't know what is PAM, so I
guess I did not set it up.
The machine is GCE VM.




> David
> ---
> #include <sys/types.h>
> #include <keyutils.h>
>
> static const char type_1[] = "dead";
> static const char type_2[] = "user";
> static const char type_3[] = "\272eld";
> static const char type_4[] = ".request_key_auth";
> static const char type_5[] = "syzkaller";
> static const char desc_1[] = "syz";
> static const char desc_2[] = "syz\1";
> static const char callout_1[] = "/dev/vcs#";
> static const char callout_2[] = ")}]";
>
> static const char payload_1[] = {
>         0x44, 0xeb, 0x63, 0xab, 0xd2, 0xdb, 0x6f, 0xd3, 0x4e, 0x9d, 0xfa, 0x7d, 0x35, 0x53, 0x90,
>         0x35, 0xf8, 0x8d, 0x7c, 0xbe, 0x94, 0x91, 0xcf, 0xb5, 0xb5, 0xe8, 0x93, 0x47, 0x32, 0x3e,
>         0x1f, 0x0f, 0x50, 0xd8, 0x07, 0x9f, 0x65, 0xf8, 0x9f, 0x4e, 0x2c, 0xa8, 0x58, 0xdb, 0xe4,
>         0x6c, 0x35, 0x7f, 0x96, 0x34, 0x89, 0x93, 0xc0, 0xb3, 0xea, 0xa0, 0x14, 0x18, 0x8f, 0xee,
>         0xb6, 0xae, 0x8b, 0x40, 0xd0, 0x36, 0x95, 0x34, 0x10, 0x17, 0xeb, 0xbe, 0x4d, 0xd6, 0x44,
>         0xf4, 0xcd, 0x78, 0x01, 0x63, 0x8a, 0x9e, 0xa9, 0xbf, 0x06, 0x0c, 0x32, 0xc7, 0x4a, 0x63,
>         0x34, 0x30, 0x5f, 0x57, 0x7a, 0xe7, 0xde, 0x87, 0x7e, 0x83, 0x4d, 0x70, 0x99, 0x23, 0x5d,
>         0x59, 0xd0, 0xa7, 0xd2, 0x3d, 0x02, 0x0f, 0x76, 0xe7, 0x1d, 0x7f, 0xe2, 0x3a, 0x54, 0xce,
>         0x0d, 0x81, 0x3f, 0xdd, 0xba, 0x81, 0xe0, 0x73, 0xd5, 0xf1, 0x80, 0x21, 0x6a, 0x34, 0xb0,
>         0x31, 0x42, 0xdb, 0x05, 0x59, 0xd3, 0xea, 0x2b, 0x81, 0x1e, 0xc2, 0xb8, 0x3f, 0x95, 0xac,
>         0x02, 0x14, 0xb0, 0xf7, 0xe9, 0x8c, 0x74, 0xc0, 0x83, 0xf1, 0xe3, 0x65, 0x5b, 0x59, 0x40,
>         0x1a, 0xe4, 0xf7, 0xe1, 0x7f, 0x84, 0x48, 0x9e, 0xb0, 0x93, 0x12, 0x58, 0xa8, 0xfa, 0xef,
>         0xcc, 0x8e, 0xe0, 0x0a, 0x39, 0xef, 0x8e, 0xa6, 0xdd, 0x52, 0x3d, 0xa4, 0x39, 0x37
> };
> static const char payload_2[] = {
>         0x2f, 0xaa, 0x95, 0x92, 0x7f, 0x7f, 0xb2, 0x2c, 0x7d, 0x2d, 0xfe, 0x14, 0x02, 0xe2, 0xeb,
>         0x70, 0x66, 0x5d, 0xea, 0x11, 0x2f, 0xc6, 0x83, 0xb0, 0x72, 0x9a, 0xfe, 0xc7, 0xf4, 0x50,
>         0x67, 0x8e, 0xad, 0xdc, 0xd6, 0xf7, 0x3c, 0x66, 0x11, 0x1d, 0x85, 0xf3, 0x48, 0x1d, 0xaf,
>         0x7b, 0xaa, 0x82, 0xd7, 0x00, 0x5c, 0x4f, 0x80, 0x4d, 0xc9, 0x51, 0xe2, 0x41, 0xd6, 0x43,
>         0x2d, 0x3f, 0x6f, 0x3c, 0x17, 0xe0, 0xea, 0xdb, 0xa2, 0x95, 0x0d, 0x2b, 0xe3, 0xea, 0x58,
>         0xcb, 0x81, 0x98, 0xb4, 0xe7, 0xf4, 0xda, 0xd5, 0x74, 0x96, 0x56, 0x6c, 0x85, 0x6c, 0xb9,
>         0xd1, 0xdc, 0x6a, 0x18, 0xbe, 0xe6, 0x5d, 0xd2, 0xb3, 0xce, 0x08, 0xcb, 0xe6, 0xad, 0xb9,
>         0x36, 0x39, 0x9e, 0x3a, 0x1e, 0xb2, 0xf4, 0x7f, 0xa1, 0xb7, 0x98, 0x05, 0xaf, 0x18, 0x93,
>         0xa2, 0xda, 0xa1, 0xfe, 0x71, 0x9b, 0xe8, 0x7d, 0x0a, 0x4a, 0xb3, 0x91, 0x09, 0xe4, 0xa8,
>         0xee, 0x41, 0x7b, 0x3b, 0xee, 0x25, 0x53, 0xb9, 0x81, 0xef, 0xb6, 0x53, 0x09, 0x14, 0x9e,
>         0x68, 0xf0, 0x75, 0x03, 0xa9, 0x2e, 0xf5, 0xc4, 0x7a, 0x44, 0xb2, 0x54, 0x86, 0xbc, 0x08,
>         0xfa, 0x16, 0xbc, 0xd3, 0x5f, 0xe9, 0xdd, 0xd7, 0x34, 0x98, 0x8e, 0xe0, 0xa3, 0xd0, 0x2b,
>         0x8b, 0x4a, 0xec, 0xf4, 0x23, 0x3c
> };
>
> int main()
> {
>         key_serial_t r24, r25, r26;
>
>         add_key(type_1, desc_1, payload_1, sizeof(payload_1), 0xffffffffffffffff);
>         r25 = request_key(type_2, desc_2, callout_1, 0xfffffffffffffffb);
>         add_key(type_5, desc_2, 0x0, 0x0, 0xfffffffffffffffb);
>         request_key(type_4, desc_1, callout_2, 0xfffffffffffffffb);
>         r26 = add_key(type_2, desc_2, payload_2, sizeof(payload_2), 0xfffffffffffffffb);
>         keyctl_unlink(r25, r26);
>
>
>         add_key(type_3, desc_1, payload_1, sizeof(payload_1) - 1, 0xffffffffffffffff);
>         r24 = request_key(type_2, desc_2, callout_1, 0xfffffffffffffffb);
>         add_key(type_5, desc_2, 0x0, 0x0, 0xfffffffffffffffb);
>         request_key(type_4, desc_1, callout_2, 0xfffffffffffffffb);
>         r25 = add_key(type_2, desc_2, payload_2, sizeof(payload_2), 0xfffffffffffffffb);
>         keyctl_unlink(r24, r25);
>
>
>         add_key(type_1, desc_1, payload_1, sizeof(payload_1) - 1, 0xffffffffffffffff);
>         r25 = request_key(type_2, desc_2, callout_1, 0xfffffffffffffffb);
>         add_key(type_5, desc_2, 0x0, 0x0, 0xfffffffffffffffb);
>         request_key(type_4, desc_1, callout_2, 0xfffffffffffffffb);
>         r26 = add_key(type_2, desc_2, payload_2, sizeof(payload_2), 0xfffffffffffffffb);
>         keyctl_unlink(r25, r26);
>         return 0;
> }
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[toc] | [prev] | [next] | [standalone]

#1571745

From	David Howells <dhowells@redhat.com>
Date	2017-02-01 18:00 +0100
Message-ID	<t66qf-8ov-37@gated-at.bofh.it>
In reply to	#1571681

Do you reboot the system between running individual programs?  If not, the
programs will be influencing each other.  Further, only those calls with valid
type and matching description values are relevant, I think.  This means those
that use:

	static const char type_2[] = "user";
	static const char desc_2[] = "syz\1";

so:

	r25 = request_key(type_2, desc_2, ...);
	...
	r26 = add_key(type_2, desc_2, ...);
	...
	r24 = request_key(type_2, desc_2, ...);
	...
	r25 = add_key(type_2, desc_2, ...);
	...
	r25 = request_key(type_2, desc_2, ...);
	...
	r26 = add_key(type_2, desc_2, ...);

The first request_key() call will fail because it doesn't find anything and
the upcall process, if it is available, has no suitable handler and will
negatively instantiate it.

The first add_key() call will then update the key to make it positively
instantiated, after which subsequent request_key() calls will return the key
and add_key() calls will update its contents.

So, it would appear that it's not the first call to request_key() of type_2,
desc_2, but one subsequent to that.  The type_4 request_key() calls should get
weeded out very quickly in sys_request_key() by key_get_type_from_user() -
which seems to happen (EPERM is returned).

Doing:

	keyctl link @us @s

before running the program on Fedora allows the request_key() to find the
add_key() results.

Do you run some of these in parallel?  Running the combo program 100,000 times
sequentially didn't produce a crash.

> The OS is debian/wheezy created with:
> $ debootstrap --include=openssh-server,curl,tar,time,strace,sudo,less,psmisc
> wheezy wheezy
> 
> I did not do any additional setup. I don't know what is PAM, so I
> guess I did not set it up.
> The machine is GCE VM.

I would imagine that PAM is part of the core OS - it does things like
controlling login service security.  Jessie apparently has it.  However,
Debian didn't use to include pam_keyinit.

David

[toc] | [prev] | [next] | [standalone]

#1571786

From	Dmitry Vyukov <dvyukov@google.com>
Date	2017-02-01 18:20 +0100
Message-ID	<t66Jz-iP-13@gated-at.bofh.it>
In reply to	#1571745

On Wed, Feb 1, 2017 at 5:50 PM, David Howells <dhowells@redhat.com> wrote:
> Do you reboot the system between running individual programs?  If not, the
> programs will be influencing each other.  Further, only those calls with valid
> type and matching description values are relevant, I think.  This means those
> that use:
>
>         static const char type_2[] = "user";
>         static const char desc_2[] = "syz\1";
>
> so:
>
>         r25 = request_key(type_2, desc_2, ...);
>         ...
>         r26 = add_key(type_2, desc_2, ...);
>         ...
>         r24 = request_key(type_2, desc_2, ...);
>         ...
>         r25 = add_key(type_2, desc_2, ...);
>         ...
>         r25 = request_key(type_2, desc_2, ...);
>         ...
>         r26 = add_key(type_2, desc_2, ...);
>
> The first request_key() call will fail because it doesn't find anything and
> the upcall process, if it is available, has no suitable handler and will
> negatively instantiate it.
>
> The first add_key() call will then update the key to make it positively
> instantiated, after which subsequent request_key() calls will return the key
> and add_key() calls will update its contents.
>
> So, it would appear that it's not the first call to request_key() of type_2,
> desc_2, but one subsequent to that.  The type_4 request_key() calls should get
> weeded out very quickly in sys_request_key() by key_get_type_from_user() -
> which seems to happen (EPERM is returned).
>
> Doing:
>
>         keyctl link @us @s
>
> before running the program on Fedora allows the request_key() to find the
> add_key() results.
>
> Do you run some of these in parallel?  Running the combo program 100,000 times
> sequentially didn't produce a crash.
>
>> The OS is debian/wheezy created with:
>> $ debootstrap --include=openssh-server,curl,tar,time,strace,sudo,less,psmisc
>> wheezy wheezy
>>
>> I did not do any additional setup. I don't know what is PAM, so I
>> guess I did not set it up.
>> The machine is GCE VM.
>
> I would imagine that PAM is part of the core OS - it does things like
> controlling login service security.  Jessie apparently has it.  However,
> Debian didn't use to include pam_keyinit.



No, syzkaller does not reboot machines after execution of a single
program. That would make the testing process too slow.
It relies on process/filesystem/namespace isolation. But keys
subsystem has global state and that's the problem.
So, yes, most likely there was some accumulated state from previous programs.

[toc] | [prev] | [standalone]

csiph-web

keys: GPF in request_key

Contents

#1571518 — keys: GPF in request_key

#1571548

#1571562

#1571559

#1571572

#1571581

#1571625

#1571666

#1571671

#1571681

#1571745

#1571786