Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1739989 > unrolled thread

percpu allocation failures

Back to article view | Back to linux.kernel

Contents

  percpu allocation failures Luis Henriques <lhenriques@suse.com> - 2017-09-26 17:40 +0200
    Re: percpu allocation failures Dennis Zhou <dennisszhou@gmail.com> - 2017-09-27 04:00 +0200
      Re: percpu allocation failures Luis Henriques <lhenriques@suse.com> - 2017-09-27 11:10 +0200

#1739989 — percpu allocation failures

Hi,

Probably already reported, but I couldn't find anything so here it
goes:

starting with 4.14-rc1 I see the following during boot:

[   25.199053] percpu: allocation failed, size=16 align=16 atomic=0, alloc from reserved chunk failed
[   25.200195] CPU: 5 PID: 723 Comm: modprobe Tainted: G            E   4.14.0-rc2 #103
[   25.201290] Hardware name: Dell Inc. Precision 5510/0N8J4R, BIOS 1.2.25 05/07/2017
[   25.202430] Call Trace:
[   25.203509]  dump_stack+0x63/0x89
[   25.204364]  pcpu_alloc+0x5cd/0x5f0
[   25.205302]  __alloc_reserved_percpu+0x18/0x20
[   25.206355]  load_module+0x733/0x2c00
[   25.207444]  ? kernel_read_file+0x1a3/0x1d0
[   25.208596]  SYSC_finit_module+0xfc/0x120
[   25.209634]  ? SYSC_finit_module+0xfc/0x120
[   25.210733]  SyS_finit_module+0xe/0x10
[   25.211747]  entry_SYSCALL_64_fastpath+0x1e/0xa9
[   25.212763] RIP: 0033:0x7f70d9e86219
[   25.213508] RSP: 002b:00007ffcd3ff8f38 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[   25.214391] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f70d9e86219
[   25.215107] RDX: 0000000000000000 RSI: 00005642ddf158cc RDI: 0000000000000000
[   25.215949] RBP: 00007ffcd3ff7f30 R08: 0000000000000000 R09: 0000000000000001
[   25.216592] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000005
[   25.217194] R13: 00005642de3d65f0 R14: 00007ffcd3ff7f10 R15: 0000000000000005
[   25.217812] nft_meta: Could not allocate 16 bytes percpu data

A few more failures follow.

A bisect ended up with the merge commit a7cbfd05f427 ("Merge branch
'for-4.14' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu").

Cheers,
-- 
Luis

[toc] | [next] | [standalone]

#1740340

Hi Luis,

This seems to be an issue with the reserved chunk being unable to
allocate memory when loading kernel modules. Unfortunately, I have not 
been successful in reproducing this with the reserved chunk allocation
path exposed or by inserting the nft_meta module.

Could you please send me the output when ran with the following patch
and the output of the percpu memory statistics file before and after
inserting the module (PERCPU_STATS)? The stats are in
/sys/kernel/debug/percpu_stats.

Thanks,
Dennis

---
 mm/percpu.c | 32 ++++++++++++++++++++++++++++++--
 1 file changed, 30 insertions(+), 2 deletions(-)

diff --git a/mm/percpu.c b/mm/percpu.c
index 59d44d6..031fd91 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -1335,6 +1335,7 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved,
 {
 	static int warn_limit = 10;
 	struct pcpu_chunk *chunk;
+	struct pcpu_block_md *block;
 	const char *err;
 	bool is_atomic = (gfp & GFP_KERNEL) != GFP_KERNEL;
 	int slot, off, cpu, ret;
@@ -1371,17 +1372,43 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved,
 	if (reserved && pcpu_reserved_chunk) {
 		chunk = pcpu_reserved_chunk;
 
+		printk(KERN_DEBUG "percpu: reserved chunk: %d, %d, %d, %d, %d, %d, %d",
+		       chunk->free_bytes, chunk->contig_bits,
+		       chunk->contig_bits_start, chunk->first_bit,
+		       chunk->start_offset, chunk->end_offset,
+		       chunk->nr_pages);
+
+		printk(KERN_DEBUG "percpu: rchunk md blocks");
+		for (block = chunk->md_blocks;
+		     block < chunk->md_blocks + pcpu_chunk_nr_blocks(chunk);
+		     block++) {
+			printk(KERN_DEBUG "   percpu: %d, %d, %d, %d, %d",
+			       block->contig_hint,
+			       block->contig_hint_start,
+			       block->left_free,
+			       block->right_free,
+			       block->first_free);
+		}
+
 		off = pcpu_find_block_fit(chunk, bits, bit_align, is_atomic);
+
+		printk(KERN_DEBUG "percpu: pcpu_find_block_fit: %d, %zu, %zu",
+		       off, bits, bit_align);
+
 		if (off < 0) {
-			err = "alloc from reserved chunk failed";
+			err = "alloc from reserved chunk failed to find fit";
 			goto fail_unlock;
 		}
 
 		off = pcpu_alloc_area(chunk, bits, bit_align, off);
+
+		printk(KERN_DEBUG "percpu: pcpu_alloc_area: %d, %zu, %zu",
+		       off, bits, bit_align);
+
 		if (off >= 0)
 			goto area_found;
 
-		err = "alloc from reserved chunk failed";
+		err = "alloc from reserved chunk failed to alloc area";
 		goto fail_unlock;
 	}
 
@@ -1547,6 +1574,7 @@ void __percpu *__alloc_reserved_percpu(size_t size, size_t align)
 {
 	return pcpu_alloc(size, align, true, GFP_KERNEL);
 }
+EXPORT_SYMBOL_GPL(__alloc_reserved_percpu);
 
 /**
  * pcpu_balance_workfn - manage the amount of free chunks and populated pages
-- 
1.8.3.1

[toc] | [prev] | [next] | [standalone]

#1740524

[Multipart message — attachments visible in raw view] — view raw

Dennis Zhou <dennisszhou@gmail.com> writes:

> Hi Luis,
>
> This seems to be an issue with the reserved chunk being unable to
> allocate memory when loading kernel modules. Unfortunately, I have not 
> been successful in reproducing this with the reserved chunk allocation
> path exposed or by inserting the nft_meta module.
>
> Could you please send me the output when ran with the following patch
> and the output of the percpu memory statistics file before and after
> inserting the module (PERCPU_STATS)? The stats are in
> /sys/kernel/debug/percpu_stats.

Please find attached all the info you requested.  Hope it helps.

Also, here's the nft script I'm using to trigger the issue locally:

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
	chain input {
		type filter hook input priority 0;
		iif lo accept
		ct state established,related accept
		tcp dport { 22 } ct state new accept
		ip6 nexthdr icmpv6 icmpv6 type { nd-neighbor-solicit,  nd-router-advert, nd-neighbor-advert } accept
		counter drop
	}
}

Cheers,
-- 
Luis

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web