Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1482786
| From | Andy Lutomirski <luto@kernel.org> |
|---|---|
| Newsgroups | linux.kernel |
| Subject | [PATCH 12/12] fork: Cache two thread stacks per cpu if CONFIG_VMAP_STACK is set |
| Date | 2016-09-13 23:40 +0200 |
| Message-ID | <sh3AS-7tK-13@gated-at.bofh.it> (permalink) |
| References | <sh3rc-7o2-3@gated-at.bofh.it> |
| Organization | linux.* mail to news gateway |
vmalloc is a bit slow, and pounding vmalloc/vfree will eventually
force a global TLB flush.
To reduce pressure on them, if CONFIG_VMAP_STACK, cache two thread
stacks per cpu. This will let us quickly allocate a hopefully
cache-hot, TLB-hot stack under heavy forking workloads (shell script
style).
On my silly pthread_create benchmark, it saves about 2 µs per
pthread_create+join with CONFIG_VMAP_STACK=y.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
---
kernel/fork.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++---------
1 file changed, 53 insertions(+), 9 deletions(-)
diff --git a/kernel/fork.c b/kernel/fork.c
index e0fd5446df58..128ef350ce53 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -159,15 +159,41 @@ void __weak arch_release_thread_stack(unsigned long *stack)
* kmemcache based allocator.
*/
# if THREAD_SIZE >= PAGE_SIZE || defined(CONFIG_VMAP_STACK)
+
+#ifdef CONFIG_VMAP_STACK
+/*
+ * vmalloc is a bit slow, and calling vfree enough times will force a TLB
+ * flush. Try to minimize the number of calls by caching stacks.
+ */
+#define NR_CACHED_STACKS 2
+static DEFINE_PER_CPU(struct vm_struct *, cached_stacks[NR_CACHED_STACKS]);
+#endif
+
static unsigned long *alloc_thread_stack_node(struct task_struct *tsk, int node)
{
#ifdef CONFIG_VMAP_STACK
- void *stack = __vmalloc_node_range(THREAD_SIZE, THREAD_SIZE,
- VMALLOC_START, VMALLOC_END,
- THREADINFO_GFP | __GFP_HIGHMEM,
- PAGE_KERNEL,
- 0, node,
- __builtin_return_address(0));
+ void *stack;
+ int i;
+
+ local_irq_disable();
+ for (i = 0; i < NR_CACHED_STACKS; i++) {
+ struct vm_struct *s = this_cpu_read(cached_stacks[i]);
+
+ if (!s)
+ continue;
+ this_cpu_write(cached_stacks[i], NULL);
+
+ tsk->stack_vm_area = s;
+ local_irq_enable();
+ return s->addr;
+ }
+ local_irq_enable();
+
+ stack = __vmalloc_node_range(THREAD_SIZE, THREAD_SIZE,
+ VMALLOC_START, VMALLOC_END,
+ THREADINFO_GFP | __GFP_HIGHMEM,
+ PAGE_KERNEL,
+ 0, node, __builtin_return_address(0));
/*
* We can't call find_vm_area() in interrupt context, and
@@ -187,10 +213,28 @@ static unsigned long *alloc_thread_stack_node(struct task_struct *tsk, int node)
static inline void free_thread_stack(struct task_struct *tsk)
{
- if (task_stack_vm_area(tsk))
+#ifdef CONFIG_VMAP_STACK
+ if (task_stack_vm_area(tsk)) {
+ unsigned long flags;
+ int i;
+
+ local_irq_save(flags);
+ for (i = 0; i < NR_CACHED_STACKS; i++) {
+ if (this_cpu_read(cached_stacks[i]))
+ continue;
+
+ this_cpu_write(cached_stacks[i], tsk->stack_vm_area);
+ local_irq_restore(flags);
+ return;
+ }
+ local_irq_restore(flags);
+
vfree(tsk->stack);
- else
- __free_pages(virt_to_page(tsk->stack), THREAD_SIZE_ORDER);
+ return;
+ }
+#endif
+
+ __free_pages(virt_to_page(tsk->stack), THREAD_SIZE_ORDER);
}
# else
static struct kmem_cache *thread_stack_cache;
--
2.7.4
Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
[PATCH 00/12] thread_info cleanups and stack caching Andy Lutomirski <luto@kernel.org> - 2016-09-13 23:30 +0200
[PATCH 03/12] um: Stop conflating task_struct::stack with thread_info Andy Lutomirski <luto@kernel.org> - 2016-09-13 23:30 +0200
Re: [PATCH 03/12] um: Stop conflating task_struct::stack with thread_info Ingo Molnar <mingo@kernel.org> - 2016-09-15 08:30 +0200
[tip:x86/asm] um/Stop conflating task_struct::stack with thread_info tip-bot for Linus Torvalds <tipbot@zytor.com> - 2016-09-15 12:50 +0200
[PATCH 12/12] fork: Cache two thread stacks per cpu if CONFIG_VMAP_STACK is set Andy Lutomirski <luto@kernel.org> - 2016-09-13 23:40 +0200
[PATCH 01/12] x86/asm: Move 'status' from struct thread_info to struct thread_struct Andy Lutomirski <luto@kernel.org> - 2016-09-13 23:40 +0200
[tip:x86/asm] x86/asm: Move the thread_info::status field to thread_struct tip-bot for Andy Lutomirski <tipbot@zytor.com> - 2016-09-15 12:50 +0200
[PATCH 10/12] lib/syscall: Pin the task stack in collect_syscall() Andy Lutomirski <luto@kernel.org> - 2016-09-13 23:40 +0200
[PATCH 08/12] x86/dumpstack: Pin the target stack in save_stack_trace_tsk() Andy Lutomirski <luto@kernel.org> - 2016-09-13 23:40 +0200
Re: [PATCH 08/12] x86/dumpstack: Pin the target stack in save_stack_trace_tsk() Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-14 17:00 +0200
Re: [PATCH 08/12] x86/dumpstack: Pin the target stack in save_stack_trace_tsk() Andy Lutomirski <luto@amacapital.net> - 2016-09-14 20:30 +0200
Re: [PATCH 08/12] x86/dumpstack: Pin the target stack in save_stack_trace_tsk() Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-14 20:40 +0200
Re: [PATCH 08/12] x86/dumpstack: Pin the target stack in save_stack_trace_tsk() Andy Lutomirski <luto@amacapital.net> - 2016-09-15 20:10 +0200
Re: [PATCH 08/12] x86/dumpstack: Pin the target stack in save_stack_trace_tsk() Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-15 20:40 +0200
Re: [PATCH 08/12] x86/dumpstack: Pin the target stack in save_stack_trace_tsk() Andy Lutomirski <luto@amacapital.net> - 2016-09-15 20:50 +0200
Re: [PATCH 08/12] x86/dumpstack: Pin the target stack in save_stack_trace_tsk() Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-15 21:20 +0200
Re: [PATCH 08/12] x86/dumpstack: Pin the target stack in save_stack_trace_tsk() Peter Zijlstra <peterz@infradead.org> - 2016-09-16 09:50 +0200
Re: [PATCH 08/12] x86/dumpstack: Pin the target stack in save_stack_trace_tsk() Andy Lutomirski <luto@amacapital.net> - 2016-09-16 17:20 +0200
Re: [PATCH 08/12] x86/dumpstack: Pin the target stack in save_stack_trace_tsk() Andy Lutomirski <luto@kernel.org> - 2016-09-16 17:40 +0200
Re: [PATCH 08/12] x86/dumpstack: Pin the target stack in save_stack_trace_tsk() Peter Zijlstra <peterz@infradead.org> - 2016-09-16 18:40 +0200
Re: [PATCH 08/12] x86/dumpstack: Pin the target stack in save_stack_trace_tsk() Peter Zijlstra <peterz@infradead.org> - 2016-09-16 17:40 +0200
Re: [PATCH 08/12] x86/dumpstack: Pin the target stack in save_stack_trace_tsk() Ingo Molnar <mingo@kernel.org> - 2016-09-15 08:40 +0200
Re: [PATCH 08/12] x86/dumpstack: Pin the target stack in save_stack_trace_tsk() Ingo Molnar <mingo@kernel.org> - 2016-09-15 11:30 +0200
[PATCH 09/12] x86/process: Pin the target stack in get_wchan() Andy Lutomirski <luto@kernel.org> - 2016-09-13 23:40 +0200
Re: [PATCH 09/12] x86/process: Pin the target stack in get_wchan() Jann Horn <jann@thejh.net> - 2016-09-17 04:10 +0200
Re: [PATCH 09/12] x86/process: Pin the target stack in get_wchan() Andy Lutomirski <luto@amacapital.net> - 2016-09-23 00:50 +0200
Re: [PATCH 09/12] x86/process: Pin the target stack in get_wchan() Andy Lutomirski <luto@amacapital.net> - 2016-09-23 01:00 +0200
Re: [PATCH 09/12] x86/process: Pin the target stack in get_wchan() Jann Horn <jann@thejh.net> - 2016-09-23 09:50 +0200
[PATCH 06/12] sched: Add try_get_task_stack() and put_task_stack() Andy Lutomirski <luto@kernel.org> - 2016-09-13 23:40 +0200
[PATCH 02/12] x86/entry: Get rid of pt_regs_to_thread_info() Andy Lutomirski <luto@kernel.org> - 2016-09-13 23:40 +0200
Re: [PATCH 02/12] x86/entry: Get rid of pt_regs_to_thread_info() Ingo Molnar <mingo@kernel.org> - 2016-09-15 08:30 +0200
[tip:x86/asm] x86/entry: Get rid of pt_regs_to_thread_info() tip-bot for Linus Torvalds <tipbot@zytor.com> - 2016-09-15 12:50 +0200
[PATCH 05/12] x86: Move thread_info into task_struct Andy Lutomirski <luto@kernel.org> - 2016-09-13 23:40 +0200
[tip:x86/asm] x86: Move thread_info into task_struct tip-bot for Andy Lutomirski <tipbot@zytor.com> - 2016-09-15 12:50 +0200
[PATCH 04/12] sched: Allow putting thread_info into task_struct Andy Lutomirski <luto@kernel.org> - 2016-09-13 23:40 +0200
[tip:x86/asm] sched/core: Allow putting thread_info into task_struct tip-bot for Andy Lutomirski <tipbot@zytor.com> - 2016-09-15 12:50 +0200
[PATCH 11/12] sched: Free the stack early if CONFIG_THREAD_INFO_IN_TASK Andy Lutomirski <luto@kernel.org> - 2016-09-13 23:40 +0200
[PATCH 07/12] kthread: to_live_kthread() needs try_get_task_stack() Andy Lutomirski <luto@kernel.org> - 2016-09-13 23:40 +0200
csiph-web