Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1684463

[PATCH 4.12 19/27] sched/core: Implement new approach to scale select_idle_cpu()

Show all headers | View raw

4.12-stable review patch.  If anyone has any objections, please let me know.

------------------

From: Peter Zijlstra <peterz@infradead.org>

commit 1ad3aaf3fcd2444406628a19a9b9e0922b95e2d4 upstream.

Hackbench recently suffered a bunch of pain, first by commit:

  4c77b18cf8b7 ("sched/fair: Make select_idle_cpu() more aggressive")

and then by commit:

  c743f0a5c50f ("sched/fair, cpumask: Export for_each_cpu_wrap()")

which fixed a bug in the initial for_each_cpu_wrap() implementation
that made select_idle_cpu() even more expensive. The bug was that it
would skip over CPUs when bits were consequtive in the bitmask.

This however gave me an idea to fix select_idle_cpu(); where the old
scheme was a cliff-edge throttle on idle scanning, this introduces a
more gradual approach. Instead of stopping to scan entirely, we limit
how many CPUs we scan.

Initial benchmarks show that it mostly recovers hackbench while not
hurting anything else, except Mason's schbench, but not as bad as the
old thing.

It also appears to recover the tbench high-end, which also suffered like
hackbench.

Tested-by: Matt Fleming <matt@codeblueprint.co.uk>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Chris Mason <clm@fb.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: hpa@zytor.com
Cc: kitsunyan <kitsunyan@inbox.ru>
Cc: linux-kernel@vger.kernel.org
Cc: lvenanci@redhat.com
Cc: riel@redhat.com
Cc: xiaolong.ye@intel.com
Link: http://lkml.kernel.org/r/20170517105350.hk5m4h4jb6dfr65a@hirez.programming.kicks-ass.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

---
 kernel/sched/fair.c     |   21 ++++++++++++++++-----
 kernel/sched/features.h |    1 +
 2 files changed, 17 insertions(+), 5 deletions(-)

--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -5772,27 +5772,38 @@ static inline int select_idle_smt(struct
 static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int target)
 {
 	struct sched_domain *this_sd;
-	u64 avg_cost, avg_idle = this_rq()->avg_idle;
+	u64 avg_cost, avg_idle;
 	u64 time, cost;
 	s64 delta;
-	int cpu;
+	int cpu, nr = INT_MAX;
 
 	this_sd = rcu_dereference(*this_cpu_ptr(&sd_llc));
 	if (!this_sd)
 		return -1;
 
-	avg_cost = this_sd->avg_scan_cost;
-
 	/*
 	 * Due to large variance we need a large fuzz factor; hackbench in
 	 * particularly is sensitive here.
 	 */
-	if (sched_feat(SIS_AVG_CPU) && (avg_idle / 512) < avg_cost)
+	avg_idle = this_rq()->avg_idle / 512;
+	avg_cost = this_sd->avg_scan_cost + 1;
+
+	if (sched_feat(SIS_AVG_CPU) && avg_idle < avg_cost)
 		return -1;
 
+	if (sched_feat(SIS_PROP)) {
+		u64 span_avg = sd->span_weight * avg_idle;
+		if (span_avg > 4*avg_cost)
+			nr = div_u64(span_avg, avg_cost);
+		else
+			nr = 4;
+	}
+
 	time = local_clock();
 
 	for_each_cpu_wrap(cpu, sched_domain_span(sd), target) {
+		if (!--nr)
+			return -1;
 		if (!cpumask_test_cpu(cpu, &p->cpus_allowed))
 			continue;
 		if (idle_cpu(cpu))
--- a/kernel/sched/features.h
+++ b/kernel/sched/features.h
@@ -55,6 +55,7 @@ SCHED_FEAT(TTWU_QUEUE, true)
  * When doing wakeups, attempt to limit superfluous scans of the LLC domain.
  */
 SCHED_FEAT(SIS_AVG_CPU, false)
+SCHED_FEAT(SIS_PROP, true)
 
 /*
  * Issue a WARN when we do multiple update_rq_clock() calls

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[PATCH 4.12 00/27] 4.12.1-stable review Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:20 +0200
  [PATCH 4.12 14/27] tpm: Issue a TPM2_Shutdown for TPM2 devices. Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:20 +0200
  [PATCH 4.12 26/27] xen: avoid deadlock in xenbus driver Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:20 +0200
  [PATCH 4.12 05/27] usb: usbip: set buffer pointers to NULL after free Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:20 +0200
  [PATCH 4.12 02/27] RDMA/uverbs: Check port number supplied by user verbs cmds Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:20 +0200
  [PATCH 4.12 19/27] sched/core: Implement new approach to scale select_idle_cpu() Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:20 +0200
  [PATCH 4.12 01/27] driver core: platform: fix race condition with driver_override Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:20 +0200
  [PATCH 4.12 06/27] Add USB quirk for HVR-950q to avoid intermittent device resets Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:20 +0200
  [PATCH 4.12 25/27] sched/numa: Hide numa_wake_affine() from UP build Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:30 +0200
  [PATCH 4.12 17/27] x86/uaccess: Optimize copy_user_enhanced_fast_string() for short strings Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:30 +0200
  [PATCH 4.12 04/27] USB: serial: cp210x: add ID for CEL EM3588 USB ZigBee stick Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:40 +0200
  [PATCH 4.12 13/27] Add "shutdown" to "struct class". Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:40 +0200
  [PATCH 4.12 11/27] xhci: Limit USB2 port wake support for AMD Promontory hosts Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:40 +0200
  [PATCH 4.12 18/27] sched/fair, cpumask: Export for_each_cpu_wrap() Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:40 +0200
  [PATCH 4.12 23/27] sched/numa: Implement NUMA node level wake_affine() Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:40 +0200
  [PATCH 4.12 24/27] sched/fair: Remove effective_load() Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:40 +0200
  [PATCH 4.12 15/27] tpm: fix a kernel memory leak in tpm-sysfs.c Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:40 +0200
  [PATCH 4.12 20/27] sched/numa: Use down_read_trylock() for the mmap_sem Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:40 +0200
  [PATCH 4.12 08/27] USB: core: fix device node leak Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:40 +0200
  [PATCH 4.12 12/27] gfs2: Fix glock rhashtable rcu bug Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:40 +0200
  [PATCH 4.12 03/27] usb: dwc3: replace %p with %pK Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:40 +0200
  [PATCH 4.12 27/27] crypto: drbg - Fixes panic in wait_for_completion call Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:40 +0200
  [PATCH 4.12 22/27] sched/fair: Simplify wake_affine() for the single socket case Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:40 +0200
  [PATCH 4.12 07/27] usb: Fix typo in the definition of Endpoint[out]Request Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-10 19:40 +0200
  Re: [PATCH 4.12 00/27] 4.12.1-stable review Guenter Roeck <linux@roeck-us.net> - 2017-07-11 03:30 +0200
    Re: [PATCH 4.12 00/27] 4.12.1-stable review Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-11 11:50 +0200

csiph-web