Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1320816 > unrolled thread

Re: [BUG] scheduler doesn't balance thread to idle cpu for 3 seconds

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [BUG] scheduler doesn't balance thread to idle cpu for 3 seconds Jan Stancek <jstancek@redhat.com> - 2016-01-28 17:00 +0100
    Re: [BUG] scheduler doesn't balance thread to idle cpu for 3 seconds Peter Zijlstra <peterz@infradead.org> - 2016-01-28 18:50 +0100
      Re: [BUG] scheduler doesn't balance thread to idle cpu for 3  seconds Jan Stancek <jstancek@redhat.com> - 2016-01-28 20:00 +0100
        Re: [BUG] scheduler doesn't balance thread to idle cpu for 3 seconds Peter Zijlstra <peterz@infradead.org> - 2016-01-29 11:20 +0100
          Re: [BUG] scheduler doesn't balance thread to idle cpu for 3  seconds Jan Stancek <jstancek@redhat.com> - 2016-01-29 11:40 +0100
            Re: [BUG] scheduler doesn't balance thread to idle cpu for 3 seconds Jan Stancek <jstancek@redhat.com> - 2016-02-08 14:50 +0100

#1320816 — Re: [BUG] scheduler doesn't balance thread to idle cpu for 3 seconds

[Multipart message — attachments visible in raw view] — view raw

On 01/27/2016 03:52 PM, Jan Stancek wrote:
> Hello,
> 
> pthread_cond_wait_1/2 [1] is rarely failing for me on 4.5.0-rc1,
> on x86_64 KVM guest with 2 CPUs.
> 
> This test [1]:
> - spawns 2 SCHED_RR threads
> - first thread with higher priority sets alarm for 2 seconds and blocks on condition
> - second thread with lower priority is busy looping for 5 seconds
> - after 2 seconds alarm signal arrives and handler signals condition
> - high priority thread should resume running

I have slightly modified testcase, so it will finish immediately when high prio
thread is done. And also to allow it to compile outside of openposix testsuite.

Testcase is attached. I'm running it in following way:

gcc -O2 -pthread pthread_cond_wait_1.c
while [ True ]; do
  time ./a.out
  sleep 1
done

for couple thousand iterations. About half of those are
on system booted with init=/bin/bash.

> 
> But rarely I see that high priority thread doesn't resume running until
> low priority thread completes its 5 second busy loop.
> 
> Looking at traces (short version attached, long version at [2]),
> I see that after 2 seconds scheduler tries to wake up main thread, but it
> appears to do that on same CPU where SCHED_RR low prio thread is running,
> so nothing happens. Then scheduler makes numerous balance attempts,
> but main thread is not balanced to idle CPU.
> 
> My guess is this started with following commit, which changed weighted_cpuload():
>   commit b92486cbf2aa230d00f160664858495c81d2b37b
>   Author: Alex Shi <alex.shi@intel.com>
>   Date:   Thu Jun 20 10:18:50 2013 +0800
>     sched: Compute runnable load avg in cpu_load and cpu_avg_load_per_task

Here are some numbers gathered from kernels with HEAD at b92486c and
previous commit 83dfd52. System is 2 CPU KVM guest.

Each iteration measures how long it took for testcase to finish.
Ideally it should take about 2 seconds.

1. HEAD at 83dfd52 sched: Update cpu load after task_tick

  finish time [s]  |   iterations
----------------------------------
[    2,   2.2]     |       3134
[  2.2,   2.5]     |         18
[  2.5,     3]     |          0
[    3,     4]     |          0
[    4,     5]     |          0
[    5,   999]     |          0


2. HEAD at b92486c sched: Compute runnable load avg in cpu_load and cpu_avg_load_per_task

  finish time [s]  |   iterations
----------------------------------
[    2,   2.2]     |       1617
[  2.2,   2.5]     |         38
[  2.5,     3]     |        727
[    3,     4]     |        399
[    4,     5]     |         17
[    5,   999]     |         11

Regards,
Jan

> 
> I could reproduce it with HEAD set at above commit, I couldn't reproduce it
> with 3.10 kernel so far.
> 
> Regards,
> Jan
> 
> [1] https://github.com/linux-test-project/ltp/blob/master/testcases/open_posix_testsuite/functional/threads/condvar/pthread_cond_wait_1.c
> [2] http://jan.stancek.eu/tmp/pthread_cond_wait_failure/sched-trace1.tar.bz2
> 

[toc] | [next] | [standalone]

#1320917

On Thu, Jan 28, 2016 at 04:55:02PM +0100, Jan Stancek wrote:
> On 01/27/2016 03:52 PM, Jan Stancek wrote:
> > Hello,
> > 
> > pthread_cond_wait_1/2 [1] is rarely failing for me on 4.5.0-rc1,
> > on x86_64 KVM guest with 2 CPUs.
> > 
> > This test [1]:
> > - spawns 2 SCHED_RR threads
> > - first thread with higher priority sets alarm for 2 seconds and blocks on condition
> > - second thread with lower priority is busy looping for 5 seconds
> > - after 2 seconds alarm signal arrives and handler signals condition
> > - high priority thread should resume running
> 
> I have slightly modified testcase, so it will finish immediately when high prio
> thread is done. And also to allow it to compile outside of openposix testsuite.

Yeah, I 'fixed' the testcase too.

So I've had it run for almost 2 hours without a single fail. I've
hot-plugged my cpu count down to 2.

How long should I have to wait for a fail?

[toc] | [prev] | [next] | [standalone]

#1320948 — Re: [BUG] scheduler doesn't balance thread to idle cpu for 3 seconds

----- Original Message -----
> From: "Peter Zijlstra" <peterz@infradead.org>
> To: "Jan Stancek" <jstancek@redhat.com>
> Cc: "alex shi" <alex.shi@intel.com>, "guz fnst" <guz.fnst@cn.fujitsu.com>, mingo@redhat.com, jolsa@redhat.com,
> riel@redhat.com, linux-kernel@vger.kernel.org
> Sent: Thursday, 28 January, 2016 6:49:03 PM
> Subject: Re: [BUG] scheduler doesn't balance thread to idle cpu for 3 seconds
> 
> On Thu, Jan 28, 2016 at 04:55:02PM +0100, Jan Stancek wrote:
> > On 01/27/2016 03:52 PM, Jan Stancek wrote:
> > > Hello,
> > > 
> > > pthread_cond_wait_1/2 [1] is rarely failing for me on 4.5.0-rc1,
> > > on x86_64 KVM guest with 2 CPUs.
> > > 
> > > This test [1]:
> > > - spawns 2 SCHED_RR threads
> > > - first thread with higher priority sets alarm for 2 seconds and blocks
> > > on condition
> > > - second thread with lower priority is busy looping for 5 seconds
> > > - after 2 seconds alarm signal arrives and handler signals condition
> > > - high priority thread should resume running
> > 
> > I have slightly modified testcase, so it will finish immediately when high
> > prio
> > thread is done. And also to allow it to compile outside of openposix
> > testsuite.
> 
> Yeah, I 'fixed' the testcase too.
> 
> So I've had it run for almost 2 hours without a single fail. I've
> hot-plugged my cpu count down to 2.

I can try that too. I'm mostly seeing this on s390 and x86_64 KVM guests,
both have 2 CPUs.

Have you noticed if iteration times vary or if they stay consitently
at ~2 seconds?

> 
> How long should I have to wait for a fail?

It's about 1000-2000 iterations for me, which I think you covered
by now in those 2 hours.

Regards,
Jan

[toc] | [prev] | [next] | [standalone]

#1321630

On Thu, Jan 28, 2016 at 01:43:13PM -0500, Jan Stancek wrote:
> > How long should I have to wait for a fail?
> 
> It's about 1000-2000 iterations for me, which I think you covered
> by now in those 2 hours.

So I've been running:

  while ! ./pthread_cond_wait_1 ; do sleep 1; done

overnight on the machine, and have yet to hit a wobbly -- that is, its
still running.

Also note that I don't think failing this test is a bug per se.
Undesirable maybe, but within spec, since SIGALRM is process wide, so it
being delivered to the SCHED_OTHER task is accepted, and SCHED_OTHER has
no timeliness guarantees.

That said; if I could reliably reproduce I'd have a go at fixing this, I
suspect there's a 'fun' problem at the bottom of this.

[toc] | [prev] | [next] | [standalone]

#1321643 — Re: [BUG] scheduler doesn't balance thread to idle cpu for 3 seconds

----- Original Message -----
> From: "Peter Zijlstra" <peterz@infradead.org>
> To: "Jan Stancek" <jstancek@redhat.com>
> Cc: "alex shi" <alex.shi@intel.com>, "guz fnst" <guz.fnst@cn.fujitsu.com>, mingo@redhat.com, jolsa@redhat.com,
> riel@redhat.com, linux-kernel@vger.kernel.org
> Sent: Friday, 29 January, 2016 11:15:22 AM
> Subject: Re: [BUG] scheduler doesn't balance thread to idle cpu for 3 seconds
> 
> On Thu, Jan 28, 2016 at 01:43:13PM -0500, Jan Stancek wrote:
> > > How long should I have to wait for a fail?
> > 
> > It's about 1000-2000 iterations for me, which I think you covered
> > by now in those 2 hours.
> 
> So I've been running:
> 
>   while ! ./pthread_cond_wait_1 ; do sleep 1; done
> 
> overnight on the machine, and have yet to hit a wobbly -- that is, its
> still running.

I have seen similar result.

Then, instead of turning CPUs off, I spawned more low prio threads to scale
with number of CPUs on system:

@@ -213,10 +213,14 @@
                printf(ERROR_PREFIX "pthread_attr_setschedparam\n");
                exit(PTS_UNRESOLVED);
        }
-       rc = pthread_create(&low_id, &low_attr, low_priority_thread, NULL);
-       if (rc != 0) {
-               printf(ERROR_PREFIX "pthread_create\n");
-               exit(PTS_UNRESOLVED);
+
+       int i, ncpus = sysconf(_SC_NPROCESSORS_ONLN);
+       for (i = 0; i < ncpus - 1; i++) {
+               rc = pthread_create(&low_id, &low_attr, low_priority_thread, NULL);
+               if (rc != 0) {
+                       printf(ERROR_PREFIX "pthread_create\n");
+                       exit(PTS_UNRESOLVED);
+               }

and let this ran on 3 bare metal x86 systems over night (v4.5-rc1). It
failed on 2 systems (12 and 24 CPUs) with 1:1000 chance, it never failed
on 3rd one (4 CPUs).

> 
> Also note that I don't think failing this test is a bug per se.
> Undesirable maybe, but within spec, since SIGALRM is process wide, so it
> being delivered to the SCHED_OTHER task is accepted, and SCHED_OTHER has
> no timeliness guarantees.
> 
> That said; if I could reliably reproduce I'd have a go at fixing this, I
> suspect there's a 'fun' problem at the bottom of this.

Thanks for trying, I'll see if I can find some more reliable way.

Regards,
Jan

[toc] | [prev] | [next] | [standalone]

#1329061

[Multipart message — attachments visible in raw view] — view raw

On 01/29/2016 11:33 AM, Jan Stancek wrote:
>>
>> Also note that I don't think failing this test is a bug per se.
>> Undesirable maybe, but within spec, since SIGALRM is process wide, so it
>> being delivered to the SCHED_OTHER task is accepted, and SCHED_OTHER has
>> no timeliness guarantees.
>>
>> That said; if I could reliably reproduce I'd have a go at fixing this, I
>> suspect there's a 'fun' problem at the bottom of this.
> 
> Thanks for trying, I'll see if I can find some more reliable way.

I think I have found a more reliably way, however it requires an older
stable kernel: 3.12.53 up to 4.1.17.

Consider following scenario:
- all tasks on system have RT sched class
- main thread of reproducer becomes the only SCHED_OTHER task on system
- when alarm(2) expires, main thread is woken up on cpu that is occupied by
  busy looping RT thread (low_priority_thread)
- because main thread was sleeping for 2 seconds, its load has decayed to 0
- the only chance for main thread to run is if it gets balanced to idle CPU
- task_tick_fair() doesn't run, there is RT task running on this CPU
- main thread is on cfs run queue but its load stays 0
- load balancer never sees this CPU (group) as busy

Attached is reproducer and script, which tries to trigger scenario above.
I can reproduce it with 4.1.17 on baremetal 4 CPU x86_64 with about 1:50 chance.

In this setup failure state persists for a long time, perhaps indefinitely.
I tried extending RUNTIME to 10 minutes, main thread still wouldn't run.

One more clue: I could work around this issue if I forced an update_entity_load_avg()
on sched_entities that have not been updated for some time, as part of
periodic rebalance_domains() call.

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index c7c1d28..1b5fe80 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -5264,6 +5264,7 @@ static void update_blocked_averages(int cpu)
 	struct rq *rq = cpu_rq(cpu);
 	struct cfs_rq *cfs_rq;
 	unsigned long flags;
+	struct rb_node *rb;

 	raw_spin_lock_irqsave(&rq->lock, flags);
 	update_rq_clock(rq);
@@ -5281,6 +5282,19 @@ static void update_blocked_averages(int cpu)
 	}

 	raw_spin_unlock_irqrestore(&rq->lock, flags);
+
+	cfs_rq = &(cpu_rq(cpu)->cfs);
+	for (rb = rb_first_postorder(&cfs_rq->tasks_timeline); rb; rb = rb_next_postorder(rb)) {
+		struct sched_entity *se = rb_entry(rb, struct sched_entity, run_node);
+
+		// Task on rq has not been updated for 500ms :-(
+		if ((cfs_rq_clock_task(cfs_rq) - se->avg.last_runnable_update) > 500L * (1 << 20))
+			update_entity_load_avg(se, 1);
+	}
 }

 /*

Regards,
Jan

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web