[4.2] commit d59cfc09c32 (sched, cgroup: replace signal_struct->group_rwsem with a global percpu_rwsem) causes regression for libvirt/kvm

From	Christian Borntraeger <borntraeger@de.ibm.com>
Newsgroups	linux.kernel
Subject	[4.2] commit d59cfc09c32 (sched, cgroup: replace signal_struct->group_rwsem with a global percpu_rwsem) causes regression for libvirt/kvm
Date	2015-09-15 14:10 +0200
Message-ID	<q8XdE-4lR-29@gated-at.bofh.it> (permalink)
Organization	linux.* mail to news gateway

Show all headers | View raw

Tejun,


commit d59cfc09c32a2ae31f1c3bc2983a0cd79afb3f14 (sched, cgroup: replace 
signal_struct->group_rwsem with a global percpu_rwsem) causes some noticably
hickups when starting several kvm guests (which libvirt will move into cgroups
- each vcpu thread and each i/o thread)
When you now start lots of guests in parallel on a bigger system (32CPUs with
2way smt in my case) the system is so busy that systemd runs into several timeouts
like "Did not receive a reply. Possible causes include: the remote application did
not send a reply, the message bus security policy blocked the reply, the reply
timeout expired, or the network connection was broken."

The problem seems to be that the newly used percpu_rwsem does a
rcu_synchronize_sched_expedited for all write downs/ups.

Hacking the percpu_rw_semaphore to always go the slow path and avoid the
synchronize_sched seems to fix the issue. For some (yet unknown to me) reason
the synchronize_sched and the fast path seems to block writers for incredibly
long times.

To trigger the problem, the guest must be CPU bound, iow idle guests seem to
not trigger the pathological case. Any idea how to improve the situation, 
this looks like a real regression for larger kvm installations.

Christian

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Back to linux.kernel | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

[4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-15 14:10 +0200
  Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Peter Zijlstra <peterz@infradead.org> - 2015-09-15 15:10 +0200
    Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-15 15:40 +0200
      Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Tejun Heo <tj@kernel.org> - 2015-09-15 16:00 +0200
      Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Paolo Bonzini <pbonzini@redhat.com> - 2015-09-15 18:50 +0200
        Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-09-15 19:40 +0200
          Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Paolo Bonzini <pbonzini@redhat.com> - 2015-09-16 10:40 +0200
            Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-16 11:00 +0200
              Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Paolo Bonzini <pbonzini@redhat.com> - 2015-09-16 11:20 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes  regression for libvirt/kvm Oleg Nesterov <oleg@redhat.com> - 2015-09-16 14:30 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Paolo Bonzini <pbonzini@redhat.com> - 2015-09-16 14:40 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes  regression for libvirt/kvm Oleg Nesterov <oleg@redhat.com> - 2015-09-16 14:50 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-16 15:00 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Tejun Heo <tj@kernel.org> - 2015-09-16 16:20 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Paolo Bonzini <pbonzini@redhat.com> - 2015-09-16 16:20 +0200
        Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-15 23:20 +0200
          Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Tejun Heo <tj@kernel.org> - 2015-09-15 23:30 +0200
            Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-09-15 23:40 +0200
              Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Tejun Heo <tj@kernel.org> - 2015-09-16 00:30 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-09-16 01:40 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Tejun Heo <tj@kernel.org> - 2015-09-16 03:30 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-09-16 06:40 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Tejun Heo <tj@kernel.org> - 2015-09-16 13:10 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-16 09:50 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-16 13:00 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Tejun Heo <tj@kernel.org> - 2015-09-16 13:10 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-16 14:00 +0200
  [PATCH cgroup/for-4.3-fixes 1/2] Revert "cgroup: simplify  threadgroup locking" Tejun Heo <tj@kernel.org> - 2015-09-16 18:00 +0200
    [PATCH cgroup/for-4.3-fixes 2/2] Revert "sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem" Tejun Heo <tj@kernel.org> - 2015-09-16 18:00 +0200
    Re: [PATCH cgroup/for-4.3-fixes 1/2] Revert "cgroup: simplify  threadgroup locking" Oleg Nesterov <oleg@redhat.com> - 2015-09-16 19:10 +0200
    Re: [PATCH cgroup/for-4.3-fixes 1/2] Revert "cgroup: simplify  threadgroup locking" Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-16 20:50 +0200

csiph-web