Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1225113

Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace signal_struct->group_rwsem with a global percpu_rwsem) causes regression for libvirt/kvm

From Christian Borntraeger <borntraeger@de.ibm.com>
Newsgroups linux.kernel
Subject Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace signal_struct->group_rwsem with a global percpu_rwsem) causes regression for libvirt/kvm
Date 2015-09-15 15:40 +0200
Message-ID <q8YCL-6fP-49@gated-at.bofh.it> (permalink)
References <q8XdE-4lR-29@gated-at.bofh.it> <q8Y9K-5HB-47@gated-at.bofh.it>
Organization linux.* mail to news gateway

Show all headers | View raw

Am 15.09.2015 um 15:05 schrieb Peter Zijlstra:
> On Tue, Sep 15, 2015 at 02:05:14PM +0200, Christian Borntraeger wrote:
>> Tejun,
>>
>>
>> commit d59cfc09c32a2ae31f1c3bc2983a0cd79afb3f14 (sched, cgroup: replace 
>> signal_struct->group_rwsem with a global percpu_rwsem) causes some noticably
>> hickups when starting several kvm guests (which libvirt will move into cgroups
>> - each vcpu thread and each i/o thread)
>> When you now start lots of guests in parallel on a bigger system (32CPUs with
>> 2way smt in my case) the system is so busy that systemd runs into several timeouts
>> like "Did not receive a reply. Possible causes include: the remote application did
>> not send a reply, the message bus security policy blocked the reply, the reply
>> timeout expired, or the network connection was broken."
>>
>> The problem seems to be that the newly used percpu_rwsem does a
>> rcu_synchronize_sched_expedited for all write downs/ups.
> 
> Can you try:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev.2015.09.11ab

yes, dev.2015.09.11a seems to help, thanks. Getting rid of the expedited hammer was
really helpful - I guess.

> 
> those include Oleg's rework of the percpu rwsem which should hopefully
> improve things somewhat.
> 
> But yes, pounding a global lock on a big machine will always suck.

By hacking out the fast path I actually degraded percpu rwsem to a real global lock, but
things were still a lot faster. 
I am wondering why the old code behaved in such fatal ways. Is there some interaction 
between waiting for a reschedule in the synchronize_sched writer and some fork code 
actually waiting for the read side to get the lock together with some rescheduling going
on waiting for a lock that fork holds? lockdep does not give me an hints so I have no clue :-(


Christian

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-15 14:10 +0200
  Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Peter Zijlstra <peterz@infradead.org> - 2015-09-15 15:10 +0200
    Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-15 15:40 +0200
      Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Tejun Heo <tj@kernel.org> - 2015-09-15 16:00 +0200
      Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Paolo Bonzini <pbonzini@redhat.com> - 2015-09-15 18:50 +0200
        Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-09-15 19:40 +0200
          Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Paolo Bonzini <pbonzini@redhat.com> - 2015-09-16 10:40 +0200
            Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-16 11:00 +0200
              Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Paolo Bonzini <pbonzini@redhat.com> - 2015-09-16 11:20 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes  regression for libvirt/kvm Oleg Nesterov <oleg@redhat.com> - 2015-09-16 14:30 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Paolo Bonzini <pbonzini@redhat.com> - 2015-09-16 14:40 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes  regression for libvirt/kvm Oleg Nesterov <oleg@redhat.com> - 2015-09-16 14:50 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-16 15:00 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Tejun Heo <tj@kernel.org> - 2015-09-16 16:20 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Paolo Bonzini <pbonzini@redhat.com> - 2015-09-16 16:20 +0200
        Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-15 23:20 +0200
          Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Tejun Heo <tj@kernel.org> - 2015-09-15 23:30 +0200
            Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-09-15 23:40 +0200
              Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Tejun Heo <tj@kernel.org> - 2015-09-16 00:30 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-09-16 01:40 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Tejun Heo <tj@kernel.org> - 2015-09-16 03:30 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-09-16 06:40 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Tejun Heo <tj@kernel.org> - 2015-09-16 13:10 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-16 09:50 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-16 13:00 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Tejun Heo <tj@kernel.org> - 2015-09-16 13:10 +0200
                Re: [4.2] commit d59cfc09c32 (sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem) causes regression for  libvirt/kvm Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-16 14:00 +0200
  [PATCH cgroup/for-4.3-fixes 1/2] Revert "cgroup: simplify  threadgroup locking" Tejun Heo <tj@kernel.org> - 2015-09-16 18:00 +0200
    [PATCH cgroup/for-4.3-fixes 2/2] Revert "sched, cgroup: replace  signal_struct->group_rwsem with a global percpu_rwsem" Tejun Heo <tj@kernel.org> - 2015-09-16 18:00 +0200
    Re: [PATCH cgroup/for-4.3-fixes 1/2] Revert "cgroup: simplify  threadgroup locking" Oleg Nesterov <oleg@redhat.com> - 2015-09-16 19:10 +0200
    Re: [PATCH cgroup/for-4.3-fixes 1/2] Revert "cgroup: simplify  threadgroup locking" Christian Borntraeger <borntraeger@de.ibm.com> - 2015-09-16 20:50 +0200

csiph-web