Re: [RFC PATCH v2] membarrier: expedited private command

From	Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Newsgroups	linux.kernel
Subject	Re: [RFC PATCH v2] membarrier: expedited private command
Date	2017-07-31 21:40 +0200
Message-ID	<u9oHL-1Rm-5@gated-at.bofh.it> (permalink)
References	(2 earlier) <u8c5Y-3fm-7@gated-at.bofh.it> <u8fwS-5yW-11@gated-at.bofh.it> <u8gCC-6cX-3@gated-at.bofh.it> <u8gVY-6za-37@gated-at.bofh.it> <u8pcS-3Ah-15@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

----- On Jul 28, 2017, at 9:58 PM, Nicholas Piggin npiggin@gmail.com wrote:

> On Fri, 28 Jul 2017 17:06:53 +0000 (UTC)
> Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
> 
>> ----- On Jul 28, 2017, at 12:46 PM, Peter Zijlstra peterz@infradead.org wrote:
>> 
>> > On Fri, Jul 28, 2017 at 03:38:15PM +0000, Mathieu Desnoyers wrote:
>> >> > Which only leaves PPC stranded.. but the 'good' news is that mpe says
>> >> > they'll probably need a barrier in switch_mm() in any case.
>> >> 
>> >> As I pointed out in my other email, I plan to do this:
>> >> 
>> >> --- a/kernel/sched/core.c
>> >> +++ b/kernel/sched/core.c
>> >> @@ -2636,6 +2636,11 @@ static struct rq *finish_task_switch(struct task_struct
>> >> *prev)
>> >>         vtime_task_switch(prev);
>> >>         perf_event_task_sched_in(prev, current);
>> > 
>> > Here would place it _inside_ the rq->lock, which seems to make more
>> > sense given the purpose of the barrier, but either way works given its
>> > definition.
>> 
>> Given its naming "...after_unlock_lock", I thought it would be clearer to put
>> it after the unlock. Anyway, this barrier does not seem to be used to ensure
>> the release barrier per se (unlock already has release semantic), but rather
>> ensures a full memory barrier wrt memory accesses that are synchronized by
>> means other than this this lock.
>> 
>> >   
>> >>         finish_lock_switch(rq, prev);
>> > 
>> > You could put the whole thing inside IS_ENABLED(CONFIG_SYSMEMBARRIER) or
>> > something.
>> 
>> I'm tempted to wait until we hear from powerpc maintainers, so we learn
>> whether they deeply care about this extra barrier in finish_task_switch()
>> before making it conditional on CONFIG_MEMBARRIER.
>> 
>> Having a guaranteed barrier after context switch on all architectures may
>> have other uses.
> 
> I haven't had time to read the thread and understand exactly why you need
> this extra barrier, I'll do it next week. Thanks for cc'ing us on it.
> 
> A smp_mb is pretty expensive on powerpc CPUs. Removing the sync from
> switch_to increased thread switch performance by 2-3%. Putting it in
> switch_mm may be a little less painful, but still we have to weigh it
> against the benefit of this new functionality. Would that be a net win
> for the average end-user? Seems unlikely.
> 
> But we also don't want to lose sys_membarrier completely. Would it be too
> painful to make  MEMBARRIER_CMD_PRIVATE_EXPEDITED return error, or make it
> fall back to a slower case if we decide not to implement it?

The need for an expedited membarrier comes from a need to use it to implement
synchronization schemes like hazard pointers, RCU, and garbage collectors in
user-space. One example is the use-case of hazard pointers. If the memory
free is implemented in the same thread doing the retire, the slowdown
introduced by non-expedited membarrier is not acceptable at all. In that case,
only an expedited membarrier brings an acceptable slowdown. The user's
alternative currently is to rely on undocumented side-effects of mprotect()
to achieve the same result. This happens to work on some architectures, and
may break in the future.

If users do not have membarrier expedited on a given architecture, and are
told that mprotect() does not provide the barrier guarantees they are looking
for, then they would have to add heavy-weight memory barriers on many
user-space fast-paths on those specific architectures, assuming they are
willing to go through that trouble.

I understand that the 2-3% overhead when switching between threads is a big
deal. Do you have numbers on the overhead added by a memory barrier in
switch_mm ? I suspect that switching between processes (including the
cost of following cache line and TLB misses) will be quite heavier in
the first place.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Thread

[RFC PATCH v2] membarrier: expedited private command Mathieu Desnoyers <mathieu.desnoyers@efficios.com> - 2017-07-27 23:20 +0200
  Re: [RFC PATCH v2] membarrier: expedited private command "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-28 00:20 +0200
    Re: [RFC PATCH v2] membarrier: expedited private command Mathieu Desnoyers <mathieu.desnoyers@efficios.com> - 2017-07-28 00:40 +0200
      Re: [RFC PATCH v2] membarrier: expedited private command "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-07-28 01:00 +0200
  Re: [RFC PATCH v2] membarrier: expedited private command Peter Zijlstra <peterz@infradead.org> - 2017-07-28 11:00 +0200
    Re: [RFC PATCH v2] membarrier: expedited private command Peter Zijlstra <peterz@infradead.org> - 2017-07-28 13:20 +0200
    Re: [RFC PATCH v2] membarrier: expedited private command Peter Zijlstra <peterz@infradead.org> - 2017-07-28 14:00 +0200
      Re: [RFC PATCH v2] membarrier: expedited private command Mathieu Desnoyers <mathieu.desnoyers@efficios.com> - 2017-07-28 17:40 +0200
        Re: [RFC PATCH v2] membarrier: expedited private command Peter Zijlstra <peterz@infradead.org> - 2017-07-28 18:50 +0200
          Re: [RFC PATCH v2] membarrier: expedited private command Mathieu Desnoyers <mathieu.desnoyers@efficios.com> - 2017-07-28 19:10 +0200
            Re: [RFC PATCH v2] membarrier: expedited private command Nicholas Piggin <npiggin@gmail.com> - 2017-07-29 04:00 +0200
              Re: [RFC PATCH v2] membarrier: expedited private command Peter Zijlstra <peterz@infradead.org> - 2017-07-29 11:30 +0200
                Re: [RFC PATCH v2] membarrier: expedited private command Nicholas Piggin <npiggin@gmail.com> - 2017-07-29 11:50 +0200
                Re: [RFC PATCH v2] membarrier: expedited private command Peter Zijlstra <peterz@infradead.org> - 2017-07-29 13:00 +0200
                Re: [RFC PATCH v2] membarrier: expedited private command Nicholas Piggin <npiggin@gmail.com> - 2017-07-29 11:50 +0200
              Re: [RFC PATCH v2] membarrier: expedited private command Mathieu Desnoyers <mathieu.desnoyers@efficios.com> - 2017-07-31 21:40 +0200
      Re: [RFC PATCH v2] membarrier: expedited private command Michael Ellerman <mpe@ellerman.id.au> - 2017-07-31 15:30 +0200
        Re: [RFC PATCH v2] membarrier: expedited private command Peter Zijlstra <peterz@infradead.org> - 2017-07-31 15:40 +0200
        Re: [RFC PATCH v2] membarrier: expedited private command Nicholas Piggin <npiggin@gmail.com> - 2017-08-01 02:40 +0200
          Re: [RFC PATCH v2] membarrier: expedited private command Mathieu Desnoyers <mathieu.desnoyers@efficios.com> - 2017-08-01 03:40 +0200
            Re: [RFC PATCH v2] membarrier: expedited private command Nicholas Piggin <npiggin@gmail.com> - 2017-08-01 04:10 +0200
              Re: [RFC PATCH v2] membarrier: expedited private command Peter Zijlstra <peterz@infradead.org> - 2017-08-01 10:20 +0200
                Re: [RFC PATCH v2] membarrier: expedited private command Nicholas Piggin <npiggin@gmail.com> - 2017-08-01 12:00 +0200
                Re: [RFC PATCH v2] membarrier: expedited private command Peter Zijlstra <peterz@infradead.org> - 2017-08-01 12:30 +0200
                Re: [RFC PATCH v2] membarrier: expedited private command Avi Kivity <avi@scylladb.com> - 2017-08-01 12:40 +0200
                Re: [RFC PATCH v2] membarrier: expedited private command Peter Zijlstra <peterz@infradead.org> - 2017-08-01 12:50 +0200
                Re: [RFC PATCH v2] membarrier: expedited private command Nicholas Piggin <npiggin@gmail.com> - 2017-08-01 12:40 +0200
                Re: [RFC PATCH v2] membarrier: expedited private command Peter Zijlstra <peterz@infradead.org> - 2017-08-01 13:10 +0200
                Re: [RFC PATCH v2] membarrier: expedited private command Nicholas Piggin <npiggin@gmail.com> - 2017-08-01 14:00 +0200
                Re: [RFC PATCH v2] membarrier: expedited private command "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-08-01 15:30 +0200
                Re: [RFC PATCH v2] membarrier: expedited private command Peter Zijlstra <peterz@infradead.org> - 2017-08-01 16:20 +0200
                Re: [RFC PATCH v2] membarrier: expedited private command "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-08-02 01:40 +0200
                Re: [RFC PATCH v2] membarrier: expedited private command Nicholas Piggin <npiggin@gmail.com> - 2017-08-02 02:50 +0200
    Re: [RFC PATCH v2] membarrier: expedited private command Mathieu Desnoyers <mathieu.desnoyers@efficios.com> - 2017-07-28 17:40 +0200

csiph-web