Re: klp_task_patch: was: [RFC PATCH v2 17/18] livepatch: change to a per-task consistency model

From	Petr Mladek <pmladek@suse.com>
Newsgroups	linux.kernel
Subject	Re: klp_task_patch: was: [RFC PATCH v2 17/18] livepatch: change to a per-task consistency model
Date	2016-05-09 14:30 +0200
Message-ID	<rwStY-5zM-15@gated-at.bofh.it> (permalink)
References	(1 earlier) <rt12O-1W5-9@gated-at.bofh.it> <rv6hJ-5S0-59@gated-at.bofh.it> <rv9fz-85-3@gated-at.bofh.it> <rvq6J-7IA-5@gated-at.bofh.it> <rvNcZ-57y-3@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

On Fri 2016-05-06 07:38:55, Josh Poimboeuf wrote:
> On Thu, May 05, 2016 at 01:57:01PM +0200, Petr Mladek wrote:
> > I have missed that the two commands are called with preemption
> > disabled. So, I had the following crazy scenario in mind:
> > 
> > 
> > CPU0				CPU1
> > 
> > klp_enable_patch()
> > 
> >   klp_target_state = KLP_PATCHED;
> > 
> >   for_each_task()
> >      set TIF_PENDING_PATCH
> > 
> > 				# task 123
> > 
> > 				if (klp_patch_pending(current)
> > 				  klp_patch_task(current)
> > 
> >                                     clear TIF_PENDING_PATCH
> > 
> > 				    smp_rmb();
> > 
> > 				    # switch to assembly of
> > 				    # klp_patch_task()
> > 
> > 				    mov klp_target_state, %r12
> > 
> > 				    # interrupt and schedule
> > 				    # another task
> > 
> > 
> >   klp_reverse_transition();
> > 
> >     klp_target_state = KLP_UNPATCHED;
> > 
> >     klt_try_to_complete_transition()
> > 
> >       task = 123;
> >       if (task->patch_state == klp_target_state;
> >          return 0;
> > 
> >     => task 123 is in target state and does
> >     not block conversion
> > 
> >   klp_complete_transition()
> > 
> > 
> >   # disable previous patch on the stack
> >   klp_disable_patch();
> > 
> >     klp_target_state = KLP_UNPATCHED;
> >   
> >   
> > 				    # task 123 gets scheduled again
> > 				    lea %r12, task->patch_state
> > 
> > 				    => it happily stores an outdated
> > 				    state
> > 
> 
> Thanks for the clear explanation, this helps a lot.
> 
> > This is why the two functions should get called with preemption
> > disabled. We should document it at least. I imagine that we will
> > use them later also in another context and nobody will remember
> > this crazy scenario.
> > 
> > Well, even disabled preemption does not help. The process on
> > CPU1 might be also interrupted by an NMI and do some long
> > printk in it.
> > 
> > IMHO, the only safe approach is to call klp_patch_task()
> > only for "current" on a safe place. Then this race is harmless.
> > The switch happen on a safe place, so that it does not matter
> > into which state the process is switched.
> 
> I'm not sure about this solution.  When klp_complete_transition() is
> called, we need all tasks to be patched, for good.  We don't want any of
> them to randomly switch to the wrong state at some later time in the
> middle of a future patch operation.  How would changing klp_patch_task()
> to only use "current" prevent that?

You are right that it is pity but it really should be safe because
it is not entirely random.

If the race happens and assign an outdated value, there are two
situations:

1. It is assigned when there is not transition in the progress.
   Then it is OK because it will be ignored by the ftrace handler.
   The right state will be set before the next transition starts.

2. It is assigned when some other transition is in progress.
   Then it is OK as long as the function is called from "current".
   The "wrong" state will be used consistently. It will switch
   to the right state on another safe state.


> > By other words, the task state might be updated only
> > 
> >    + by the task itself on a safe place
> >    + by other task when the updated on is sleeping on a safe place
> > 
> > This should be well documented and the API should help to avoid
> > a misuse.
> 
> I think we could fix it to be safe for future callers who might not have
> preemption disabled with a couple of changes to klp_patch_task():
> disabling preemption and testing/clearing the TIF_PATCH_PENDING flag
> before changing the patch state:
> 
>   void klp_patch_task(struct task_struct *task)
>   {
>   	preempt_disable();
>   
>   	if (test_and_clear_tsk_thread_flag(task, TIF_PATCH_PENDING))
>   		task->patch_state = READ_ONCE(klp_target_state);
>   
>   	preempt_enable();
>   }

It reduces the race window a bit but it is still there. For example,
NMI still might add a huge delay between reading klp_target_state
and assigning task->patch state.

What about the following?

/*
 * This function might assign an outdated value if the transaction
`* is reverted and finalized in parallel. But it is safe. If the value
 * is assigned outside of a transaction, it is ignored and the next
 * transaction will set the right one. Or if it gets assigned
 * inside another transaction, it will repeat the cycle and
 * set the right state.
 */
void klp_update_current_patch_state()
{
	while (test_and_clear_tsk_thread_flag(current, TIF_PATCH_PENDING))
		current->patch_state = READ_ONCE(klp_target_state);
}

Note that the disabled preemption helped only partially,
so I think that it was not really needed.

Hmm, it means that the task->patch_state  might be either
KLP_PATCHED or KLP_UNPATCHED outside a transition. I wonder
if the tristate really brings some advantages.


Alternatively, we might synchronize the operation with klp_mutex.
The function is called in a slow path and in a safe context.
Well, it might cause contention on the lock when many CPUs are
trying to update their tasks.

Best Regards,
Petr

Thread

[RFC PATCH v2 00/18] livepatch: hybrid consistency model Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-28 22:50 +0200
  [RFC PATCH v2 14/18] livepatch: remove unnecessary object loaded check Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-28 22:50 +0200
  [RFC PATCH v2 10/18] livepatch/powerpc: add TIF_PATCH_PENDING thread flag Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-28 22:50 +0200
    Re: [RFC PATCH v2 10/18] livepatch/powerpc: add TIF_PATCH_PENDING  thread flag Petr Mladek <pmladek@suse.com> - 2016-05-03 11:10 +0200
      Re: [RFC PATCH v2 10/18] livepatch/powerpc: add TIF_PATCH_PENDING  thread flag Miroslav Benes <mbenes@suse.cz> - 2016-05-03 14:10 +0200
  [RFC PATCH v2 16/18] livepatch: store function sizes Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-28 22:50 +0200
  [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ tracking Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-28 22:50 +0200
    Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ tracking Andy Lutomirski <luto@amacapital.net> - 2016-04-29 20:10 +0200
      Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ tracking Andy Lutomirski <luto@amacapital.net> - 2016-04-29 22:20 +0200
        Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ  tracking Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-29 22:30 +0200
          Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ tracking Andy Lutomirski <luto@amacapital.net> - 2016-04-29 22:40 +0200
            Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ  tracking Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-29 23:30 +0200
              Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ tracking Andy Lutomirski <luto@amacapital.net> - 2016-04-29 23:40 +0200
                Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ  tracking Jiri Kosina <jikos@kernel.org> - 2016-04-30 00:20 +0200
                Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ  tracking Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-30 01:00 +0200
                Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ tracking Andy Lutomirski <luto@amacapital.net> - 2016-04-30 02:20 +0200
                Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ  tracking Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-30 00:50 +0200
                Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ tracking Andy Lutomirski <luto@amacapital.net> - 2016-04-30 02:10 +0200
                Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ  tracking Josh Poimboeuf <jpoimboe@redhat.com> - 2016-05-02 16:00 +0200
                Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ tracking Andy Lutomirski <luto@amacapital.net> - 2016-05-02 18:00 +0200
                Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ  tracking Josh Poimboeuf <jpoimboe@redhat.com> - 2016-05-02 19:40 +0200
                Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ tracking Andy Lutomirski <luto@amacapital.net> - 2016-05-02 20:20 +0200
                Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ  tracking Ingo Molnar <mingo@kernel.org> - 2016-05-02 20:40 +0200
                Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ  tracking Josh Poimboeuf <jpoimboe@redhat.com> - 2016-05-02 21:50 +0200
                Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ  tracking Jiri Kosina <jikos@kernel.org> - 2016-05-02 22:00 +0200
                Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ  tracking Jiri Kosina <jikos@kernel.org> - 2016-05-02 22:10 +0200
                Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ tracking Andy Lutomirski <luto@amacapital.net> - 2016-05-03 02:50 +0200
                RE: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ  tracking David Laight <David.Laight@ACULAB.COM> - 2016-05-04 17:20 +0200
      Re: [RFC PATCH v2 05/18] sched: add task flag for preempt IRQ  tracking Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-29 22:20 +0200
  [RFC PATCH v2 09/18] livepatch/x86: add TIF_PATCH_PENDING thread flag Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-28 22:50 +0200
    Re: [RFC PATCH v2 09/18] livepatch/x86: add TIF_PATCH_PENDING thread flag Andy Lutomirski <luto@amacapital.net> - 2016-04-29 20:10 +0200
      Re: [RFC PATCH v2 09/18] livepatch/x86: add TIF_PATCH_PENDING thread  flag Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-29 22:20 +0200
  [RFC PATCH v2 02/18] x86/asm/head: use a common function for starting CPUs Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-28 22:50 +0200
  [RFC PATCH v2 13/18] livepatch: separate enabled and patched states Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-28 22:50 +0200
    Re: [RFC PATCH v2 13/18] livepatch: separate enabled and patched  states Petr Mladek <pmladek@suse.com> - 2016-05-03 11:40 +0200
      Re: [RFC PATCH v2 13/18] livepatch: separate enabled and patched  states Josh Poimboeuf <jpoimboe@redhat.com> - 2016-05-03 15:50 +0200
  [RFC PATCH v2 06/18] x86: dump_trace() error handling Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-28 22:50 +0200
    Re: [RFC PATCH v2 06/18] x86: dump_trace() error handling Minfei Huang <mnghuan@gmail.com> - 2016-04-29 15:50 +0200
      Re: [RFC PATCH v2 06/18] x86: dump_trace() error handling Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-29 16:10 +0200
  [RFC PATCH v2 03/18] x86/asm/head: standardize the bottom of the stack for idle tasks Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-28 23:00 +0200
    Re: [RFC PATCH v2 03/18] x86/asm/head: standardize the bottom of the  stack for idle tasks Brian Gerst <brgerst@gmail.com> - 2016-04-29 20:50 +0200
      Re: [RFC PATCH v2 03/18] x86/asm/head: standardize the bottom of the  stack for idle tasks Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-29 22:30 +0200
    Re: [RFC PATCH v2 03/18] x86/asm/head: standardize the bottom of the  stack for idle tasks Andy Lutomirski <luto@kernel.org> - 2016-04-29 21:40 +0200
      Re: [RFC PATCH v2 03/18] x86/asm/head: standardize the bottom of the  stack for idle tasks Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-29 23:00 +0200
        Re: [RFC PATCH v2 03/18] x86/asm/head: standardize the bottom of the  stack for idle tasks Andy Lutomirski <luto@amacapital.net> - 2016-04-29 23:40 +0200
          Re: [RFC PATCH v2 03/18] x86/asm/head: standardize the bottom of the  stack for idle tasks Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-30 01:30 +0200
            Re: [RFC PATCH v2 03/18] x86/asm/head: standardize the bottom of the  stack for idle tasks Andy Lutomirski <luto@amacapital.net> - 2016-04-30 02:20 +0200
  [RFC PATCH v2 04/18] x86: move _stext marker before head code Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-28 23:00 +0200
  [RFC PATCH v2 01/18] x86/asm/head: clean up initial stack variable Josh Poimboeuf <jpoimboe@redhat.com> - 2016-04-28 23:00 +0200
  Re: [RFC PATCH v2 17/18] livepatch: change to a per-task consistency  model Petr Mladek <pmladek@suse.com> - 2016-05-04 10:50 +0200
    Re: [RFC PATCH v2 17/18] livepatch: change to a per-task consistency  model Josh Poimboeuf <jpoimboe@redhat.com> - 2016-05-04 18:00 +0200
      Re: [RFC PATCH v2 17/18] livepatch: change to a per-task consistency  model Miroslav Benes <mbenes@suse.cz> - 2016-05-05 11:50 +0200
      Re: [RFC PATCH v2 17/18] livepatch: change to a per-task consistency  model Petr Mladek <pmladek@suse.com> - 2016-05-05 15:10 +0200
  barriers: was: [RFC PATCH v2 17/18] livepatch: change to a per-task  consistency model Petr Mladek <pmladek@suse.com> - 2016-05-04 14:40 +0200
    Re: barriers: was: [RFC PATCH v2 17/18] livepatch: change to a  per-task consistency model Peter Zijlstra <peterz@infradead.org> - 2016-05-04 16:00 +0200
      Re: barriers: was: [RFC PATCH v2 17/18] livepatch: change to a  per-task consistency model Josh Poimboeuf <jpoimboe@redhat.com> - 2016-05-04 19:00 +0200
    Re: barriers: was: [RFC PATCH v2 17/18] livepatch: change to a  per-task consistency model Petr Mladek <pmladek@suse.com> - 2016-05-04 16:20 +0200
      Re: barriers: was: [RFC PATCH v2 17/18] livepatch: change to a  per-task consistency model Josh Poimboeuf <jpoimboe@redhat.com> - 2016-05-04 19:30 +0200
        Re: barriers: was: [RFC PATCH v2 17/18] livepatch: change to a  per-task consistency model Petr Mladek <pmladek@suse.com> - 2016-05-05 13:30 +0200
        Re: barriers: was: [RFC PATCH v2 17/18] livepatch: change to a  per-task consistency model Miroslav Benes <mbenes@suse.cz> - 2016-05-09 17:50 +0200
    Re: barriers: was: [RFC PATCH v2 17/18] livepatch: change to a  per-task consistency model Josh Poimboeuf <jpoimboe@redhat.com> - 2016-05-04 19:10 +0200
      Re: barriers: was: [RFC PATCH v2 17/18] livepatch: change to a  per-task consistency model Petr Mladek <pmladek@suse.com> - 2016-05-05 12:30 +0200
  klp_task_patch: was: [RFC PATCH v2 17/18] livepatch: change to a  per-task consistency model Petr Mladek <pmladek@suse.com> - 2016-05-04 16:50 +0200
    Re: klp_task_patch: was: [RFC PATCH v2 17/18] livepatch: change to  a per-task consistency model Jiri Kosina <jikos@kernel.org> - 2016-05-04 17:00 +0200
    Re: klp_task_patch: was: [RFC PATCH v2 17/18] livepatch: change to a  per-task consistency model Josh Poimboeuf <jpoimboe@redhat.com> - 2016-05-04 20:00 +0200
      Re: klp_task_patch: was: [RFC PATCH v2 17/18] livepatch: change to a  per-task consistency model Petr Mladek <pmladek@suse.com> - 2016-05-05 14:00 +0200
        Re: klp_task_patch: was: [RFC PATCH v2 17/18] livepatch: change to a  per-task consistency model Josh Poimboeuf <jpoimboe@redhat.com> - 2016-05-06 14:40 +0200
          Re: klp_task_patch: was: [RFC PATCH v2 17/18] livepatch: change to a  per-task consistency model Petr Mladek <pmladek@suse.com> - 2016-05-09 14:30 +0200
  Re: [RFC PATCH v2 17/18] livepatch: change to a per-task consistency  model Petr Mladek <pmladek@suse.com> - 2016-05-06 13:40 +0200
    Re: [RFC PATCH v2 17/18] livepatch: change to a per-task consistency  model Josh Poimboeuf <jpoimboe@redhat.com> - 2016-05-06 14:50 +0200
  Re: [RFC PATCH v2 17/18] livepatch: change to a per-task consistency  model Miroslav Benes <mbenes@suse.cz> - 2016-05-09 11:50 +0200
  Re: [RFC PATCH v2 17/18] livepatch: change to a per-task consistency  model Miroslav Benes <mbenes@suse.cz> - 2016-05-10 13:50 +0200

csiph-web