Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1680461 > unrolled thread

Re: [PATCH 1/2] selftests: ftrace: Do not failure if there is unsupported tests

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 1/2] selftests: ftrace: Do not failure if there is  unsupported tests Masami Hiramatsu <mhiramat@kernel.org> - 2017-07-03 18:00 +0200
    Re: [PATCH 1/2] selftests: ftrace: Do not failure if there is  unsupported tests Steven Rostedt <rostedt@goodmis.org> - 2017-07-03 18:00 +0200
      Re: [PATCH 1/2] selftests: ftrace: Do not failure if there is  unsupported tests Masami Hiramatsu <mhiramat@kernel.org> - 2017-07-04 02:40 +0200

#1680461 — Re: [PATCH 1/2] selftests: ftrace: Do not failure if there is unsupported tests

On Mon, 3 Jul 2017 09:15:43 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> On Mon,  3 Jul 2017 14:25:36 +0900
> Masami Hiramatsu <mhiramat@kernel.org> wrote:
> 
> > Do not return failure exit code (1) for unsupported testcases,
> > since it is expected for stable kernels.
> > 
> > Previously, ftracetest is expected to run only on current
> > release for avoiding regressions. However, nowadays we run
> > it on stable kernels. This means some test cases must return
> > unsupported result. In such case, we should NOT exit
> > ftracetest with error status for unsupported results so that
> > kselftest (upper tests wrapper) shows it passed correctly.
> 
> I wonder if we should change kselftest instead. There are case where we
> want to report "unsupported" as a failure. For instance, I have tests
> where I enable everything, and if a test returns "unsupported" then it
> is a failure for me.

OK, that's a possible usecase. 

> 
> Can we add an option in kselftest, or to ftracetest that decides if
> unsupported is a failure or not? Otherwise I can not ack this patch.

I would rather like to add an option to ftracetest instead of
kselftest, because whether the tested feature should be supported
or not is hard to decide from testing framework. It should be
checked by manual.

Thank you,

> 
> -- Steve
> 
> 
> > 
> > Note that we continue to treat unresolved results as failure,
> > if test writers would like to notice user that the test result
> > should be reviewed, they can use exit_unresolved.
> > 
> > Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
> > ---
> >  tools/testing/selftests/ftrace/ftracetest |    2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest
> > index 14a03ea..290cd42 100755
> > --- a/tools/testing/selftests/ftrace/ftracetest
> > +++ b/tools/testing/selftests/ftrace/ftracetest
> > @@ -187,7 +187,7 @@ eval_result() { # sigval
> >      $UNSUPPORTED)
> >        prlog "	[UNSUPPORTED]"
> >        UNSUPPORTED_CASES="$UNSUPPORTED_CASES $CASENO"
> > -      return 1 # this is not a bug, but the result should be reported.
> > +      return 0 # this is not a bug.
> >      ;;
> >      $XFAIL)
> >        prlog "	[XFAIL]"
> 


-- 
Masami Hiramatsu <mhiramat@kernel.org>

[toc] | [next] | [standalone]

#1680462

On Tue, 4 Jul 2017 00:52:32 +0900
Masami Hiramatsu <mhiramat@kernel.org> wrote:
> 
> > 
> > Can we add an option in kselftest, or to ftracetest that decides if
> > unsupported is a failure or not? Otherwise I can not ack this patch.  
> 
> I would rather like to add an option to ftracetest instead of
> kselftest, because whether the tested feature should be supported
> or not is hard to decide from testing framework. It should be
> checked by manual.

Can we do both? That is, add an option to have ftracetest not fail on
"unsupported" but have it fail by default. We can have kselftest just
pass in a parameter to ftracetest that has unsupported not fail?

But if that is too difficult, then I can live with modifying my test
case to add the option.

-- Steve

[toc] | [prev] | [next] | [standalone]

#1680623

On Mon, 3 Jul 2017 11:59:06 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> On Tue, 4 Jul 2017 00:52:32 +0900
> Masami Hiramatsu <mhiramat@kernel.org> wrote:
> > 
> > > 
> > > Can we add an option in kselftest, or to ftracetest that decides if
> > > unsupported is a failure or not? Otherwise I can not ack this patch.  
> > 
> > I would rather like to add an option to ftracetest instead of
> > kselftest, because whether the tested feature should be supported
> > or not is hard to decide from testing framework. It should be
> > checked by manual.
> 
> Can we do both? That is, add an option to have ftracetest not fail on
> "unsupported" but have it fail by default. We can have kselftest just
> pass in a parameter to ftracetest that has unsupported not fail?

I rather like to treat unsupported as success (or XFAIL) by default
and add "--fail-unsupported" option.

Thank you,

> But if that is too difficult, then I can live with modifying my test
> case to add the option.
> 
> -- Steve


-- 
Masami Hiramatsu <mhiramat@kernel.org>

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web