Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1419714 > unrolled thread

[PATCH 0/4] scripts/coccicheck: add paramap and indexing options

Back to article view | Back to linux.kernel

Contents

  [PATCH 0/4] scripts/coccicheck: add paramap and indexing options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-10 22:50 +0200
    [PATCH 2/4] coccicheck: enable paramap support "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-10 22:50 +0200
      Re: [Cocci] [PATCH 2/4] coccicheck: enable paramap support SF Markus Elfring <elfring@users.sourceforge.net> - 2016-06-11 08:00 +0200
        Re: [Cocci] [PATCH 2/4] coccicheck: enable paramap support Julia Lawall <julia.lawall@lip6.fr> - 2016-06-11 08:00 +0200
    [PATCH 3/4] scripts: add glimpse.sh for indexing the kernel "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-10 22:50 +0200
      Re: [Cocci] [PATCH 3/4] scripts: add glimpse.sh for indexing the  kernel SF Markus Elfring <elfring@users.sourceforge.net> - 2016-06-11 19:20 +0200
    [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-10 22:50 +0200
      Re: [PATCH 4/4] coccicheck: add indexing enhancement options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-10 23:10 +0200
        Re: [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-10 23:20 +0200
          Re: [PATCH 4/4] coccicheck: add indexing enhancement options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-10 23:30 +0200
            Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options Wolfram Sang <wsa@the-dreams.de> - 2016-06-10 23:50 +0200
              Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-10 23:50 +0200
                Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options Wolfram Sang <wsa@the-dreams.de> - 2016-06-11 00:00 +0200
                  Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-11 00:10 +0200
                    Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-11 00:30 +0200
                      Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options Wolfram Sang <wsa@the-dreams.de> - 2016-06-11 07:50 +0200
                        Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement  options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-11 08:00 +0200
                          Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options Wolfram Sang <wsa@the-dreams.de> - 2016-06-11 08:10 +0200
                          Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-13 20:40 +0200
                            Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options Wolfram Sang <wsa@the-dreams.de> - 2016-06-13 21:00 +0200
                              Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement  options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-13 21:50 +0200
                                Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-13 23:30 +0200
                                  Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement  options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-14 07:10 +0200
                    Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement  options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-11 07:20 +0200
                      Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options Wolfram Sang <wsa@the-dreams.de> - 2016-06-11 08:00 +0200
                        Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement  options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-11 08:10 +0200
                  Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement  options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-11 07:30 +0200
                    Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-13 20:50 +0200
                Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement  options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-11 07:20 +0200
            Re: [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-13 21:40 +0200
              Re: [PATCH 4/4] coccicheck: add indexing enhancement options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-13 22:00 +0200
                Re: [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-13 23:30 +0200
                  Re: [PATCH 4/4] coccicheck: add indexing enhancement options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-14 07:30 +0200
                    Re: [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-14 21:30 +0200
                      Re: [PATCH 4/4] coccicheck: add indexing enhancement options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-14 22:50 +0200
                        Re: [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-14 23:20 +0200
                          Re: [PATCH 4/4] coccicheck: add indexing enhancement options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-14 23:20 +0200
                            Re: [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-15 00:10 +0200
                              Re: [PATCH 4/4] coccicheck: add indexing enhancement options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-15 09:50 +0200
                                Re: [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-15 17:40 +0200
                                  Re: [PATCH 4/4] coccicheck: add indexing enhancement options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-15 17:50 +0200
                                    Re: [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-15 20:00 +0200
      Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options SF Markus Elfring <elfring@users.sourceforge.net> - 2016-06-11 08:00 +0200
    Re: [Cocci] [PATCH 0/4] scripts/coccicheck: add paramap and indexing  options SF Markus Elfring <elfring@users.sourceforge.net> - 2016-06-11 07:30 +0200

Page 1 of 3  [1] 2 3  Next page →

#1419714 — [PATCH 0/4] scripts/coccicheck: add paramap and indexing options

coccicheck hasn't been updated for a while. The backports
project has been using some features for a while now that
we should be able to also take advantage of with coccicheck,
the most important one is paramap support.

glimpseindex stuff wasn't even building but today I decided
to go tackle and fix that, the public open source release is
is now working and you can optionally use that. Note that
using git performs just as well, glimpseindex just shaves off
a bit of time, however if since we can support it now we do it.

Luis R. Rodriguez (4):
  coccicheck: move spatch binary check up
  coccicheck: enable paramap support
  scripts: add glimpse.sh for indexing the kernel
  coccicheck: add indexing enhancement options

 scripts/coccicheck | 62 +++++++++++++++++++++++++++++++++++++++++++++++-------
 scripts/glimpse.sh | 12 +++++++++++
 2 files changed, 66 insertions(+), 8 deletions(-)
 create mode 100755 scripts/glimpse.sh

-- 
2.8.2

[toc] | [next] | [standalone]

#1419715 — [PATCH 2/4] coccicheck: enable paramap support

Coccinelle has had paramap support since 1.0.2, this means
it supports --jobs, enabling built-in multithreaded functionality,
instead of needing one to script it out. Just look for --jobs
in the help output to determine if this is supported.

Also enable the load balancing to be dynamic, so that if a
thread finishes early we keep feeding it.

If --jobs is not supported we fallback to the old mechanism.

As a small example, prior to this change, on an 8-core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report
...

real    29m14.912s
user    103m1.796s
sys     0m4.464s

After:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 28 +++++++++++++++++++++++++---
 1 file changed, 25 insertions(+), 3 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index aa5e78fba270..eeb5fdc142ca 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -12,8 +12,8 @@ if [ ! -x "$SPATCH" ]; then
     exit 1
 fi
 
-trap kill_running SIGTERM SIGINT
-declare -a SPATCH_PID
+USE_JOBS="no"
+$SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
 # The verbosity may be set by the environmental parameter V=
 # as for example with 'make V=1 coccicheck'
@@ -82,7 +82,21 @@ if [ "$ONLINE" = "0" ] ; then
     echo ''
 fi
 
-run_cmd() {
+if [ "$USE_JOBS" = "no" ]; then
+	trap kill_running SIGTERM SIGINT
+	declare -a SPATCH_PID
+else
+	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
+fi
+
+run_cmd_paramap() {
+	if [ $VERBOSE -ne 0 ] ; then
+		echo "Running ($NPROC in parallel): $@"
+	fi
+	$@
+}
+
+run_cmd_old() {
 	local i
 	if [ $VERBOSE -ne 0 ] ; then
 		echo "Running ($NPROC in parallel): $@"
@@ -97,6 +111,14 @@ run_cmd() {
 	wait
 }
 
+run_cmd() {
+	if [ "$USE_JOBS" = "yes" ]; then
+		run_cmd_paramap $@
+	else
+		run_cmd_old $@
+	fi
+}
+
 kill_running() {
 	for i in $(seq 0 $(( NPROC - 1 )) ); do
 		if [ $VERBOSE -eq 2 ] ; then
-- 
2.8.2

[toc] | [prev] | [next] | [standalone]

#1419924 — Re: [Cocci] [PATCH 2/4] coccicheck: enable paramap support

> Also enable the load balancing to be dynamic, so that
> if a thread finishes early we keep feeding it.

Is this functionality influenced by the parameter "chunksize"?

Regards,
Markus

[toc] | [prev] | [next] | [standalone]

#1419925 — Re: [Cocci] [PATCH 2/4] coccicheck: enable paramap support

On Sat, 11 Jun 2016, SF Markus Elfring wrote:

> > Also enable the load balancing to be dynamic, so that
> > if a thread finishes early we keep feeding it.
> 
> Is this functionality influenced by the parameter "chunksize"?

Yes, without chunksize the distribution of work to processes is static.

julia

[toc] | [prev] | [next] | [standalone]

#1419716 — [PATCH 3/4] scripts: add glimpse.sh for indexing the kernel

Glimpse is a tool you can use to index the kernel. The tool
was recently open sourced under the ISC license and can be
obtained at:

https://github.com/mcgrof/glimpse.git

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/glimpse.sh | 12 ++++++++++++
 1 file changed, 12 insertions(+)
 create mode 100755 scripts/glimpse.sh

diff --git a/scripts/glimpse.sh b/scripts/glimpse.sh
new file mode 100755
index 000000000000..5fd24e49f31b
--- /dev/null
+++ b/scripts/glimpse.sh
@@ -0,0 +1,12 @@
+#!/bin/bash
+
+DIR=$(dirname $(readlink -f $0))
+DIR="${DIR}/../"
+
+GLIMPSEINDEX="`which ${GLIMPSEINDEX:=glimpseindex}`"
+if [ ! -x "$GLIMPSEINDEX" ]; then
+	echo 'glimpseindex can be obtained at https://github.com/mcgrof/glimpse.git'
+	exit 1
+fi
+
+find $DIR/* -name "*.[ch]" | $GLIMPSEINDEX -o -H . -F
-- 
2.8.2

[toc] | [prev] | [next] | [standalone]

#1420023 — Re: [Cocci] [PATCH 3/4] scripts: add glimpse.sh for indexing the kernel

> Glimpse is a tool you can use to index the kernel. The tool
> was recently open sourced under the ISC license and can be
> obtained at:

How do you think about to mention the script addition also directly in
the commit message?


> @@ -0,0 +1,12 @@
> +#!/bin/bash
> +
> +DIR=$(dirname $(readlink -f $0))
> +DIR="${DIR}/../"

Would you like to use the following variable assignment (instead of two
before)?

+DIR="$(dirname $(readlink -f $0))/../"


By the way: How are the chances to achieve further software improvements?
https://github.com/gvelez17/glimpse/issues

Regards,
Markus

[toc] | [prev] | [next] | [standalone]

#1419719 — [PATCH 4/4] coccicheck: add indexing enhancement options

Enable indexing optimizations heuristics. Coccinelle has
support to make use of its own enhanced "grep" mechanisms
instead of using regular grep for searching code 'coccigrep',
in practice though this seems to not perform better than
regular grep however its expected to help with some use cases
so we use that if you have no other indexing options in place
available.

Since git has its own index, support for using 'git grep' has been
added to Coccinelle, that should on average perform better than
using the internal cocci grep, and regular grep. Lastly, Coccinelle
has had support for glimpseindex for a long while, however the
tool was previously closed source, its now open sourced, and
provides the best performance, so support that if we can detect
you have a glimpse index.

These tests have been run on an 8 core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report

Before this patch with no indexing or anything:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

Using coccigrep (after this patch if you have no .git):

real    16m27.650s
user    128m47.904s
sys     0m2.176s

If you have .git and therefore use gitgrep:

real    16m21.220s
user    129m30.940s
sys     0m2.060s

And if you have a .glimpse_index:

real    16m14.794s
user    128m42.356s
sys     0m1.880s

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index eeb5fdc142ca..f31c9a152559 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -5,6 +5,8 @@
 # version 1.0.0-rc11.
 #
 
+DIR=$(dirname $(readlink -f $0))
+DIR="${DIR}/../"
 SPATCH="`which ${SPATCH:=spatch}`"
 
 if [ ! -x "$SPATCH" ]; then
@@ -15,6 +17,20 @@ fi
 USE_JOBS="no"
 $SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
+# 0. --use-glimpse currently outperforms all. Refer
+#    to scripts/glimpse.sh for details.
+# 1. Second best is --use-gitgrep, this is very comparable to --use-glimpse
+# 2. Use --use-coccigrep if no indexing options are available and your
+#    version of coccinelle supports it
+USE_GLIMPSE="no"
+$SPATCH --help | grep "\-\-use\-glimpse" > /dev/null && [ -f $DIR/.glimpse_index ] && USE_GLIMPSE="yes"
+
+USE_GITGREP="no"
+$SPATCH --help | grep "\-\-use\-gitgrep" > /dev/null && [ -d $DIR/.git ] && USE_GITGREP="yes"
+
+USE_COCCIGREP="no"
+$SPATCH --help | grep "\-\-use\-coccigrep" > /dev/null && USE_COCCIGREP="yes"
+
 # The verbosity may be set by the environmental parameter V=
 # as for example with 'make V=1 coccicheck'
 
@@ -89,6 +105,14 @@ else
 	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
 fi
 
+if [ "$USE_GLIMPSE" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-glimpse"
+elif [ "$USE_GITGREP" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-gitgrep"
+elif [ "$USE_COCCIGREP" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-coccigrep"
+fi
+
 run_cmd_paramap() {
 	if [ $VERBOSE -ne 0 ] ; then
 		echo "Running ($NPROC in parallel): $@"
-- 
2.8.2

[toc] | [prev] | [next] | [standalone]

#1419726 — Re: [PATCH 4/4] coccicheck: add indexing enhancement options

On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:

> Enable indexing optimizations heuristics. Coccinelle has
> support to make use of its own enhanced "grep" mechanisms
> instead of using regular grep for searching code 'coccigrep',
> in practice though this seems to not perform better than
> regular grep however its expected to help with some use cases
> so we use that if you have no other indexing options in place
> available.
> 
> Since git has its own index, support for using 'git grep' has been
> added to Coccinelle, that should on average perform better than
> using the internal cocci grep, and regular grep. Lastly, Coccinelle
> has had support for glimpseindex for a long while, however the
> tool was previously closed source, its now open sourced, and
> provides the best performance, so support that if we can detect
> you have a glimpse index.
> 
> These tests have been run on an 8 core system:
> 
> Before:
> 
> $ export COCCI=scripts/coccinelle/free/kfree.cocci
> $ time make coccicheck MODE=report
> 
> Before this patch with no indexing or anything:
> 
> real    16m22.435s
> user    128m30.060s
> sys     0m2.712s
> 
> Using coccigrep (after this patch if you have no .git):
> 
> real    16m27.650s
> user    128m47.904s
> sys     0m2.176s
> 
> If you have .git and therefore use gitgrep:
> 
> real    16m21.220s
> user    129m30.940s
> sys     0m2.060s
> 
> And if you have a .glimpse_index:
> 
> real    16m14.794s
> user    128m42.356s
> sys     0m1.880s

I don't see any convincing differences in these times.

I believe that Coccinelle's internal grep is always used, even with no 
option.

I'm puzzled why glimpse gives no benefit.

julia


> 
> Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> ---
>  scripts/coccicheck | 24 ++++++++++++++++++++++++
>  1 file changed, 24 insertions(+)
> 
> diff --git a/scripts/coccicheck b/scripts/coccicheck
> index eeb5fdc142ca..f31c9a152559 100755
> --- a/scripts/coccicheck
> +++ b/scripts/coccicheck
> @@ -5,6 +5,8 @@
>  # version 1.0.0-rc11.
>  #
>  
> +DIR=$(dirname $(readlink -f $0))
> +DIR="${DIR}/../"
>  SPATCH="`which ${SPATCH:=spatch}`"
>  
>  if [ ! -x "$SPATCH" ]; then
> @@ -15,6 +17,20 @@ fi
>  USE_JOBS="no"
>  $SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
>  
> +# 0. --use-glimpse currently outperforms all. Refer
> +#    to scripts/glimpse.sh for details.
> +# 1. Second best is --use-gitgrep, this is very comparable to --use-glimpse
> +# 2. Use --use-coccigrep if no indexing options are available and your
> +#    version of coccinelle supports it
> +USE_GLIMPSE="no"
> +$SPATCH --help | grep "\-\-use\-glimpse" > /dev/null && [ -f $DIR/.glimpse_index ] && USE_GLIMPSE="yes"
> +
> +USE_GITGREP="no"
> +$SPATCH --help | grep "\-\-use\-gitgrep" > /dev/null && [ -d $DIR/.git ] && USE_GITGREP="yes"
> +
> +USE_COCCIGREP="no"
> +$SPATCH --help | grep "\-\-use\-coccigrep" > /dev/null && USE_COCCIGREP="yes"
> +
>  # The verbosity may be set by the environmental parameter V=
>  # as for example with 'make V=1 coccicheck'
>  
> @@ -89,6 +105,14 @@ else
>  	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
>  fi
>  
> +if [ "$USE_GLIMPSE" = "yes" ]; then
> +	OPTIONS="$OPTIONS --use-glimpse"
> +elif [ "$USE_GITGREP" = "yes" ]; then
> +	OPTIONS="$OPTIONS --use-gitgrep"
> +elif [ "$USE_COCCIGREP" = "yes" ]; then
> +	OPTIONS="$OPTIONS --use-coccigrep"
> +fi
> +
>  run_cmd_paramap() {
>  	if [ $VERBOSE -ne 0 ] ; then
>  		echo "Running ($NPROC in parallel): $@"
> -- 
> 2.8.2
> 
> 

[toc] | [prev] | [next] | [standalone]

#1419733 — Re: [PATCH 4/4] coccicheck: add indexing enhancement options

On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote:
> 
> 
> On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> 
> > Enable indexing optimizations heuristics. Coccinelle has
> > support to make use of its own enhanced "grep" mechanisms
> > instead of using regular grep for searching code 'coccigrep',
> > in practice though this seems to not perform better than
> > regular grep however its expected to help with some use cases
> > so we use that if you have no other indexing options in place
> > available.
> > 
> > Since git has its own index, support for using 'git grep' has been
> > added to Coccinelle, that should on average perform better than
> > using the internal cocci grep, and regular grep. Lastly, Coccinelle
> > has had support for glimpseindex for a long while, however the
> > tool was previously closed source, its now open sourced, and
> > provides the best performance, so support that if we can detect
> > you have a glimpse index.
> > 
> > These tests have been run on an 8 core system:
> > 
> > Before:
> > 
> > $ export COCCI=scripts/coccinelle/free/kfree.cocci
> > $ time make coccicheck MODE=report
> > 
> > Before this patch with no indexing or anything:
> > 
> > real    16m22.435s
> > user    128m30.060s
> > sys     0m2.712s
> > 
> > Using coccigrep (after this patch if you have no .git):
> > 
> > real    16m27.650s
> > user    128m47.904s
> > sys     0m2.176s
> > 
> > If you have .git and therefore use gitgrep:
> > 
> > real    16m21.220s
> > user    129m30.940s
> > sys     0m2.060s
> > 
> > And if you have a .glimpse_index:
> > 
> > real    16m14.794s
> > user    128m42.356s
> > sys     0m1.880s
> 
> I don't see any convincing differences in these times.
> 
> I believe that Coccinelle's internal grep is always used, even with no 
> option.

Ah that would explain it. This uses coccinelle 1.0.5, is the default
there to use --use-coccigrep if no other index is specified ?

> I'm puzzled why glimpse gives no benefit.

Well, slightly better.

  Luis

[toc] | [prev] | [next] | [standalone]

#1419738 — Re: [PATCH 4/4] coccicheck: add indexing enhancement options

On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:

> On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote:
> > 
> > 
> > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > 
> > > Enable indexing optimizations heuristics. Coccinelle has
> > > support to make use of its own enhanced "grep" mechanisms
> > > instead of using regular grep for searching code 'coccigrep',
> > > in practice though this seems to not perform better than
> > > regular grep however its expected to help with some use cases
> > > so we use that if you have no other indexing options in place
> > > available.
> > > 
> > > Since git has its own index, support for using 'git grep' has been
> > > added to Coccinelle, that should on average perform better than
> > > using the internal cocci grep, and regular grep. Lastly, Coccinelle
> > > has had support for glimpseindex for a long while, however the
> > > tool was previously closed source, its now open sourced, and
> > > provides the best performance, so support that if we can detect
> > > you have a glimpse index.
> > > 
> > > These tests have been run on an 8 core system:
> > > 
> > > Before:
> > > 
> > > $ export COCCI=scripts/coccinelle/free/kfree.cocci
> > > $ time make coccicheck MODE=report
> > > 
> > > Before this patch with no indexing or anything:
> > > 
> > > real    16m22.435s
> > > user    128m30.060s
> > > sys     0m2.712s
> > > 
> > > Using coccigrep (after this patch if you have no .git):
> > > 
> > > real    16m27.650s
> > > user    128m47.904s
> > > sys     0m2.176s
> > > 
> > > If you have .git and therefore use gitgrep:
> > > 
> > > real    16m21.220s
> > > user    129m30.940s
> > > sys     0m2.060s
> > > 
> > > And if you have a .glimpse_index:
> > > 
> > > real    16m14.794s
> > > user    128m42.356s
> > > sys     0m1.880s
> > 
> > I don't see any convincing differences in these times.
> > 
> > I believe that Coccinelle's internal grep is always used, even with no 
> > option.
> 
> Ah that would explain it. This uses coccinelle 1.0.5, is the default
> there to use --use-coccigrep if no other index is specified ?

It has been the default for a long time.

> > I'm puzzled why glimpse gives no benefit.
> 
> Well, slightly better.

No, it should be much better.  You would have to look at the standard 
error to see if you are getting any benefit.  There should be very few 
occurrences of Skipping if you are really using glimpse.  In any case, if 
you asked for glimpse and it was not able to provide it, there should be 
warning messages at the top of stderr.

julia

[toc] | [prev] | [next] | [standalone]

#1419750 — Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Multipart message — attachments visible in raw view] — view raw

> > Well, slightly better.
> 
> No, it should be much better.  You would have to look at the standard 

I use id-utils regularly and it is indeed at least a magnitude better.
The indexing often pays off already with the first coccinelle run for
me. Highly recommended.

[toc] | [prev] | [next] | [standalone]

#1419751 — Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

On Fri, Jun 10, 2016 at 11:43:57PM +0200, Wolfram Sang wrote:
> > > Well, slightly better.
> > 
> > No, it should be much better.  You would have to look at the standard 
> 
> I use id-utils regularly and it is indeed at least a magnitude better.
> The indexing often pays off already with the first coccinelle run for
> me. Highly recommended.

AFAICT coccinelle does not have integration support for id-utils though.

  Luis

[toc] | [prev] | [next] | [standalone]

#1419755 — Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Multipart message — attachments visible in raw view] — view raw

> AFAICT coccinelle does not have integration support for id-utils though.

I used it just today ;) -- "--use-idutils ./ID"

ID was generated with simple 'mkid -s'.

[toc] | [prev] | [next] | [standalone]

#1419756 — Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

On Fri, Jun 10, 2016 at 11:51:26PM +0200, Wolfram Sang wrote:
> > AFAICT coccinelle does not have integration support for id-utils though.
> 
> I used it just today ;) -- "--use-idutils ./ID"
> 
> ID was generated with simple 'mkid -s'.
> 

Sweet, testing that now.

  Luis

[toc] | [prev] | [next] | [standalone]

#1419816 — Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

On Sat, Jun 11, 2016 at 12:08:32AM +0200, Luis R. Rodriguez wrote:
> On Fri, Jun 10, 2016 at 11:51:26PM +0200, Wolfram Sang wrote:
> > > AFAICT coccinelle does not have integration support for id-utils though.
> > 
> > I used it just today ;) -- "--use-idutils ./ID"
> > 
> > ID was generated with simple 'mkid -s'.
> > 
> 
> Sweet, testing that now.

Boooyah :D

real    16m11.692s
user    127m50.388s
sys     0m2.168s

  Luis

[toc] | [prev] | [next] | [standalone]

#1419916 — Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Multipart message — attachments visible in raw view] — view raw

> real    16m11.692s
> user    127m50.388s
> sys     0m2.168s

That's better but not a magnitude, I wonder.

[toc] | [prev] | [next] | [standalone]

#1419917 — Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

On Sat, 11 Jun 2016, Wolfram Sang wrote:

> 
> > real    16m11.692s
> > user    127m50.388s
> > sys     0m2.168s
> 
> That's better but not a magnitude, I wonder.

I think that it is because the filtering that Coccinelle does already 
works pretty well, and there are quite a lot of files (7514) that contain 
kfree.

julia

[toc] | [prev] | [next] | [standalone]

#1419927 — Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Multipart message — attachments visible in raw view] — view raw

> works pretty well, and there are quite a lot of files (7514) that contain 
> kfree.

Ah, kfree. That explains, I missed that info.

[toc] | [prev] | [next] | [standalone]

#1421098 — Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

On Sat, Jun 11, 2016 at 07:54:39AM +0200, Julia Lawall wrote:
> 
> 
> On Sat, 11 Jun 2016, Wolfram Sang wrote:
> 
> > 
> > > real    16m11.692s
> > > user    127m50.388s
> > > sys     0m2.168s
> > 
> > That's better but not a magnitude, I wonder.
> 
> I think that it is because the filtering that Coccinelle does already 
> works pretty well, and there are quite a lot of files (7514) that contain 
> kfree.

Is there another scripts/coccinelle/ file I can use to test against to demo
against glimpse/idutils/gitgrep best?

  Luis

[toc] | [prev] | [next] | [standalone]

#1421153 — Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Multipart message — attachments visible in raw view] — view raw

> Is there another scripts/coccinelle/ file I can use to test against to demo
> against glimpse/idutils/gitgrep best?

I'd think this one may be a candidate:

scripts/coccinelle/misc/irqf_oneshot.cocci

Not too many, but quite some matches over the tree.

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | linux.kernel

csiph-web