Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1424512

[PATCH v2 2/8] coccicheck: enable parmap support

Show all headers | View raw

Coccinelle has had parmap support since 1.0.2, this means
it supports --jobs, enabling built-in multithreaded functionality,
instead of needing one to script it out. Just look for --jobs
in the help output to determine if this is supported.

Also enable the load balancing to be dynamic, so that if a
thread finishes early we keep feeding it.

Note: now that we have all things handled for us, redirect stderr to
stdout as well to capture any possible errors or warnings issued by
coccinelle.

If --jobs is not supported we fallback to the old mechanism.
This also now accepts DEBUG_FILE= to specify where you want
stderr to be redirected to, by default we redirect stderr to
/dev/null.

Also since while at it propagate back into the shell script any
coccinelle error code. When used in serialized mode where all cocci
files are run this also stops processing if an error has occured.
This lets us handle some errors in coccinelle cocci files and if
they bail out we should inspect the errors. This will be more
useful later to help annotate coccinelle version dependency
requirements. This will let you run only SmPL files that your
system supports.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>

As a small example, prior to this change, on an 8-core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report DEBUG_FILE=cocci.err
...

real    29m14.912s
user    103m1.796s
sys     0m4.464s

After:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

v2:

o redirect coccinelle stderr to /dev/null by default and
  only if DEBUG_FILE is used do we pass it to a file
o fix typo of paramap/parmap

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 40 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 37 insertions(+), 3 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index aa5e78fba270..7acef3efc258 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -12,8 +12,8 @@ if [ ! -x "$SPATCH" ]; then
     exit 1
 fi
 
-trap kill_running SIGTERM SIGINT
-declare -a SPATCH_PID
+USE_JOBS="no"
+$SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
 # The verbosity may be set by the environmental parameter V=
 # as for example with 'make V=1 coccicheck'
@@ -82,7 +82,33 @@ if [ "$ONLINE" = "0" ] ; then
     echo ''
 fi
 
-run_cmd() {
+if [ "$USE_JOBS" = "no" ]; then
+	trap kill_running SIGTERM SIGINT
+	declare -a SPATCH_PID
+else
+	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
+fi
+
+run_cmd_parmap() {
+	if [ $VERBOSE -ne 0 ] ; then
+		echo "Running ($NPROC in parallel): $@"
+	fi
+	if [ "$DEBUG_FILE" != "/dev/null" -a "$DEBUG_FILE" != "" ]; then
+		if [ -f $DEBUG_FILE ]; then
+			echo "Debug file $DEBUG_FILE exists, bailing"
+			exit
+		fi
+	else
+		DEBUG_FILE="/dev/null"
+	fi
+	$@ 2>$DEBUG_FILE
+	if [[ $? -ne 0 ]]; then
+		echo "coccicheck failed"
+		exit $?
+	fi
+}
+
+run_cmd_old() {
 	local i
 	if [ $VERBOSE -ne 0 ] ; then
 		echo "Running ($NPROC in parallel): $@"
@@ -97,6 +123,14 @@ run_cmd() {
 	wait
 }
 
+run_cmd() {
+	if [ "$USE_JOBS" = "yes" ]; then
+		run_cmd_parmap $@
+	else
+		run_cmd_old $@
+	fi
+}
+
 kill_running() {
 	for i in $(seq 0 $(( NPROC - 1 )) ); do
 		if [ $VERBOSE -eq 2 ] ; then
-- 
2.8.2

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[PATCH v2 0/8] coccicheck: modernize "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-17 00:40 +0200
  [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-17 00:40 +0200
    Re: [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel Julia Lawall <julia.lawall@lip6.fr> - 2016-06-17 11:50 +0200
      Re: [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-17 17:20 +0200
        Re: [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel Julia Lawall <julia.lawall@lip6.fr> - 2016-06-17 17:40 +0200
          Re: [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-18 03:30 +0200
            Re: [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel Julia Lawall <julia.lawall@lip6.fr> - 2016-06-18 08:00 +0200
              Re: [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-20 21:20 +0200
            Re: [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel Julia Lawall <julia.lawall@lip6.fr> - 2016-06-18 12:40 +0200
            Re: [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel Julia Lawall <julia.lawall@lip6.fr> - 2016-06-18 12:40 +0200
      Re: [Cocci] [PATCH v2 4/8] scripts: add glimpse.sh for indexing the  kernel SF Markus Elfring <elfring@users.sourceforge.net> - 2016-06-17 17:30 +0200
        Re: [Cocci] [PATCH v2 4/8] scripts: add glimpse.sh for indexing the  kernel Julia Lawall <julia.lawall@lip6.fr> - 2016-06-17 18:20 +0200
  [PATCH v2 2/8] coccicheck: enable parmap support "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-17 00:40 +0200
  [PATCH v2 7/8] coccicheck: refer to coccicheck bottest wiki for documentation "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-17 00:40 +0200
  [PATCH v2 5/8] coccicheck: replace --very-quiet with --quit when debugging "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-17 00:40 +0200
  [PATCH v2 8/8] scripts/coccinelle: require coccinelle >= 1.0.4 on device_node_continue.cocci "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-17 00:40 +0200

csiph-web