Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1421248 > unrolled thread

Re: [PATCH 4/4] coccicheck: add indexing enhancement options

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-13 21:40 +0200
    Re: [PATCH 4/4] coccicheck: add indexing enhancement options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-13 22:00 +0200
      Re: [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-13 23:30 +0200
        Re: [PATCH 4/4] coccicheck: add indexing enhancement options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-14 07:30 +0200
          Re: [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-14 21:30 +0200
            Re: [PATCH 4/4] coccicheck: add indexing enhancement options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-14 22:50 +0200
              Re: [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-14 23:20 +0200
                Re: [PATCH 4/4] coccicheck: add indexing enhancement options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-14 23:20 +0200
                  Re: [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-15 00:10 +0200
                    Re: [PATCH 4/4] coccicheck: add indexing enhancement options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-15 09:50 +0200
                      Re: [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-15 17:40 +0200
                        Re: [PATCH 4/4] coccicheck: add indexing enhancement options Julia Lawall <julia.lawall@lip6.fr> - 2016-06-15 17:50 +0200
                          Re: [PATCH 4/4] coccicheck: add indexing enhancement options "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-06-15 20:00 +0200

#1421248 — Re: [PATCH 4/4] coccicheck: add indexing enhancement options

On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote:
> 
> 
> On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote:
> > > 
> > > 
> > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > > 
> > > > Enable indexing optimizations heuristics. Coccinelle has
> > > > support to make use of its own enhanced "grep" mechanisms
> > > > instead of using regular grep for searching code 'coccigrep',
> > > > in practice though this seems to not perform better than
> > > > regular grep however its expected to help with some use cases
> > > > so we use that if you have no other indexing options in place
> > > > available.
> > > > 
> > > > Since git has its own index, support for using 'git grep' has been
> > > > added to Coccinelle, that should on average perform better than
> > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle
> > > > has had support for glimpseindex for a long while, however the
> > > > tool was previously closed source, its now open sourced, and
> > > > provides the best performance, so support that if we can detect
> > > > you have a glimpse index.
> > > > 
> > > > These tests have been run on an 8 core system:
> > > > 
> > > > Before:
> > > > 
> > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci
> > > > $ time make coccicheck MODE=report
> > > > 
> > > > Before this patch with no indexing or anything:
> > > > 
> > > > real    16m22.435s
> > > > user    128m30.060s
> > > > sys     0m2.712s
> > > > 
> > > > Using coccigrep (after this patch if you have no .git):
> > > > 
> > > > real    16m27.650s
> > > > user    128m47.904s
> > > > sys     0m2.176s
> > > > 
> > > > If you have .git and therefore use gitgrep:
> > > > 
> > > > real    16m21.220s
> > > > user    129m30.940s
> > > > sys     0m2.060s
> > > > 
> > > > And if you have a .glimpse_index:
> > > > 
> > > > real    16m14.794s
> > > > user    128m42.356s
> > > > sys     0m1.880s
> > > 
> > > I don't see any convincing differences in these times.
> > > 
> > > I believe that Coccinelle's internal grep is always used, even with no 
> > > option.
> > 
> > Ah that would explain it. This uses coccinelle 1.0.5, is the default
> > there to use --use-coccigrep if no other index is specified ?
> 
> It has been the default for a long time.
> 
> > > I'm puzzled why glimpse gives no benefit.
> > 
> > Well, slightly better.
> 
> No, it should be much better.  You would have to look at the standard 
> error to see if you are getting any benefit.  There should be very few 
> occurrences of Skipping if you are really using glimpse.  In any case, if 
> you asked for glimpse and it was not able to provide it, there should be 
> warning messages at the top of stderr.

I'll redirect stderr to stdout by default when parmap support is used then.

  Luis

[toc] | [next] | [standalone]

#1421272

On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:

> On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote:
> > 
> > 
> > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > 
> > > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote:
> > > > 
> > > > 
> > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > > > 
> > > > > Enable indexing optimizations heuristics. Coccinelle has
> > > > > support to make use of its own enhanced "grep" mechanisms
> > > > > instead of using regular grep for searching code 'coccigrep',
> > > > > in practice though this seems to not perform better than
> > > > > regular grep however its expected to help with some use cases
> > > > > so we use that if you have no other indexing options in place
> > > > > available.
> > > > > 
> > > > > Since git has its own index, support for using 'git grep' has been
> > > > > added to Coccinelle, that should on average perform better than
> > > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle
> > > > > has had support for glimpseindex for a long while, however the
> > > > > tool was previously closed source, its now open sourced, and
> > > > > provides the best performance, so support that if we can detect
> > > > > you have a glimpse index.
> > > > > 
> > > > > These tests have been run on an 8 core system:
> > > > > 
> > > > > Before:
> > > > > 
> > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci
> > > > > $ time make coccicheck MODE=report
> > > > > 
> > > > > Before this patch with no indexing or anything:
> > > > > 
> > > > > real    16m22.435s
> > > > > user    128m30.060s
> > > > > sys     0m2.712s
> > > > > 
> > > > > Using coccigrep (after this patch if you have no .git):
> > > > > 
> > > > > real    16m27.650s
> > > > > user    128m47.904s
> > > > > sys     0m2.176s
> > > > > 
> > > > > If you have .git and therefore use gitgrep:
> > > > > 
> > > > > real    16m21.220s
> > > > > user    129m30.940s
> > > > > sys     0m2.060s
> > > > > 
> > > > > And if you have a .glimpse_index:
> > > > > 
> > > > > real    16m14.794s
> > > > > user    128m42.356s
> > > > > sys     0m1.880s
> > > > 
> > > > I don't see any convincing differences in these times.
> > > > 
> > > > I believe that Coccinelle's internal grep is always used, even with no 
> > > > option.
> > > 
> > > Ah that would explain it. This uses coccinelle 1.0.5, is the default
> > > there to use --use-coccigrep if no other index is specified ?
> > 
> > It has been the default for a long time.
> > 
> > > > I'm puzzled why glimpse gives no benefit.
> > > 
> > > Well, slightly better.
> > 
> > No, it should be much better.  You would have to look at the standard 
> > error to see if you are getting any benefit.  There should be very few 
> > occurrences of Skipping if you are really using glimpse.  In any case, if 
> > you asked for glimpse and it was not able to provide it, there should be 
> > warning messages at the top of stderr.
> 
> I'll redirect stderr to stdout by default when parmap support is used then.

Usually I put them in different files.

julia

[toc] | [prev] | [next] | [standalone]

#1421352

On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> 
> 
> On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote:
> > > 
> > > 
> > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > > 
> > > > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote:
> > > > > 
> > > > > 
> > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > > > > 
> > > > > > Enable indexing optimizations heuristics. Coccinelle has
> > > > > > support to make use of its own enhanced "grep" mechanisms
> > > > > > instead of using regular grep for searching code 'coccigrep',
> > > > > > in practice though this seems to not perform better than
> > > > > > regular grep however its expected to help with some use cases
> > > > > > so we use that if you have no other indexing options in place
> > > > > > available.
> > > > > > 
> > > > > > Since git has its own index, support for using 'git grep' has been
> > > > > > added to Coccinelle, that should on average perform better than
> > > > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle
> > > > > > has had support for glimpseindex for a long while, however the
> > > > > > tool was previously closed source, its now open sourced, and
> > > > > > provides the best performance, so support that if we can detect
> > > > > > you have a glimpse index.
> > > > > > 
> > > > > > These tests have been run on an 8 core system:
> > > > > > 
> > > > > > Before:
> > > > > > 
> > > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci
> > > > > > $ time make coccicheck MODE=report
> > > > > > 
> > > > > > Before this patch with no indexing or anything:
> > > > > > 
> > > > > > real    16m22.435s
> > > > > > user    128m30.060s
> > > > > > sys     0m2.712s
> > > > > > 
> > > > > > Using coccigrep (after this patch if you have no .git):
> > > > > > 
> > > > > > real    16m27.650s
> > > > > > user    128m47.904s
> > > > > > sys     0m2.176s
> > > > > > 
> > > > > > If you have .git and therefore use gitgrep:
> > > > > > 
> > > > > > real    16m21.220s
> > > > > > user    129m30.940s
> > > > > > sys     0m2.060s
> > > > > > 
> > > > > > And if you have a .glimpse_index:
> > > > > > 
> > > > > > real    16m14.794s
> > > > > > user    128m42.356s
> > > > > > sys     0m1.880s
> > > > > 
> > > > > I don't see any convincing differences in these times.
> > > > > 
> > > > > I believe that Coccinelle's internal grep is always used, even with no 
> > > > > option.
> > > > 
> > > > Ah that would explain it. This uses coccinelle 1.0.5, is the default
> > > > there to use --use-coccigrep if no other index is specified ?
> > > 
> > > It has been the default for a long time.
> > > 
> > > > > I'm puzzled why glimpse gives no benefit.
> > > > 
> > > > Well, slightly better.
> > > 
> > > No, it should be much better.  You would have to look at the standard 
> > > error to see if you are getting any benefit.  There should be very few 
> > > occurrences of Skipping if you are really using glimpse.  In any case, if 
> > > you asked for glimpse and it was not able to provide it, there should be 
> > > warning messages at the top of stderr.
> > 
> > I'll redirect stderr to stdout by default when parmap support is used then.
> 
> Usually I put them in different files.

We can do that as well but I would only want to deal with parmap support case.
Any preference? How about .coccicheck.stderr.$PID where PID would be the PID of
the shell script?

  Luis

[toc] | [prev] | [next] | [standalone]

#1421582

On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:

> On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > 
> > 
> > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > 
> > > On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote:
> > > > 
> > > > 
> > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > > > 
> > > > > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote:
> > > > > > 
> > > > > > 
> > > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > 
> > > > > > > Enable indexing optimizations heuristics. Coccinelle has
> > > > > > > support to make use of its own enhanced "grep" mechanisms
> > > > > > > instead of using regular grep for searching code 'coccigrep',
> > > > > > > in practice though this seems to not perform better than
> > > > > > > regular grep however its expected to help with some use cases
> > > > > > > so we use that if you have no other indexing options in place
> > > > > > > available.
> > > > > > > 
> > > > > > > Since git has its own index, support for using 'git grep' has been
> > > > > > > added to Coccinelle, that should on average perform better than
> > > > > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle
> > > > > > > has had support for glimpseindex for a long while, however the
> > > > > > > tool was previously closed source, its now open sourced, and
> > > > > > > provides the best performance, so support that if we can detect
> > > > > > > you have a glimpse index.
> > > > > > > 
> > > > > > > These tests have been run on an 8 core system:
> > > > > > > 
> > > > > > > Before:
> > > > > > > 
> > > > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci
> > > > > > > $ time make coccicheck MODE=report
> > > > > > > 
> > > > > > > Before this patch with no indexing or anything:
> > > > > > > 
> > > > > > > real    16m22.435s
> > > > > > > user    128m30.060s
> > > > > > > sys     0m2.712s
> > > > > > > 
> > > > > > > Using coccigrep (after this patch if you have no .git):
> > > > > > > 
> > > > > > > real    16m27.650s
> > > > > > > user    128m47.904s
> > > > > > > sys     0m2.176s
> > > > > > > 
> > > > > > > If you have .git and therefore use gitgrep:
> > > > > > > 
> > > > > > > real    16m21.220s
> > > > > > > user    129m30.940s
> > > > > > > sys     0m2.060s
> > > > > > > 
> > > > > > > And if you have a .glimpse_index:
> > > > > > > 
> > > > > > > real    16m14.794s
> > > > > > > user    128m42.356s
> > > > > > > sys     0m1.880s
> > > > > > 
> > > > > > I don't see any convincing differences in these times.
> > > > > > 
> > > > > > I believe that Coccinelle's internal grep is always used, even with no 
> > > > > > option.
> > > > > 
> > > > > Ah that would explain it. This uses coccinelle 1.0.5, is the default
> > > > > there to use --use-coccigrep if no other index is specified ?
> > > > 
> > > > It has been the default for a long time.
> > > > 
> > > > > > I'm puzzled why glimpse gives no benefit.
> > > > > 
> > > > > Well, slightly better.
> > > > 
> > > > No, it should be much better.  You would have to look at the standard 
> > > > error to see if you are getting any benefit.  There should be very few 
> > > > occurrences of Skipping if you are really using glimpse.  In any case, if 
> > > > you asked for glimpse and it was not able to provide it, there should be 
> > > > warning messages at the top of stderr.
> > > 
> > > I'll redirect stderr to stdout by default when parmap support is used then.
> > 
> > Usually I put them in different files.
> 
> We can do that as well but I would only want to deal with parmap support 
> case. Any preference? How about .coccicheck.stderr.$PID where PID would 
> be the PID of the shell script?

I don't understand the connection with parmap.

Originally our use of parmap made output files based on pids.  Maybe this 
is the default for parmap.  I found this completely unusable.  I guess one 
could look at the dates to see which file is the most recent one, but it 
seems tedious.  If you are putting the standard output in x.out, then put 
the standard error in x.err.

julia

[toc] | [prev] | [next] | [standalone]

#1422247

On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> 
> 
> On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > 
> > > 
> > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > 
> > > > I'll redirect stderr to stdout by default when parmap support is used then.
> > > 
> > > Usually I put them in different files.
> > 
> > We can do that as well but I would only want to deal with parmap support 
> > case. Any preference? How about .coccicheck.stderr.$PID where PID would 
> > be the PID of the shell script?
> 
> I don't understand the connection with parmap.

When parmap support is not available the cocciscript will currently
disregard stderr, output is provided as it comes to stdout from each
thread I guess.

> Originally our use of parmap made output files based on pids.  Maybe this 
> is the default for parmap.  I found this completely unusable.  I guess one 
> could look at the dates to see which file is the most recent one, but it 
> seems tedious.  If you are putting the standard output in x.out, then put 
> the standard error in x.err.

I'll use ${DIR}/coccicheck.$$.err for stderr.

  Luis

[toc] | [prev] | [next] | [standalone]

#1422319

On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:

> On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > 
> > 
> > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > 
> > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > 
> > > > 
> > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > 
> > > > > I'll redirect stderr to stdout by default when parmap support is used then.
> > > > 
> > > > Usually I put them in different files.
> > > 
> > > We can do that as well but I would only want to deal with parmap support 
> > > case. Any preference? How about .coccicheck.stderr.$PID where PID would 
> > > be the PID of the shell script?
> > 
> > I don't understand the connection with parmap.
> 
> When parmap support is not available the cocciscript will currently
> disregard stderr, output is provided as it comes to stdout from each
> thread I guess.

Deepa's recent patch to coccicheck made apparent that Coccicheck uses 
--very-quiet, so there is standard error.

> > Originally our use of parmap made output files based on pids.  Maybe this 
> > is the default for parmap.  I found this completely unusable.  I guess one 
> > could look at the dates to see which file is the most recent one, but it 
> > seems tedious.  If you are putting the standard output in x.out, then put 
> > the standard error in x.err.
> 
> I'll use ${DIR}/coccicheck.$$.err for stderr.

What is ${DIR}? and what is $$?

julia

[toc] | [prev] | [next] | [standalone]

#1422344

On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote:
> 
> 
> On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > > 
> > > 
> > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > 
> > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > > 
> > > > > 
> > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > 
> > > > > > I'll redirect stderr to stdout by default when parmap support is used then.
> > > > > 
> > > > > Usually I put them in different files.
> > > > 
> > > > We can do that as well but I would only want to deal with parmap support 
> > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would 
> > > > be the PID of the shell script?
> > > 
> > > I don't understand the connection with parmap.
> > 
> > When parmap support is not available the cocciscript will currently
> > disregard stderr, output is provided as it comes to stdout from each
> > thread I guess.
> 
> Deepa's recent patch to coccicheck made apparent that Coccicheck uses 
> --very-quiet, so there is standard error.

OK I'm disegarding the redirect for non-parmap for now but we'd have to
determine if we want to append or add one per PID... I rather leave that
stuff as-is and encourage folks to upgrade coccinelle.

> > > Originally our use of parmap made output files based on pids.  Maybe this 
> > > is the default for parmap.  I found this completely unusable.  I guess one 
> > > could look at the dates to see which file is the most recent one, but it 
> > > seems tedious.  If you are putting the standard output in x.out, then put 
> > > the standard error in x.err.
> > 
> > I'll use ${DIR}/coccicheck.$$.err for stderr.
> 
> What is ${DIR}? and what is $$?

When you run scripts/coccicheck we take the absolute directory
of it and then go down one level of directory, so in this case it
would be the base directory of the Linux kernel.

$$ is the PID of the bash script.

  Luis

[toc] | [prev] | [next] | [standalone]

#1422349

On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:

> On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote:
> > 
> > 
> > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > 
> > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > > > 
> > > > 
> > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > 
> > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > > > 
> > > > > > 
> > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > > 
> > > > > > > I'll redirect stderr to stdout by default when parmap support is used then.
> > > > > > 
> > > > > > Usually I put them in different files.
> > > > > 
> > > > > We can do that as well but I would only want to deal with parmap support 
> > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would 
> > > > > be the PID of the shell script?
> > > > 
> > > > I don't understand the connection with parmap.
> > > 
> > > When parmap support is not available the cocciscript will currently
> > > disregard stderr, output is provided as it comes to stdout from each
> > > thread I guess.
> > 
> > Deepa's recent patch to coccicheck made apparent that Coccicheck uses 
> > --very-quiet, so there is standard error.
> 
> OK I'm disegarding the redirect for non-parmap for now but we'd have to
> determine if we want to append or add one per PID... I rather leave that
> stuff as-is and encourage folks to upgrade coccinelle.

If coccicheck is using --very-quiet, there will not be much stderr of 
interest when using parmap either.

I'm not sure to understan the issue about appending.  Appending what to 
what?  While Coccinelle is running, you have a directory with the same 
name as your semantic patch (or a directory name that you specify with 
--tmp-dir) that has separate files for each core's standard output and 
standard error.  At the end, when all the code has been treated, 
Coccinelle writes the successive stdouts to standard output, and the 
sucessive stderrs to standard error.

> > > > Originally our use of parmap made output files based on pids.  Maybe this 
> > > > is the default for parmap.  I found this completely unusable.  I guess one 
> > > > could look at the dates to see which file is the most recent one, but it 
> > > > seems tedious.  If you are putting the standard output in x.out, then put 
> > > > the standard error in x.err.
> > > 
> > > I'll use ${DIR}/coccicheck.$$.err for stderr.
> > 
> > What is ${DIR}? and what is $$?
> 
> When you run scripts/coccicheck we take the absolute directory
> of it and then go down one level of directory, so in this case it
> would be the base directory of the Linux kernel.
> 
> $$ is the PID of the bash script.

OK.  I still don't find PIDs useful, but I guess if we are talking about 
the entire output of coccicheck, there is not much else to do.  Normally, 
I don't want these files accumulating, and just write over the old ones.

julia

[toc] | [prev] | [next] | [standalone]

#1422380

On Tue, Jun 14, 2016 at 11:17:13PM +0200, Julia Lawall wrote:
> 
> 
> On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote:
> > > 
> > > 
> > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > > 
> > > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > > > > 
> > > > > 
> > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > 
> > > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > > > > 
> > > > > > > 
> > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > > > 
> > > > > > > > I'll redirect stderr to stdout by default when parmap support is used then.
> > > > > > > 
> > > > > > > Usually I put them in different files.
> > > > > > 
> > > > > > We can do that as well but I would only want to deal with parmap support 
> > > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would 
> > > > > > be the PID of the shell script?
> > > > > 
> > > > > I don't understand the connection with parmap.
> > > > 
> > > > When parmap support is not available the cocciscript will currently
> > > > disregard stderr, output is provided as it comes to stdout from each
> > > > thread I guess.
> > > 
> > > Deepa's recent patch to coccicheck made apparent that Coccicheck uses 
> > > --very-quiet, so there is standard error.
> > 
> > OK I'm disegarding the redirect for non-parmap for now but we'd have to
> > determine if we want to append or add one per PID... I rather leave that
> > stuff as-is and encourage folks to upgrade coccinelle.
> 
> If coccicheck is using --very-quiet, there will not be much stderr of 
> interest when using parmap either.

OK I don't follow. Does coccinelle only direct error to stderr when --very-quiet
is used ? Or does using --very-quiet suppress stderr ?

> I'm not sure to understan the issue about appending.  Appending what to 
> what?
> While Coccinelle is running, you have a directory with the same 
> name as your semantic patch (or a directory name that you specify with 
> --tmp-dir) that has separate files for each core's standard output and 
> standard error.  At the end, when all the code has been treated, 
> Coccinelle writes the successive stdouts to standard output, and the 
> sucessive stderrs to standard error.

Ah I see. OK yeah never mind about appending.

> > > > > Originally our use of parmap made output, specia files based on pids.  Maybe this 
> > > > > is the default for parmap.  I found this completely unusable.  I guess one 
> > > > > could look at the dates to see which file is the most recent one, but it 
> > > > > seems tedious.  If you are putting the standard output in x.out, then put 
> > > > > the standard error in x.err.
> > > > 
> > > > I'll use ${DIR}/coccicheck.$$.err for stderr.
> > > 
> > > What is ${DIR}? and what is $$?
> > 
> > When you run scripts/coccicheck we take the absolute directory
> > of it and then go down one level of directory, so in this case it
> > would be the base directory of the Linux kernel.
> > 
> > $$ is the PID of the bash script.
> 
> OK.  I still don't find PIDs useful, but I guess if we are talking about 
> the entire output of coccicheck, there is not much else to do.  Normally, 
> I don't want these files accumulating, and just write over the old ones.

Which is why I would much prefer to instead just redirect in coccicheck
case stderr to stdout from coccinelle. Is that preferred?

  Luis

[toc] | [prev] | [next] | [standalone]

#1422734

On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:

> On Tue, Jun 14, 2016 at 11:17:13PM +0200, Julia Lawall wrote:
> >
> >
> > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> >
> > > On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote:
> > > >
> > > >
> > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > > >
> > > > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > > > > >
> > > > > >
> > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > >
> > > > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > > > >
> > > > > > > > > I'll redirect stderr to stdout by default when parmap support is used then.
> > > > > > > >
> > > > > > > > Usually I put them in different files.
> > > > > > >
> > > > > > > We can do that as well but I would only want to deal with parmap support
> > > > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would
> > > > > > > be the PID of the shell script?
> > > > > >
> > > > > > I don't understand the connection with parmap.
> > > > >
> > > > > When parmap support is not available the cocciscript will currently
> > > > > disregard stderr, output is provided as it comes to stdout from each
> > > > > thread I guess.
> > > >
> > > > Deepa's recent patch to coccicheck made apparent that Coccicheck uses
> > > > --very-quiet, so there is standard error.
> > >
> > > OK I'm disegarding the redirect for non-parmap for now but we'd have to
> > > determine if we want to append or add one per PID... I rather leave that
> > > stuff as-is and encourage folks to upgrade coccinelle.
> >
> > If coccicheck is using --very-quiet, there will not be much stderr of
> > interest when using parmap either.
>
> OK I don't follow. Does coccinelle only direct error to stderr when --very-quiet
> is used ? Or does using --very-quiet suppress stderr ?

--very-quiet suppresses most administrative messages that go to stderr.
There are still actual errors.  Bu you don't see eg what file is being
currently processed.

> > > > > > Originally our use of parmap made output, specia files based on pids.  Maybe this
> > > > > > is the default for parmap.  I found this completely unusable.  I guess one
> > > > > > could look at the dates to see which file is the most recent one, but it
> > > > > > seems tedious.  If you are putting the standard output in x.out, then put
> > > > > > the standard error in x.err.
> > > > >
> > > > > I'll use ${DIR}/coccicheck.$$.err for stderr.
> > > >
> > > > What is ${DIR}? and what is $$?
> > >
> > > When you run scripts/coccicheck we take the absolute directory
> > > of it and then go down one level of directory, so in this case it
> > > would be the base directory of the Linux kernel.
> > >
> > > $$ is the PID of the bash script.
> >
> > OK.  I still don't find PIDs useful, but I guess if we are talking about
> > the entire output of coccicheck, there is not much else to do.  Normally,
> > I don't want these files accumulating, and just write over the old ones.
>
> Which is why I would much prefer to instead just redirect in coccicheck
> case stderr to stdout from coccinelle. Is that preferred?

Then things will be merged in strange ways.

Why not just let the user decide what to do with these things?

julia

[toc] | [prev] | [next] | [standalone]

#1423144

On Wed, Jun 15, 2016 at 09:39:49AM +0200, Julia Lawall wrote:
> 
> 
> On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Tue, Jun 14, 2016 at 11:17:13PM +0200, Julia Lawall wrote:
> > >
> > >
> > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > >
> > > > On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote:
> > > > >
> > > > >
> > > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > > > >
> > > > > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > > > > > >
> > > > > > >
> > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > >
> > > > > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > > > > >
> > > > > > > > > > I'll redirect stderr to stdout by default when parmap support is used then.
> > > > > > > > >
> > > > > > > > > Usually I put them in different files.
> > > > > > > >
> > > > > > > > We can do that as well but I would only want to deal with parmap support
> > > > > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would
> > > > > > > > be the PID of the shell script?
> > > > > > >
> > > > > > > I don't understand the connection with parmap.
> > > > > >
> > > > > > When parmap support is not available the cocciscript will currently
> > > > > > disregard stderr, output is provided as it comes to stdout from each
> > > > > > thread I guess.
> > > > >
> > > > > Deepa's recent patch to coccicheck made apparent that Coccicheck uses
> > > > > --very-quiet, so there is standard error.
> > > >
> > > > OK I'm disegarding the redirect for non-parmap for now but we'd have to
> > > > determine if we want to append or add one per PID... I rather leave that
> > > > stuff as-is and encourage folks to upgrade coccinelle.
> > >
> > > If coccicheck is using --very-quiet, there will not be much stderr of
> > > interest when using parmap either.
> >
> > OK I don't follow. Does coccinelle only direct error to stderr when --very-quiet
> > is used ? Or does using --very-quiet suppress stderr ?
> 
> --very-quiet suppresses most administrative messages that go to stderr.
> There are still actual errors.  Bu you don't see eg what file is being
> currently processed.

OK thanks. I remove --very-quiet now if --profile is used within SPFLAGS, I'll extend
this to also avoid --very-quiet if --show-trying is used. SPFLAGS is where you can
specify extra options that the script doesn't specifically support.

> > > > > > > Originally our use of parmap made output, specia files based on pids.  Maybe this
> > > > > > > is the default for parmap.  I found this completely unusable.  I guess one
> > > > > > > could look at the dates to see which file is the most recent one, but it
> > > > > > > seems tedious.  If you are putting the standard output in x.out, then put
> > > > > > > the standard error in x.err.
> > > > > >
> > > > > > I'll use ${DIR}/coccicheck.$$.err for stderr.
> > > > >
> > > > > What is ${DIR}? and what is $$?
> > > >
> > > > When you run scripts/coccicheck we take the absolute directory
> > > > of it and then go down one level of directory, so in this case it
> > > > would be the base directory of the Linux kernel.
> > > >
> > > > $$ is the PID of the bash script.
> > >
> > > OK.  I still don't find PIDs useful, but I guess if we are talking about
> > > the entire output of coccicheck, there is not much else to do.  Normally,
> > > I don't want these files accumulating, and just write over the old ones.
> >
> > Which is why I would much prefer to instead just redirect in coccicheck
> > case stderr to stdout from coccinelle. Is that preferred?
> 
> Then things will be merged in strange ways.
> 
> Why not just let the user decide what to do with these things?

Sure, what should be the default?

  Luis

[toc] | [prev] | [next] | [standalone]

#1423157

On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:

> On Wed, Jun 15, 2016 at 09:39:49AM +0200, Julia Lawall wrote:
> >
> >
> > On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:
> >
> > > On Tue, Jun 14, 2016 at 11:17:13PM +0200, Julia Lawall wrote:
> > > >
> > > >
> > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > > >
> > > > > On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote:
> > > > > >
> > > > > >
> > > > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > > > > >
> > > > > > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > > >
> > > > > > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > > > > > >
> > > > > > > > > > > I'll redirect stderr to stdout by default when parmap support is used then.
> > > > > > > > > >
> > > > > > > > > > Usually I put them in different files.
> > > > > > > > >
> > > > > > > > > We can do that as well but I would only want to deal with parmap support
> > > > > > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would
> > > > > > > > > be the PID of the shell script?
> > > > > > > >
> > > > > > > > I don't understand the connection with parmap.
> > > > > > >
> > > > > > > When parmap support is not available the cocciscript will currently
> > > > > > > disregard stderr, output is provided as it comes to stdout from each
> > > > > > > thread I guess.
> > > > > >
> > > > > > Deepa's recent patch to coccicheck made apparent that Coccicheck uses
> > > > > > --very-quiet, so there is standard error.
> > > > >
> > > > > OK I'm disegarding the redirect for non-parmap for now but we'd have to
> > > > > determine if we want to append or add one per PID... I rather leave that
> > > > > stuff as-is and encourage folks to upgrade coccinelle.
> > > >
> > > > If coccicheck is using --very-quiet, there will not be much stderr of
> > > > interest when using parmap either.
> > >
> > > OK I don't follow. Does coccinelle only direct error to stderr when --very-quiet
> > > is used ? Or does using --very-quiet suppress stderr ?
> >
> > --very-quiet suppresses most administrative messages that go to stderr.
> > There are still actual errors.  Bu you don't see eg what file is being
> > currently processed.
>
> OK thanks. I remove --very-quiet now if --profile is used within SPFLAGS, I'll extend
> this to also avoid --very-quiet if --show-trying is used. SPFLAGS is where you can
> specify extra options that the script doesn't specifically support.

If it is more convenient, you don't actually have to remove --very-quiet.
You can just put --quiet before --show-trying or --profile.  --quiet will
override --very-quiet.

> > > > > > > > Originally our use of parmap made output, specia files based on pids.  Maybe this
> > > > > > > > is the default for parmap.  I found this completely unusable.  I guess one
> > > > > > > > could look at the dates to see which file is the most recent one, but it
> > > > > > > > seems tedious.  If you are putting the standard output in x.out, then put
> > > > > > > > the standard error in x.err.
> > > > > > >
> > > > > > > I'll use ${DIR}/coccicheck.$$.err for stderr.
> > > > > >
> > > > > > What is ${DIR}? and what is $$?
> > > > >
> > > > > When you run scripts/coccicheck we take the absolute directory
> > > > > of it and then go down one level of directory, so in this case it
> > > > > would be the base directory of the Linux kernel.
> > > > >
> > > > > $$ is the PID of the bash script.
> > > >
> > > > OK.  I still don't find PIDs useful, but I guess if we are talking about
> > > > the entire output of coccicheck, there is not much else to do.  Normally,
> > > > I don't want these files accumulating, and just write over the old ones.
> > >
> > > Which is why I would much prefer to instead just redirect in coccicheck
> > > case stderr to stdout from coccinelle. Is that preferred?
> >
> > Then things will be merged in strange ways.
> >
> > Why not just let the user decide what to do with these things?
>
> Sure, what should be the default?

I would normally just expect standard output and standard error to appear
randomly on the screen.  That is, no management effort from the tool at
all.

julia

[toc] | [prev] | [next] | [standalone]

#1423274

On Wed, Jun 15, 2016 at 05:44:01PM +0200, Julia Lawall wrote:
> 
> 
> On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:
> > OK thanks. I remove --very-quiet now if --profile is used within SPFLAGS, I'll extend
> > this to also avoid --very-quiet if --show-trying is used. SPFLAGS is where you can
> > specify extra options that the script doesn't specifically support.
> 
> If it is more convenient, you don't actually have to remove --very-quiet.
> You can just put --quiet before --show-trying or --profile.  --quiet will
> override --very-quiet.


How about just:

if [ "$SPFLAGS"  == *"--profile"* -o "$SPFLAGS"  == "--show-trying" ]; then     
        FLAGS="--quiet $SPFLAGS"                                                
else                                                                            
        FLAGS="--very-quiet $SPFLAGS"                                           
fi 

> > > > > > > > > Originally our use of parmap made output, specia files based on pids.  Maybe this
> > > > > > > > > is the default for parmap.  I found this completely unusable.  I guess one
> > > > > > > > > could look at the dates to see which file is the most recent one, but it
> > > > > > > > > seems tedious.  If you are putting the standard output in x.out, then put
> > > > > > > > > the standard error in x.err.
> > > > > > > >
> > > > > > > > I'll use ${DIR}/coccicheck.$$.err for stderr.
> > > > > > >
> > > > > > > What is ${DIR}? and what is $$?
> > > > > >
> > > > > > When you run scripts/coccicheck we take the absolute directory
> > > > > > of it and then go down one level of directory, so in this case it
> > > > > > would be the base directory of the Linux kernel.
> > > > > >
> > > > > > $$ is the PID of the bash script.
> > > > >
> > > > > OK.  I still don't find PIDs useful, but I guess if we are talking about
> > > > > the entire output of coccicheck, there is not much else to do.  Normally,
> > > > > I don't want these files accumulating, and just write over the old ones.
> > > >
> > > > Which is why I would much prefer to instead just redirect in coccicheck
> > > > case stderr to stdout from coccinelle. Is that preferred?
> > >
> > > Then things will be merged in strange ways.
> > >
> > > Why not just let the user decide what to do with these things?
> >
> > Sure, what should be the default?
> 
> I would normally just expect standard output and standard error to appear
> randomly on the screen.  That is, no management effort from the tool at
> all.

But the thing is, stderr is ignored now given that a shell script is used
wrapped over a Makefile so if we want what you describe I think we do
have to by default do 2>&1 on the spatch run command.

  Luis

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web