Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #248505 > unrolled thread

grep: show matching line from pattern file

Back to article view | Back to linux.debian.user

Contents

  grep: show matching line from pattern file Jim Popovitch <jim@k4vqc.com> - 2022-05-28 22:00 +0200
    Re: grep: show matching line from pattern file The Wanderer <wanderer@fastmail.fm> - 2022-05-28 22:10 +0200
      Re: grep: show matching line from pattern file Greg Wooledge <greg@wooledge.org> - 2022-05-28 23:20 +0200
        Re: grep: show matching line from pattern file The Wanderer <wanderer@fastmail.fm> - 2022-05-28 23:30 +0200
        Re: grep: show matching line from pattern file Jim Popovitch <jim@k4vqc.com> - 2022-05-29 19:50 +0200
        Re: grep: show matching line from pattern file Will Mengarini <seldon@eskimo.com> - 2022-06-03 00:30 +0200
          Re: grep: show matching line from pattern file David Christensen <dpchrist@holgerdanske.com> - 2022-06-03 01:00 +0200
            Re: grep: show matching line from pattern file Will Mengarini <seldon@eskimo.com> - 2022-06-03 02:20 +0200
    Re: grep: show matching line from pattern file Jörg-Volker Peetz <jvpeetz@web.de> - 2022-05-29 15:10 +0200
      Re: grep: show matching line from pattern file David Wright <deblis@lionunicorn.co.uk> - 2022-05-29 15:50 +0200
        Re: grep: show matching line from pattern file duh <fill_in_the_blanks@email.com> - 2022-06-02 21:20 +0200
          Re: grep: show matching line from pattern file Greg Wooledge <greg@wooledge.org> - 2022-06-02 21:20 +0200
            Re: grep: show matching line from pattern file Richard Hector <richard@walnut.gen.nz> - 2022-06-04 05:00 +0200

#248505 — grep: show matching line from pattern file

Not exactly Debian specific, but hoping that someone here can help.

I have a file of regex patterns and I use grep like so:

   ~$ grep -f patterns.txt /var/log/syslog 

What I'd like to get is a listing of all lines, specifically the line
numbers of the regexps in patterns.txt, that match entries in
/var/log/syslog.   Is there a way to do this?

-Jim P.

[toc] | [next] | [standalone]

#248506

[Multipart message — attachments visible in raw view] — view raw

On 2022-05-28 at 15:40, Jim Popovitch wrote:

> Not exactly Debian specific, but hoping that someone here can help.
> 
> I have a file of regex patterns and I use grep like so:
> 
>    ~$ grep -f patterns.txt /var/log/syslog 
> 
> What I'd like to get is a listing of all lines, specifically the line
> numbers of the regexps in patterns.txt, that match entries in
> /var/log/syslog.   Is there a way to do this?

I don't know of a standardized way to do that (if anyone else wants to
suggest one, I'm open to learn), but of course it *can* be done, via
scripting. Off the top of my head, I came up with the following

for line in $(seq 1 $(wc -l patterns.txt | cut -d ' ' -f 1)) ; do
  if grep $(head -n $line patterns.txt | tail -n 1) /var/log/syslog >
/dev/null ; then
    echo $line ;
  fi
done

I just tested that on my own system, with a different file (since I'm
not root right now) and a couple of exact-string patterns found by
examining that file, and it seems to work as intended. YMMV.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw

[toc] | [prev] | [next] | [standalone]

#248508

On Sat, May 28, 2022 at 04:02:39PM -0400, The Wanderer wrote:
> On 2022-05-28 at 15:40, Jim Popovitch wrote:
> > I have a file of regex patterns and I use grep like so:
> > 
> >    ~$ grep -f patterns.txt /var/log/syslog 
> > 
> > What I'd like to get is a listing of all lines, specifically the line
> > numbers of the regexps in patterns.txt, that match entries in
> > /var/log/syslog.   Is there a way to do this?
> 
> I don't know of a standardized way to do that (if anyone else wants to
> suggest one, I'm open to learn), but of course it *can* be done, via
> scripting. Off the top of my head, I came up with the following
> 
> for line in $(seq 1 $(wc -l patterns.txt | cut -d ' ' -f 1)) ; do
>   if grep $(head -n $line patterns.txt | tail -n 1) /var/log/syslog >
> /dev/null ; then
>     echo $line ;
>   fi
> done

The quoting here is... completely absent (and that's extremely bad), but
also importantly, one would ideally like to avoid running grep a thousand
times, especially if the target logfile is large.

I believe this is the kind of job for which perl is well-suited.  I'm not
great at perl, but I'll give it a shot.

Here's a version with some extra information as output, so I can verify
that it's doing something reasonably close to correct:


#!/usr/bin/perl
use strict; use warnings;

my @patlist;
open PATS, "<patterns.txt" || die "can't open patterns.txt";
chomp(@patlist = <PATS>);
close PATS;

while (<>) {
    chomp;
    for (my $i = 0; $i <= $#patlist; $i++) {
	print "$i|$patlist[$i]|$_\n" if /$patlist[$i]/;
    }
}


Now, to test it, we need a patterns.txt file:


unicorn:~$ cat patterns.txt 
PATH
HOME|~
a...e


And an input (log) file:


unicorn:~$ cat file
zebra
Home, home on the range.
Oops, I meant HOME on the range.

applesauce


And here's what it does:


unicorn:~$ ./foo file
1|HOME|~|Oops, I meant HOME on the range.
2|a...e|applesauce


Pattern numbers 1 and 2 (the second and third, since it starts at 0) were
matched, so we have a line for each of those.

If that's kinda what you wanted, then you can adjust this to do precisely
what you wanted.  It shouldn't take a lot of work, I hope.  Well, I guess
that depends on what you really want.

Bash is not well-suited to this task, and even if we were to take The
Wanderer's script and fix all the issues in it, it would still be a
vastly inferior solution.  Some tools are just not meant for some jobs.

[toc] | [prev] | [next] | [standalone]

#248509

[Multipart message — attachments visible in raw view] — view raw

On 2022-05-28 at 17:11, Greg Wooledge wrote:

> On Sat, May 28, 2022 at 04:02:39PM -0400, The Wanderer wrote:
>
>> On 2022-05-28 at 15:40, Jim Popovitch wrote:
>> > I have a file of regex patterns and I use grep like so:
>> > 
>> >    ~$ grep -f patterns.txt /var/log/syslog 
>> > 
>> > What I'd like to get is a listing of all lines, specifically the line
>> > numbers of the regexps in patterns.txt, that match entries in
>> > /var/log/syslog.   Is there a way to do this?
>> 
>> I don't know of a standardized way to do that (if anyone else wants to
>> suggest one, I'm open to learn), but of course it *can* be done, via
>> scripting. Off the top of my head, I came up with the following
>> 
>> for line in $(seq 1 $(wc -l patterns.txt | cut -d ' ' -f 1)) ; do
>>   if grep $(head -n $line patterns.txt | tail -n 1) /var/log/syslog >
>> /dev/null ; then
>>     echo $line ;
>>   fi
>> done
> 
> The quoting here is... completely absent (and that's extremely bad), but
> also importantly, one would ideally like to avoid running grep a thousand
> times, especially if the target logfile is large.

A brother of mine has schooled me on several things I did wrong here
already, and I am now going over my long-tail stockpile of scripts with
shellcheck, seeing what I can learn. (I normally do quote variables, but
for some reason it slipped my mind this time until after I'd already hit
Send. Some of these are known-safe values which won't need quoting, but
others aren't, so the principle remains valid to cite.)

This wasn't especially intended as a great solution, in any case; it
started out as me trying to write shell-like pseudocode, but then I
couldn't see any obvious reason why it wouldn't work, and when I tried
it out I only needed a few tweaks before it did. There's a *reason* I
had the "YMMV" on that message.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw

[toc] | [prev] | [next] | [standalone]

#248542

On Sat, 2022-05-28 at 17:11 -0400, Greg Wooledge wrote:
> On Sat, May 28, 2022 at 04:02:39PM -0400, The Wanderer wrote:
> > On 2022-05-28 at 15:40, Jim Popovitch wrote:
> > > I have a file of regex patterns and I use grep like so:
> > > 
> > >    ~$ grep -f patterns.txt /var/log/syslog 
> > > 
> > > What I'd like to get is a listing of all lines, specifically the line
> > > numbers of the regexps in patterns.txt, that match entries in
> > > /var/log/syslog.   Is there a way to do this?
> > 
> > I don't know of a standardized way to do that (if anyone else wants to
> > suggest one, I'm open to learn), but of course it *can* be done, via
> > scripting. Off the top of my head, I came up with the following
> > 
> > for line in $(seq 1 $(wc -l patterns.txt | cut -d ' ' -f 1)) ; do
> >   if grep $(head -n $line patterns.txt | tail -n 1) /var/log/syslog >
> > /dev/null ; then
> >     echo $line ;
> >   fi
> > done
> 
> The quoting here is... completely absent (and that's extremely bad), but
> also importantly, one would ideally like to avoid running grep a thousand
> times, especially if the target logfile is large.
> 
> I believe this is the kind of job for which perl is well-suited.  I'm not
> great at perl, but I'll give it a shot.
> 
> Here's a version with some extra information as output, so I can verify
> that it's doing something reasonably close to correct:
> 
> 
> #!/usr/bin/perl
> use strict; use warnings;
> 
> my @patlist;
> open PATS, "<patterns.txt" || die "can't open patterns.txt";
> chomp(@patlist = <PATS>);
> close PATS;
> 
> while (<>) {
>     chomp;
>     for (my $i = 0; $i <= $#patlist; $i++) {
> 	print "$i|$patlist[$i]|$_\n" if /$patlist[$i]/;
>     }
> }
> 
> 
> Now, to test it, we need a patterns.txt file:
> 
> 
> unicorn:~$ cat patterns.txt 
> PATH
> HOME|~
> a...e
> 
> 
> And an input (log) file:
> 
> 
> unicorn:~$ cat file
> zebra
> Home, home on the range.
> Oops, I meant HOME on the range.
> 
> applesauce
> 
> 
> And here's what it does:
> 
> 
> unicorn:~$ ./foo file
> 1|HOME|~|Oops, I meant HOME on the range.
> 2|a...e|applesauce
> 
> 
> Pattern numbers 1 and 2 (the second and third, since it starts at 0) were
> matched, so we have a line for each of those.
> 
> If that's kinda what you wanted, then you can adjust this to do precisely
> what you wanted.  It shouldn't take a lot of work, I hope.  Well, I guess
> that depends on what you really want.
> 
> Bash is not well-suited to this task, and even if we were to take The
> Wanderer's script and fix all the issues in it, it would still be a
> vastly inferior solution.  Some tools are just not meant for some jobs.
> 

Thanks Greg, that is exactly what I needed, and double thanks for the
details in explaining it, etc. 

-Jim P.

[toc] | [prev] | [next] | [standalone]

#248682

* Greg Wooledge <greg@wooledge.org> [22-05/28=Sa 17:11 -0400]:
> [...] 
> #!/usr/bin/perl
> use strict; use warnings;
> [...] 
> open PATS, "<patterns.txt" || die "can't open patterns.txt";
> [...] 

You need "or die", not "|| die", because of precedence: what you coded
checks whether "<patterns.txt" is logically true (it is), whereas you
wanted to check whether the result of open() is logically true.

In this transcript, the number before the prompt-ending '$' is $?:
--------------------------------
debian/pts/4 bash3 ~ 14:56 0$perl -e 'open "gweeblefleep" || die'
debian/pts/4 bash3 ~ 14:57 0$perl -e 'open "gweeblefleep" or die'
Died at -e line 1.
debian/pts/4 bash3 ~ 14:57 2$
--------------------------------

-- 
                 Will Mengarini  <seldon@eskimo.com>
                perl -le"print unpack '%C*',MENGARINI"

[toc] | [prev] | [next] | [standalone]

#248683

On 6/2/22 15:13, Will Mengarini wrote:
> * Greg Wooledge <greg@wooledge.org> [22-05/28=Sa 17:11 -0400]:
>> [...]
>> #!/usr/bin/perl
>> use strict; use warnings;
>> [...]
>> open PATS, "<patterns.txt" || die "can't open patterns.txt";
>> [...]
> 
> You need "or die", not "|| die", because of precedence: what you coded
> checks whether "<patterns.txt" is logically true (it is), whereas you
> wanted to check whether the result of open() is logically true.


+1  That is a good explanation of a Perl fine point/ gotcha.


> In this transcript, the number before the prompt-ending '$' is $?:
> --------------------------------
> debian/pts/4 bash3 ~ 14:56 0$perl -e 'open "gweeblefleep" || die'
> debian/pts/4 bash3 ~ 14:57 0$perl -e 'open "gweeblefleep" or die'
> Died at -e line 1.
> debian/pts/4 bash3 ~ 14:57 2$
> --------------------------------


What is your shell?  PS1?


David

[toc] | [prev] | [next] | [standalone]

#248684

* David Christensen <dpchrist@holgerdanske.com> [22-06/02=Thu 15:50 -0700]:
> On 6/2/22 15:13, Will Mengarini wrote:
>> * Greg Wooledge <greg@wooledge.org> [22-05/28=Sa 17:11 -0400]:
>>> [...]
>>> #!/usr/bin/perl
>>> use strict; use warnings;
>>> [...]
>>> open PATS, "<patterns.txt" || die "can't open patterns.txt";
>>> [...]
>>
>> You need "or die", not "|| die", because of precedence: what you coded
>> checks whether "<patterns.txt" is logically true (it is), whereas you
>> wanted to check whether the result of open() is logically true.
>
> +1  That is a good explanation of a Perl fine point/ gotcha.
>
>> In this transcript, the number before the prompt-ending '$' is $?:
>> --------------------------------
>> debian/pts/4 bash3 ~ 14:56 0$perl -e 'open "gweeblefleep" || die'
>> debian/pts/4 bash3 ~ 14:57 0$perl -e 'open "gweeblefleep" or die'
>> Died at -e line 1.
>> debian/pts/4 bash3 ~ 14:57 2$
>> --------------------------------
>
> What is your shell?  PS1?

The shell is Bash 5.1.4.  My PS1 is constructed by an elaborate script
that's old enough to have sex in Thailand, but you can get the effect
of what I posted by setting PS1 with the line
--------------------------------
PS1="\\h/${TTY#/dev/} \\s$SHLVL \\w \\A \$?\\\$"
--------------------------------
assuming you're running at least Bash 2.05a.  You may prefer
--------------------------------
PS1="\\h/${TTY#/dev/} \\s^$SHLVL \\w \\A \$?\\\$"
--------------------------------.

My original script was coded for Bash 1.4.7, and had to do
--------------------------------
PS1="\\h${TTY#/dev/} \\s$SHLVL \\w \`s=\$?;date +%H:%M;exit \$s\` \$?\\\$"
--------------------------------
because \A wasn't available, so 'date' had to be run in a subshell
that needed to take care to save and restore $?.  (The variable
it uses for that, s, goes away when the subshell does; and that
scary-looking exit just exits the subshell, resetting $?.)

-- 
                 Will Mengarini  <seldon@eskimo.com>
         Free software: the Source will be with you, always.
               sh -c 'echo -n MENGARINI|sum -s|colrm 4'

[toc] | [prev] | [next] | [standalone]

#248528

Jim Popovitch wrote on 28/05/2022 21:40:
> Not exactly Debian specific, but hoping that someone here can help.
> 
> I have a file of regex patterns and I use grep like so:
> 
>     ~$ grep -f patterns.txt /var/log/syslog
> 
> What I'd like to get is a listing of all lines, specifically the line
> numbers of the regexps in patterns.txt, that match entries in
> /var/log/syslog.   Is there a way to do this?
> 
> -Jim P.

How about this:

$ grep -of patterns.txt /var/log/syslog.1 | grep -n -f - patterns.txt

Regards,
Jörg.

[toc] | [prev] | [next] | [standalone]

#248529

On Sun 29 May 2022 at 15:02:35 (+0200), Jörg-Volker Peetz wrote:
> Jim Popovitch wrote on 28/05/2022 21:40:
> > Not exactly Debian specific, but hoping that someone here can help.
> > 
> > I have a file of regex patterns and I use grep like so:
> > 
> >     ~$ grep -f patterns.txt /var/log/syslog
> > 
> > What I'd like to get is a listing of all lines, specifically the line
> > numbers of the regexps in patterns.txt, that match entries in
> > /var/log/syslog.   Is there a way to do this?
> 
> How about this:
> 
> $ grep -of patterns.txt /var/log/syslog.1 | grep -n -f - patterns.txt

That will only work for literal patterns, not regex ones.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#248679

On 5/29/22 9:44 AM, David Wright wrote:
> On Sun 29 May 2022 at 15:02:35 (+0200), Jörg-Volker Peetz wrote:
>> Jim Popovitch wrote on 28/05/2022 21:40:
>>> Not exactly Debian specific, but hoping that someone here can help.
>>>
>>> I have a file of regex patterns and I use grep like so:
>>>
>>>      ~$ grep -f patterns.txt /var/log/syslog
>>>
>>> What I'd like to get is a listing of all lines, specifically the line
>>> numbers of the regexps in patterns.txt, that match entries in
>>> /var/log/syslog.   Is there a way to do this?
>> How about this:
>>
>> $ grep -of patterns.txt /var/log/syslog.1 | grep -n -f - patterns.txt
> That will only work for literal patterns, not regex ones.
>
> Cheers,
> David.
>

I may be missing a lot but I will risk throwing in my 2 cents (which is
no longer worth the proverbial amount with

the current inflation)


$cat -n /var/log/syslog | grep warn

and it found "warn" in the syslog file and provided line numbers. I have
not used the -f option

(but am now aware of it in the pastfrom your post -- thank you. Might
file that away for future use but will not press

my luck at the moment).

My approach  saves me the effort of writing a scriptl. Ahh, the Perl
philosophy of "there is more than one way to do it".

[toc] | [prev] | [next] | [standalone]

#248680

On Thu, Jun 02, 2022 at 03:12:23PM -0400, duh wrote:

> > > Jim Popovitch wrote on 28/05/2022 21:40:
> > > > I have a file of regex patterns and I use grep like so:
> > > > 
> > > >      ~$ grep -f patterns.txt /var/log/syslog
> > > > 
> > > > What I'd like to get is a listing of all lines, specifically the line
> > > > numbers of the regexps in patterns.txt, that match entries in
> > > > /var/log/syslog.   Is there a way to do this?

> $cat -n /var/log/syslog | grep warn
> 
> and it found "warn" in the syslog file and provided line numbers. I have
> not used the -f option

You're getting the line numbers from the log file.  The OP wanted the line
numbers of the patterns in the -f pattern file.

Why?  I have no idea.  There is no standard option to do this, because
it's not a common requirement.  That's why I wrote one from scratch
in perl.

[toc] | [prev] | [next] | [standalone]

#248730

On 3/06/22 07:17, Greg Wooledge wrote:
> On Thu, Jun 02, 2022 at 03:12:23PM -0400, duh wrote:
> 
>> > > Jim Popovitch wrote on 28/05/2022 21:40:
>> > > > I have a file of regex patterns and I use grep like so:
>> > > > 
>> > > >      ~$ grep -f patterns.txt /var/log/syslog
>> > > > 
>> > > > What I'd like to get is a listing of all lines, specifically the line
>> > > > numbers of the regexps in patterns.txt, that match entries in
>> > > > /var/log/syslog.   Is there a way to do this?
> 
>> $cat -n /var/log/syslog | grep warn
>> 
>> and it found "warn" in the syslog file and provided line numbers. I have
>> not used the -f option
> 
> You're getting the line numbers from the log file.  The OP wanted the line
> numbers of the patterns in the -f pattern file.
> 
> Why?  I have no idea.  There is no standard option to do this, because
> it's not a common requirement.  That's why I wrote one from scratch
> in perl.
> 

I don't know what the OP's use case is, but here's an example I might use:

I have a bunch of custom ignore files for logcheck. After a software 
upgrade, I might want to check which patterns no longer match anything, 
and can be deleted or modified.

I'd really still want to check with real egrep, though, rather than 
using perl's re engine instead.

Cheers,
Richard

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web