Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #223716 > unrolled thread

one liner, how do you know which match happened ...

Back to article view | Back to linux.debian.user

Contents

  one liner, how do you know which match happened ... Albretch Mueller <lbrtchx@gmail.com> - 2020-06-20 12:30 +0200
    Re: one liner, how do you know which match happened ... <tomas@tuxteam.de> - 2020-06-20 13:10 +0200
      Re: one liner, how do you know which match happened ... Albretch Mueller <lbrtchx@gmail.com> - 2020-06-24 12:40 +0200
        Re: one liner, how do you know which match happened ... tomas@tuxteam.de - 2020-06-24 13:00 +0200
          Re: one liner, how do you know which match happened ... Greg Wooledge <wooledg@eeg.ccf.org> - 2020-06-24 14:20 +0200
            Re: one liner, how do you know which match happened ... Albretch Mueller <lbrtchx@gmail.com> - 2020-06-24 17:30 +0200
    Re: one liner, how do you know which match happened ... David Wright <deblis@lionunicorn.co.uk> - 2020-06-20 16:40 +0200
    Re: one liner, how do you know which match happened ... David Christensen <dpchrist@holgerdanske.com> - 2020-06-20 22:30 +0200
    Re: one liner, how do you know which match happened ... davidson <davidson@freevolt.org> - 2020-06-20 23:10 +0200
      Re: one liner, how do you know which match happened ... davidson <davidson@freevolt.org> - 2020-06-20 23:30 +0200
        Re: one liner, how do you know which match happened ... Klaus Singvogel <deb-user-ml@singvogel.net> - 2020-06-21 01:10 +0200
    Re: one liner, how do you know which match happened ... Greg Wooledge <wooledg@eeg.ccf.org> - 2020-06-22 13:50 +0200

#223716 — one liner, how do you know which match happened ...

_X=".\(html\|txt\)"
_SDIR="$(pwd)"

_AR_TERMS=(
Kant
"Gilbert Ryle"
Hegel
)

for iZ in ${!_AR_TERMS[@]}; do
 find "${_SDIR}" -type f -iregex .*"${_X}" -exec grep -il
"${_AR_TERMS[$iZ]}" {} \;
done # iZ: terms search/grep'ped inside text files;  echo "~";


# this would be much faster

find "${_SDIR}" -type f -iregex .*"${_X}" -exec grep -il
"Kant\|Gilbert Ryle\|Hegel" {} \;

but how do I know which match happened in order to save it into separate files?

 grep doesn't do replacements:

 https://stackoverflow.com/questions/16197406/grep-regex-replace-specific-find-in-text-file

 but at least (in my way to understand reality, since it must try such
searches sequentially) it should give  you the index of the match and
if grep doesn't do that I am sure some other batch utility would (I
havenever used sed in my code)

 lbrtchx

[toc] | [next] | [standalone]

#223717

[Multipart message — attachments visible in raw view] — view raw

On Sat, Jun 20, 2020 at 12:20:41PM +0200, Albretch Mueller wrote:
> _X=".\(html\|txt\)"
> _SDIR="$(pwd)"
> 
> _AR_TERMS=(
> Kant
> "Gilbert Ryle"
> Hegel
> )
> 
> for iZ in ${!_AR_TERMS[@]}; do
>  find "${_SDIR}" -type f -iregex .*"${_X}" -exec grep -il
> "${_AR_TERMS[$iZ]}" {} \;
> done # iZ: terms search/grep'ped inside text files;  echo "~";
> 
> 
> # this would be much faster
> 
> find "${_SDIR}" -type f -iregex .*"${_X}" -exec grep -il
> "Kant\|Gilbert Ryle\|Hegel" {} \;
> 
> but how do I know which match happened in order to save it into separate files?

Hm. The first approach goes three times through your files, once for
each term. The second goes once, for a combined regular expression.

So no wonder the second approach is faster.

But to actually attack the problem you should be aware that the
second method is doing *something different* from the first one:
"grep -l" will stop at the first hit, so even if you could ask
grep which one of the alternatives it found, it'll miss Hegel
in a file where Kant figures first. Is that what you want?

Once you have answered that question, you'll be able to proceed.
One possibility is postprocessing your output: grep outputs the
hit line, and you can match that against the individual terms;
you'd have to drop the "-l" for that, making things somewhat slower.

Another possibility is to keep the "-l" and to re-grep the files
found against all the individual patterns.

Cheers
-- t

[toc] | [prev] | [next] | [standalone]

#223944

> "grep -l" will stop at the first hit, so even if you could ask
> grep which one of the alternatives it found, it'll miss Hegel
> in a file where Kant figures first. Is that what you want?

 Yes, I am fine with that. I include the matches in the Array in a way
that the first one is the most important to me anyway, but I need to
know which one, which I think should be someone possible, since grep
must go through the array using some index, no?

 lbrtchx

[toc] | [prev] | [next] | [standalone]

#223946

[Multipart message — attachments visible in raw view] — view raw

On Wed, Jun 24, 2020 at 12:33:19PM +0200, Albretch Mueller wrote:
> > "grep -l" will stop at the first hit, so even if you could ask
> > grep which one of the alternatives it found, it'll miss Hegel
> > in a file where Kant figures first. Is that what you want?
> 
>  Yes, I am fine with that. I include the matches in the Array in a way
> that the first one is the most important to me anyway, but I need to
> know which one, which I think should be someone possible, since grep
> must go through the array using some index, no?

OK, if you just want the first hit, there is "--max-count" (aka "-m"),
which you can use with the combined regular expression. This (given
the value 1) will spit out the first matching line, which you can
then post-process to extract the exact match.

Post-processing is left as an exercise to the reader :-)

That should be about as fast as "grep -l" can get, take or give.

Cheers
-- tomás

[toc] | [prev] | [next] | [standalone]

#223950

On Wed, Jun 24, 2020 at 12:59:03PM +0200, tomas@tuxteam.de wrote:
> On Wed, Jun 24, 2020 at 12:33:19PM +0200, Albretch Mueller wrote:
> > > "grep -l" will stop at the first hit, so even if you could ask
> > > grep which one of the alternatives it found, it'll miss Hegel
> > > in a file where Kant figures first. Is that what you want?
> > 
> >  Yes, I am fine with that. I include the matches in the Array in a way
> > that the first one is the most important to me anyway, but I need to
> > know which one, which I think should be someone possible, since grep
> > must go through the array using some index, no?

Grep doesn't know anything about "arrays".  Grep is a separate utility,
which has its own specification (from POSIX, which in turn derives from
historical implementations), and GNU grep has its own specific
implementation and extensions.

Grep receives its input from the command-line arguments, and standard
input if no filename arguments are given.  That's all.  It doesn't know
anything else about its parent process's variables, or your intentions.

> OK, if you just want the first hit, there is "--max-count" (aka "-m"),
> which you can use with the combined regular expression. This (given
> the value 1) will spit out the first matching line, which you can
> then post-process to extract the exact match.
> 
> Post-processing is left as an exercise to the reader :-)
> 
> That should be about as fast as "grep -l" can get, take or give.

One small problem here:

       -l, --files-with-matches
              Suppress normal output; instead print the  name  of  each  input
              file  from  which  output would normally have been printed.  The
              scanning will stop on the first match.

You don't *see* the matching line when using the -l option.

Once again, bash and grep are really NOT good choices for this goal,
insofar as I've been able to deduce what the goal is.

If you want to match N files against M strings (or regexes) and collate
them into groups according to which string is first seen in each file,
use a real programming language.  Open each file, then read line by line,
and match each line against each of your input strings, until a match
is found, or you hit the end of the file.  Store results accordingly.

Stop trying to use inappropriate tools.  You're just making your life
harder.

[toc] | [prev] | [next] | [standalone]

#223957

On 6/24/20, Greg Wooledge <wooledg@eeg.ccf.org> wrote:
> If you want to match N files against M strings (or regexes) and collate
> them into groups according to which string is first seen in each file,
> use a real programming language.

  Oh, well! I was just too hopeful.

> Stop trying to use inappropriate tools.  You're just making your life
> harder.

  Thank you

 lbrtchx

[toc] | [prev] | [next] | [standalone]

#223729

On Sat 20 Jun 2020 at 12:20:41 (+0200), Albretch Mueller wrote:
> _X=".\(html\|txt\)"
> _SDIR="$(pwd)"
> 
> _AR_TERMS=(
> Kant
> "Gilbert Ryle"
> Hegel
> )
> 
> for iZ in ${!_AR_TERMS[@]}; do
>  find "${_SDIR}" -type f -iregex .*"${_X}" -exec grep -il
> "${_AR_TERMS[$iZ]}" {} \;
> done # iZ: terms search/grep'ped inside text files;  echo "~";
> 
> 
> # this would be much faster
> 
> find "${_SDIR}" -type f -iregex .*"${_X}" -exec grep -il
> "Kant\|Gilbert Ryle\|Hegel" {} \;
> 
> but how do I know which match happened in order to save it into separate files?
> 
>  grep doesn't do replacements:
> 
>  https://stackoverflow.com/questions/16197406/grep-regex-replace-specific-find-in-text-file
> 
>  but at least (in my way to understand reality, since it must try such
> searches sequentially) it should give  you the index of the match and
> if grep doesn't do that I am sure some other batch utility would (I
> havenever used sed in my code)

This script apparently has to potentially open one file for each
search string in _AR_TERMS. So why does the Subject line start with
"one liner"? Are you competing in some sort of obfuscation contest?

There's no point in opening these files on the fly because their
number is limited only by the number of search strings, so their
future contents (the matched filenames) should be stored in lists
which themselves should be in an array of such lists, indexed by
search string. Choose your language for implementing this (there
are plenty) rather than trying to beat grep into submission.

One might assume that the sensible solution will involve searching
each file for multiple strings, rather than rescanning all the files
for each search string. If the latter were necessary, it would make
sense to read the files into memory structures (lists/hashes etc),
and search them there. It makes more sense to me to consider these
factors before playing about with fragments of shell code.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#223765

On 2020-06-20 03:20, Albretch Mueller wrote:
> _X=".\(html\|txt\)"
> _SDIR="$(pwd)"
> 
> _AR_TERMS=(
> Kant
> "Gilbert Ryle"
> Hegel
> )
> 
> for iZ in ${!_AR_TERMS[@]}; do
>   find "${_SDIR}" -type f -iregex .*"${_X}" -exec grep -il
> "${_AR_TERMS[$iZ]}" {} \;
> done # iZ: terms search/grep'ped inside text files;  echo "~";
> 
> 
> # this would be much faster
> 
> find "${_SDIR}" -type f -iregex .*"${_X}" -exec grep -il
> "Kant\|Gilbert Ryle\|Hegel" {} \;
> 
> but how do I know which match happened in order to save it into separate files?
> 
>   grep doesn't do replacements:
> 
>   https://stackoverflow.com/questions/16197406/grep-regex-replace-specific-find-in-text-file
> 
>   but at least (in my way to understand reality, since it must try such
> searches sequentially) it should give  you the index of the match and
> if grep doesn't do that I am sure some other batch utility would (I
> havenever used sed in my code)
> 
>   lbrtchx

It is difficult to infer your intent from code alone.  It would help if 
you posted a problem statement in plain English.


Are you willing to use a language other than sh(1)?  If so, what 
language(s)?


David

[toc] | [prev] | [next] | [standalone]

#223770

On Sat, 20 Jun 2020, Albretch Mueller wrote:

> _X=".\(html\|txt\)"
> _SDIR="$(pwd)"
>
> _AR_TERMS=(
> Kant
> "Gilbert Ryle"
> Hegel
> )
>
> for iZ in ${!_AR_TERMS[@]}; do
> find "${_SDIR}" -type f -iregex .*"${_X}" -exec grep -il
> "${_AR_TERMS[$iZ]}" {} \;
> done # iZ: terms search/grep'ped inside text files;  echo "~";
>
>
> # this would be much faster
>
> find "${_SDIR}" -type f -iregex .*"${_X}" -exec grep -il
> "Kant\|Gilbert Ryle\|Hegel" {} \;

There is also

   -exec '{}' +

instead of the -exec '{}' ';' version. You could compare them.

There was a recent thread here about this?

> but how do I know which match happened in order to save it into
> separate files?

Remove -l flag, use -o flag.

  $ man grep # Read it now. Read it later too.
  ...
  -l, --files-with-matches
        Suppress normal output; instead print the name of each input
        file from which output would normally have been printed.  The
        scanning will stop on the first match.
  ...
   -o, --only-matching

        Print only the matched (non-empty) parts of a matching line,
        with each such part on a separate output line.
  ...
  -n, --line-number

        Prefix each line of output with the 1-based line number within
        its input file.
  ...
  -Z, --null
        Output a zero byte (the ASCII NUL character) instead of the
        character that normally follows a file name.  For example, grep
        -lZ outputs a zero byte after each file name instead of the
        usual newline.  This option makes the output unambiguous, even
        in the presence of file names containing unusual characters
        like newlines.  This option can be used with commands like find
        -print0, perl -0, sort -z, and xargs -0 to process arbitrary
        file names, even those that contain newline characters.

> grep doesn't do replacements:
>
> https://stackoverflow.com/questions/16197406/grep-regex-replace-specific-find-in-text-file

No. Do you want replacements? You haven't asked for that.

> but at least (in my way to understand reality, since it must try such
> searches sequentially) it should give  you the index of the match

It can. See -n flag above.

$ find . -type f \
   \( -name "*.[Hh][Tt][Mm][Ll]" -o -name "[Tt][Xx][Tt]" \) \
   -exec grep -ino "Waldo\|Ice Cream"  '{}' + | uniq
[...]
./wikipedia.org/Transport_Layer_Security.html:999:Ice Cream
./wikipedia.org/Transport_Layer_Security.html:1623:Ice Cream
./wikipedia.org/Edward_R_Murrow.html:338:Waldo
./wikipedia.org/Firefox_version_history.html:3219:Ice Cream
./wikipedia.org/Firefox_version_history.html:3710:Ice Cream
./wikipedia.org/Firefox_version_history.html:6021:Ice Cream
./wikipedia.org/Rwandan_genocide.html:993:Waldo
./wikipedia.org/Agar.html:61:Ice cream
./wikipedia.org/Agar.html:61:ice cream
./wikipedia.org/Turmeric.html:341:ice cream
./wikipedia.org/There_Will_Be_Blood.html:150:ice cream
./wikipedia.org/Boko_haram.html:1328:ice cream
./wikipedia.org/Phantasm_(franchise).html:126:ice cream
./wikipedia.org/Phantasm_(franchise).html:226:ice cream
./wikipedia.org/Thundarr_the_Barbarian.html:153:Waldo
./wikipedia.org/Ecuador.html:1487:waldo
./wikipedia.org/French_Guiana.html:886:Ice cream
./wikipedia.org/French_Guiana.html:886:ice cream
./wikipedia.org/Henry_david_thoreau.html:136:Waldo
./wikipedia.org/Henry_david_thoreau.html:235:Waldo
[...]

> and if grep doesn't do that I am sure some other batch utility would

"Batch utility" sounds like COBOL talk, or something else paleographic
and interesting.

> (I havenever used sed in my code)

I believe sed is a write-only language that is overwhelmingly useful
at command line.

If I find myself putting it in a script, I try to find another way.
For example, awk is far nicer to look at.

-- 
@almightygenie 8 Jun 2020 | @Windex
   Thanks, Windex. That's a relief. Your drink is even more refreshing
   now that I know that it deplores racism & discrimination.
twitter.com/almightygenie/status/1270096054864809988

[toc] | [prev] | [next] | [standalone]

#223777

On Sat, 20 Jun 2020, davidson wrote:

Gah.

> $ find . -type f \
>  \( -name "*.[Hh][Tt][Mm][Ll]" -o -name "[Tt][Xx][Tt]" \) \

  -o -name "*.[Tt][Xx][Tt]"

-- 
@almightygenie 8 Jun 2020 | @Windex
   Thanks, Windex. That's a relief. Your drink is even more refreshing
   now that I know that it deplores racism & discrimination.
twitter.com/almightygenie/status/1270096054864809988

[toc] | [prev] | [next] | [standalone]

#223784

davidson wrote:
>  -o -name "*.[Tt][Xx][Tt]"

or use: -iname "*.txt"

Regards,
	Klaus.
-- 
Klaus Singvogel
GnuPG-Key-ID: 1024R/5068792D  1994-06-27

[toc] | [prev] | [next] | [standalone]

#223873

Many wonderful comments have already been given.  Here are some nitpicks
that will no longer matter once you stop writing this program in bash,
which is a poor choice for this particular job.

On Sat, Jun 20, 2020 at 12:20:41PM +0200, Albretch Mueller wrote:
> for iZ in ${!_AR_TERMS[@]}; do

The quoting here is incorrect.  It should be:

for iZ in "${!_ar_terms[@]}"; do

You probably got away with it because you haven't used indices that
contain whitespace yet.  (I'm not even sure whether you're using an
indexed array, or an associative array.  Obviously with indexed arrays,
your indices are always integers, so the improper quoting won't be
able to bite you -- but that changes as soon as you use an associative
array where the indices can be arbitrary strings.)

Also, do not use all-caps variable names for regular variables.  All-caps
names are reserved for environment variables and special internal shell
variables.  Use at least one lower case letter in each variable to avoid
collisions with reserved names.

>  find "${_SDIR}" -type f -iregex .*"${_X}" -exec grep -il

Here, the .* (or at least the *) should be quoted too.  Otherwise, the *
is part of a live glob that could potentially match filenames in the
current working directory, which would break the script.

unicorn:~$ x=rc
unicorn:~$ args find . -iregex .*"$x"
47 args: <find> <.> <-iregex> <.acrorc> <.aumixrc> <.avirc> <.bashrc> <.bwaprc> <.cdrrc> <.ceciliarc> <.civclientrc> <.dialogrc> <.doomrc> <.dvdriprc> <.exrc> <.fehrc> <.fvwm2rc> <.gnomerc> <.inputrc> <.ircrc> <.jnewsrc> <.kderc> <.kermrc> <.lessrc> <.lynxrc> <.mangrc> <.mcoprc> <.micqrc> <.mp3capnrc> <.mp3commanderrc> <.muttrc> <.newsrc> <.oldnewsrc> <.pinerc> <.point2playrc> <.ppprc> <.qwebrc> <.reportbugrc> <.rtorrent.rc> <.slrnrc> <.twinrc> <.twmrc> <.vimrc> <.xfmailrc> <.xinitrc> <.xmodmaprc> <.zshrc>

That's definitely NOT what you want.

unicorn:~$ args find . -iregex ".*$x"
4 args: <find> <.> <-iregex> <.*rc>

That's what you want.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web