Re: finding directories based on files within?

From	Martijn Dekker <martijn@inlv.demon.nl>
Newsgroups	comp.unix.shell
Subject	Re: finding directories based on files within?
Date	2021-05-13 04:50 +0200
Message-ID	<ig3igiF70kpU1@mid.individual.net> (permalink)
References	<eli$2105101745@qaz.wtf>

Show all headers | View raw

Op 10-05-21 om 23:13 schreef Eli the Bearded:
> I currently have a simple shell script along the lines of:
> 
> 	find "$1" -type d \
> 		-exec test -f {}/SENTINEL-1 \; \
> 		-exec test -f {}/SENTINEL-2 \; \
> 		-print
> 
> The idea is to list directories that satisfy the condition of having two
> exactly named files within them. This code works, but it's not exactly
> fast (two execs for each matched directory), and it doesn't scale well:

To do that sort of thing well with the standard shell and utilities, 
your only recourse is to make 'find' invoke another shell that does the 
work in a separate script. That script could be in a file or passed to 
the shell via the -c option, as below.

	find "$1" -type d -exec sh -c '
			for dir do
				if test -f "$dir/SENTINEL-1" \
				&& test -f "$dir/SENTINEL-2"
				then
					printf %s\\n "$dir"
				fi
			done
		' dummy {} +

The {} + (instead of {} \;) saves up all the arguments and passes as 
many of them as possible to the sh invocation. This feature is in POSIX 
and is widespread now (although still unfortunately quite broken in 
BusyBox 'find').

The above should work for your stated purpose, but if you want to do 
anything more complicated, you're quickly going to run into limitations. 
For instance, you can't set any variables in the child shell and expect 
to use them outside of it, nor can the separate script use any of your 
shell functions.

These limitations are why I integrated the 'find' utility into the shell 
as one of the modular loop constructs in my shell library, modernish, 
which runs on nearly every POSIX shell and OS (see my .sig for the 
link). As far as I know, this is a unique feature that does not exist 
anywhere else. In modernish, you can do:

. modernish
use var/loop/find
found=0
LOOP find dir in "$1" -type d
DO
	if test -f "$dir/SENTINEL-1" \
	&& test -f "$dir/SENTINEL-2"
	then
		printf 'Found: %s\n' "$dir"
		found=$((found+1))
	fi
DONE
printf 'Found %d matching directories\n' "$found"

The loop construct invokes 'find' in the background and feeds the 
results into the shell in the specified variable (in this case, dir). 
This gives you robust processing for free; even newlines in file names 
are no problem (provided you quote your variables).

The counter in the example is just to demonstrate that variables survive 
the loop as you would expect. The loop body runs in the main shell 
environment and can do anything you want without limitations. Internally 
it's just a 'while read && eval; do ... done' loop, connected to a file 
descriptor that maintains a pipe to the 'find' utility. So things like 
'continue' and 'break' also work as expected; the pipe is automatically 
broken if the loop is exited for any reason, terminating the 'find' 
background process. If you're curious about how it works, just look at 
the library code -- it's well commented.

If you'll indulge me a little more, below is functionally identical 
example that uses a few more shell language modernisation features in 
modernish (see notes below):

. modernish
use safe  # remove most quoting headaches
use var/loop/find
dirsfound=0
LOOP find dir in $1 -type d
DO
	if is present $dir/SENTINEL-1 \
	&& is present $dir/SENTINEL-2
	then
		putln "Found: $dir"
		let dirsfound+=1
	fi
DONE
putln "Found $dirsfound directories matching criteria"

Notes:
- The safe mode turns off global split/glob to remove most quoting 
headaches. With split and glob gone, only empty removal (meaning: an 
unquoted empty variable is removed and will not become an empty 
argument) remains as a potential issue.
- The is() function is part of a more readable alternative to the 
horrible 'test' botch of a command that, unlike 'test', deals correctly 
with empty removal. Use 'is present' to see if there's anything there, 
'is reg' to test for a regular file, 'is -L reg' to test for a regular 
file or symlink to a regular file.
- putln() avoids echo's portability pitfalls without needing to resort 
to printf. Each argument is printed on a separate line. There's also 
put() which prints space-separated arguments without a newline.
- the 'let' arithmetic command is provided as a shell function on any 
shell that doesn't have a bash/ksh/zsh-style 'let' command built in.

-- 
||    modernish -- harness the shell
||    https://github.com/modernish/modernish
||
||    KornShell lives!
||    https://github.com/ksh93/ksh

Thread

finding directories based on files within? Eli the Bearded <*@eli.users.panix.com> - 2021-05-10 22:13 +0000
  Re: finding directories based on files within? Spiros Bousbouras <spibou@gmail.com> - 2021-05-13 02:38 +0000
  Re: finding directories based on files within? Martijn Dekker <martijn@inlv.demon.nl> - 2021-05-13 04:50 +0200
  Re: finding directories based on files within? Ed Morton <mortonspam@gmail.com> - 2021-05-13 07:20 -0500
    Re: finding directories based on files within? Ed Morton <mortonspam@gmail.com> - 2021-05-13 09:04 -0500
  Re: finding directories based on files within? Axel Reichert <mail@axel-reichert.de> - 2021-05-13 14:56 +0200
    Re: finding directories based on files within? Ed Morton <mortonspam@gmail.com> - 2021-05-13 08:44 -0500
      Re: finding directories based on files within? Axel Reichert <mail@axel-reichert.de> - 2021-05-13 21:09 +0200
  Re: finding directories based on files within? merlyn@stonehenge.com (Randal L. Schwartz) - 2021-05-13 07:31 -0700
    Re: finding directories based on files within? Eli the Bearded <*@eli.users.panix.com> - 2021-05-13 17:11 +0000
    Re: finding directories based on files within? Christian Weisgerber <naddy@mips.inka.de> - 2021-05-13 22:04 +0000
      Re: finding directories based on files within? merlyn@stonehenge.com (Randal L. Schwartz) - 2021-05-15 06:45 -0700
      find2perl (Was: finding directories based on files within?) gazelle@shell.xmission.com (Kenny McCormack) - 2021-05-15 17:32 +0000
        Re: find2perl Dan Espen <dan1espen@gmail.com> - 2021-05-15 13:46 -0400
        Re: find2perl (Was: finding directories based on files within?) Christian Weisgerber <naddy@mips.inka.de> - 2021-05-15 20:17 +0000
          Re: find2perl (Was: finding directories based on files within?) gazelle@shell.xmission.com (Kenny McCormack) - 2021-05-16 13:06 +0000
        Re: find2perl merlyn@stonehenge.com (Randal L. Schwartz) - 2021-05-15 15:34 -0700

csiph-web