Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.misc > #1583 > unrolled thread

calling script function on output of find

Back to article view | Back to comp.os.linux.misc

Contents

  calling script function on output of find John Stumbles <john.stumbles@ntlworld.com> - 2011-07-05 10:01 +0000
    Re: calling script function on output of find Joe Makowiec <makowiec@invalid.invalid> - 2011-07-05 10:20 +0000
      Re: calling script function on output of find John Stumbles <john.stumbles@ntlworld.com> - 2011-07-09 12:04 +0000
    Re: calling script function on output of find dave.gma+news002@googlemail.com.invalid (Dave Gibson) - 2011-07-05 19:17 +0100
      Re: calling script function on output of find unruh <unruh@wormhole.physics.ubc.ca> - 2011-07-05 20:01 +0000
        Re: calling script function on output of find John Stumbles <john.stumbles@ntlworld.com> - 2011-07-09 12:21 +0000
          Re: calling script function on output of find Bill Marcum <bill@lat.localnet> - 2011-07-12 19:43 -0400
            Re: calling script function on output of find John Stumbles <john.stumbles@ntlworld.com> - 2011-07-16 00:26 +0000
      Re: calling script function on output of find John Stumbles <john.stumbles@ntlworld.com> - 2011-07-09 12:48 +0000
        Re: calling script function on output of find dave.gma+news002@googlemail.com.invalid (Dave Gibson) - 2011-07-09 19:52 +0100
    Re: calling script function on output of find Chris Davies <chris-usenet@roaima.co.uk> - 2011-07-07 09:37 +0100
      Re: calling script function on output of find John Stumbles <john.stumbles@ntlworld.com> - 2011-07-08 10:02 +0000
        Re: calling script function on output of find Chris Davies <chris-usenet@roaima.co.uk> - 2011-07-08 16:48 +0100
          Re: calling script function on output of find unruh <unruh@wormhole.physics.ubc.ca> - 2011-07-08 17:46 +0000
            Re: calling script function on output of find Chris Davies <chris-usenet@roaima.co.uk> - 2011-07-08 20:56 +0100
              Re: calling script function on output of find unruh <unruh@wormhole.physics.ubc.ca> - 2011-07-08 21:14 +0000
                Re: calling script function on output of find Chris Davies <chris-usenet@roaima.co.uk> - 2011-07-09 11:17 +0100
          Re: calling script function on output of find John Stumbles <john.stumbles@ntlworld.com> - 2011-07-09 12:51 +0000
            Re: calling script function on output of find Chris Davies <chris-usenet@roaima.co.uk> - 2011-07-10 10:45 +0100
              Re: calling script function on output of find John Stumbles <john.stumbles@ntlworld.com> - 2011-07-10 23:14 +0000

#1583 — calling script function on output of find

I have

for f in `find $SOURCE_DIR -type f -size +0`; do 
    process_file "$f"
done

where process_file is a function within the same bash script.

However if $SOURCE_DIR has spaces in the name 
e.g. "/foo/bar/space in name"
the script tries to process_file on parts of the pathname
i.e.
  process_file /foo/bar/space 
  process_file in
  process_file name

This seems to be a limitation of the "for f in ..." construct and I 
should probably be doing something else - but what?!


-- 
John Stumbles

Pessimists are never disappointed

[toc] | [next] | [standalone]

#1584

On 05 Jul 2011 in comp.os.linux.misc, John Stumbles wrote:

> I have
> 
> for f in `find $SOURCE_DIR -type f -size +0`; do 
>     process_file "$f"
> done
> 
> where process_file is a function within the same bash script.
> 
> However if $SOURCE_DIR has spaces in the name 
> e.g. "/foo/bar/space in name"
> the script tries to process_file on parts of the pathname
> i.e.
>   process_file /foo/bar/space 
>   process_file in
>   process_file name
> 
> This seems to be a limitation of the "for f in ..." construct and I 
> should probably be doing something else - but what?!

Try using the -exec switch to find:

find $SOURCE_DIR -type f -size +0 -exec process_file {} \;

The stuff at the end is necessary; "{}" inserts the filename and "\;" 
terminates the command.

I just tried it (find /home/public/ -name "*.txt" -exec wc {} \;) and 
it worked on directories and files with spaces in the names:

 233  2473 34406 /home/public/Logwatch for makowieccom (Linux).txt
 0  5 33 /home/public/directory with spaces/networkcards.txt
 24  43 319 /home/public/directory with spaces/faxJudysResume.txt

-- 
Joe Makowiec
http://makowiec.org/
Email: http://makowiec.org/contact/?Joe
Usenet Improvement Project: http://twovoyagers.com/improve-usenet.org/

[toc] | [prev] | [next] | [standalone]

#1714

On Tue, 05 Jul 2011 10:20:02 +0000, Joe Makowiec wrote:


> Try using the -exec switch to find:
> 
> find $SOURCE_DIR -type f -size +0 -exec process_file {} \;

Thanks. But unfortunately that doesn't seem to work where the command to 
be -exec-ed is a function within the script, rather than an external 
command that 'find' can find.

I guess I could do some smoke-and-mirrors stuff where calling my script 
with a switch (such as --process_file) makes it execute the required 
function. But that is so much like the crap one used to have to do to 
implement functions in MS-DOS .BATch files it makes me shudder even to 
think of it :-/


-- 
John Stumbles

Fundamentalist agnostic

[toc] | [prev] | [next] | [standalone]

#1604

John Stumbles <john.stumbles@ntlworld.com> wrote:
> I have
> 
> for f in `find $SOURCE_DIR -type f -size +0`; do 
>    process_file "$f"
> done
> 
> where process_file is a function within the same bash script.
> 
> However if $SOURCE_DIR has spaces in the name 
> e.g. "/foo/bar/space in name"
> the script tries to process_file on parts of the pathname
> i.e.
>  process_file /foo/bar/space 
>  process_file in
>  process_file name
> 
> This seems to be a limitation of the "for f in ..." construct and I 

Captured output (`...`) is subject to word splitting.

> should probably be doing something else - but what?!

find "$SOURCE_DIR" -type f -size +0 |
while IFS= read -r f ; do
  process_file "$f"
done

[toc] | [prev] | [next] | [standalone]

#1606

On 2011-07-05, Dave Gibson <dave.gma+news002@googlemail.com.invalid> wrote:
> John Stumbles <john.stumbles@ntlworld.com> wrote:
>> I have
>> 
>> for f in `find $SOURCE_DIR -type f -size +0`; do 
>>    process_file "$f"
>> done
>> 
>> where process_file is a function within the same bash script.
>> 
>> However if $SOURCE_DIR has spaces in the name 
>> e.g. "/foo/bar/space in name"
>> the script tries to process_file on parts of the pathname
>> i.e.
>>  process_file /foo/bar/space 
>>  process_file in
>>  process_file name
>> 
>> This seems to be a limitation of the "for f in ..." construct and I 
>
> Captured output (`...`) is subject to word splitting.
>
>> should probably be doing something else - but what?!
>
> find "$SOURCE_DIR" -type f -size +0 |
> while IFS= read -r f ; do
>   process_file "$f"
> done

or
find "$SOURCE_DIR" -type f -size +0 -print0| xargs -n 1 -0 process_file
(assumeing process_file can only process one at a time. If it can
process a bunch from one argument list, remove the "-n 1"
)

[toc] | [prev] | [next] | [standalone]

#1715

On Tue, 05 Jul 2011 20:01:58 +0000, unruh wrote:

> find "$SOURCE_DIR" -type f -size +0 -print0| xargs -n 1 -0 process_file

That also has the problem that process_file is a function within my 
script and xargs can't find it (i.e. the same problem as with the 
	-exec process_file {} \; 
solution which Joe M proposed, which I just replied to).

-- 
John Stumbles

An atheist is a person with no invisible means of support

[toc] | [prev] | [next] | [standalone]

#1758

On 2011-07-09, John Stumbles <john.stumbles@ntlworld.com> wrote:
> On Tue, 05 Jul 2011 20:01:58 +0000, unruh wrote:
>
>> find "$SOURCE_DIR" -type f -size +0 -print0| xargs -n 1 -0 process_file
>
> That also has the problem that process_file is a function within my 
> script and xargs can't find it (i.e. the same problem as with the 
> 	-exec process_file {} \; 
> solution which Joe M proposed, which I just replied to).
>
Then you should write a script containing that function.


-- 
Inside every older person is a younger person wondering what the hell
happened.

[toc] | [prev] | [next] | [standalone]

#1819

On Tue, 12 Jul 2011 19:43:58 -0400, Bill Marcum wrote:

> Then you should write a script containing that function.

That seems an even more kludgy solution than the one I suggested (and 
implemented just to test it) of rewriting the script so that when it sees 
the switch --process_file it runs process_file() and calling -exec $0 --
process_file. At least that keeps all the code in one file instead of 
splitting it into two!

-- 
John Stumbles

I am neither for nor against apathy

[toc] | [prev] | [next] | [standalone]

#1716

On Tue, 05 Jul 2011 19:17:29 +0100, Dave Gibson wrote:

> find "$SOURCE_DIR" -type f -size +0 | while IFS= read -r f ; do
>   process_file "$f"
> done

OK. That works.
But I don't like cargo-cult coding so let me see if I understand it.

RTFM-ing (man bash) tells me "One line is read ... and the first word is 
assigned to the first name, the second word to the second name, and so 
on..." and "The characters in IFS  are used to split the line into words."

Also "If the value of IFS  is null, no word splitting occurs."

So by setting IFS= (to null) before calling 'read' we should get the 
output of find split line-by-line e.g.
  dir one/file a
  dir one/file b
  dir two/file z
etc
which 'read' then assigns, one at a time (in the while loop), to the 
variable named 'f' which then gets passed (quoted to avoid splitting) to 
'process_file "$f"'.

Neat.

One question: why the -r switch ("Backslash does not act as an escape 
character") to read?

-- 
John Stumbles

Science doesn’t understand everything.
Religion doesn’t understand anything.

[toc] | [prev] | [next] | [standalone]

#1719

John Stumbles <john.stumbles@ntlworld.com> wrote:
> On Tue, 05 Jul 2011 19:17:29 +0100, Dave Gibson wrote:
> 
>> find "$SOURCE_DIR" -type f -size +0 | while IFS= read -r f ; do
>>   process_file "$f"
>> done

> One question: why the -r switch ("Backslash does not act as an escape 
> character") to read?

To ensure that a backslash has the same meaning in read's input as it
does in find's output.

In an empty directory:

  $ touch 'foo\bar'
  $ find . -name 'foo*'
  ./foo\bar
  $ find . -name 'foo*' | { read f ; printf '%s\n' "$f" ; }
  ./foobar
  $ find . -name 'foo*' | { read -r f ; printf '%s\n' "$f" ; }
  ./foo\bar

[toc] | [prev] | [next] | [standalone]

#1655

John Stumbles <john.stumbles@ntlworld.com> wrote:
> This seems to be a limitation of the "for f in ..." construct and I 
> should probably be doing something else - but what?!

The other posters have given correct constructs, but I haven't seen any
of them explain why theirs would work but yours doesn't.

It's all to do with quoting. If you double-quote a "$variable" then the
spaces that it may contain are kept as part of the item. If you don't
quuote $variable then it is subject to splitting on whitespace. Single
quotes around '$variable' shop the string being expanded at all (the
string remains as a literal - in this example being 9 characters long).

    mkdir 'one' 'two' 'one two'
    touch 'one/first' 'two/second' 'one two/others'

    NUMBERS='one two'
    ls $NUMBERS                 # Lists two directories 'one' and 'two'
    ls "$NUMBERS"               # Lists single directory 'one two'
    ls '$NUMBERS'               # Fails as '$NUMBERS' does not exist

Coming back to your specific example, part of the answer to your question
is to quote "$SOURCE_DIR" within your 'find' construct. However,
the 'for' loop will still break when 'find' itself returns pathnames
containing spaces - and this will happen because you have already said
that $SOURCE_DIR does contain spaces. At that point you need to invert the
construct to avoid 'for' having the opportunity to split on whitespace.

Chris

[toc] | [prev] | [next] | [standalone]

#1688

On Thu, 07 Jul 2011 09:37:42 +0100, Chris Davies wrote:

> The other posters have given correct constructs, but I haven't seen any
> of them explain why theirs would work but yours doesn't.
> 
> It's all to do with quoting. If you double-quote a "$variable" then the
> spaces that it may contain are kept as part of the item. If you don't
> quuote $variable then it is subject to splitting on whitespace. Single
> quotes around '$variable' shop the string being expanded at all (the
> string remains as a literal - in this example being 9 characters long).
> 
>     mkdir 'one' 'two' 'one two'
>     touch 'one/first' 'two/second' 'one two/others'
> 
>     NUMBERS='one two'
>     ls $NUMBERS                 # Lists two directories 'one' and 'two'
>     ls "$NUMBERS"               # Lists single directory 'one two' ls
>     '$NUMBERS'               # Fails as '$NUMBERS' does not exist

Nice explanation!

> Coming back to your specific example, part of the answer to your
> question is to quote "$SOURCE_DIR" within your 'find' construct.
> However, the 'for' loop will still break when 'find' itself returns
> pathnames containing spaces - and this will happen because you have
> already said that $SOURCE_DIR does contain spaces. At that point you
> need to invert the construct to avoid 'for' having the opportunity to
> split on whitespace.

I tried quoting "$SOURCE_DIR" and found that the problem was that 'for' 
was splitting on the whitespaces, but I don't get what you mean by 
'invert the construct' - ?

-- 
John Stumbles

There's nowt as queer as folk.
Especially other folk.

[toc] | [prev] | [next] | [standalone]

#1698

On Thu, 07 Jul 2011 09:37:42 +0100, Chris Davies wrote:
> Coming back to your specific example, part of the answer to your
> question is to quote "$SOURCE_DIR" within your 'find' construct.
> However, the 'for' loop will still break when 'find' itself returns
> pathnames containing spaces - and this will happen because you have
> already said that $SOURCE_DIR does contain spaces. At that point you
> need to invert the construct to avoid 'for' having the opportunity to
> split on whitespace.

John Stumbles <john.stumbles@ntlworld.com> wrote:
> I tried quoting "$SOURCE_DIR" and found that the problem was that 'for' 
> was splitting on the whitespaces, but I don't get what you mean by 
> 'invert the construct' - ?

OK. Take your original construct apart:

    for f in `find $SOURCE_DIR -type f -size +0`
    do 
        process_file "$f"
    done

First we run the find command and discover that because $SOURCE_DIR
contains a directory with a space in its name, we crash and burn:
    find $SOURCE_DIR -type f -size +0

So this needs to be replaced with this:
    find "$SOURCE_DIR" -type f -size +0

Now we can interpolate that output into the for loop. However, in doing so
we now realise that every single response will also contain whitespace,
because $SOURCE_DIR contained whitespace. Consider a directory called
'one two' containing the files 'a' and 'b'. The find output will produce
this list:
        one two/a
        one two/b

The result is an attempt to execute this for command, which results in
the shell complaining (if we're lucky!) that neither one nor two exist:
    for f in one two/a one two/b; do process_file "$f"; done

What you really wanted was this, but you can't have it:
    for f in 'one two/a' 'one two/b'; do process_file "$f"; done

So, there is now no sensible option but to turn the construct inside out,
which is why the other posters suggested such things as this:
    find "$SOURCE_DIR" -type f -size +0 | while read f; do process_file "$f"; done


Please note, that if your process_file needs to update variables that
are used later in your script, or if your pathnames contain embedded
newlines and/or leading spaces, this solution WILL NOT WORK. Come back
here for more pathologically correct answers.

Chris

[toc] | [prev] | [next] | [standalone]

#1703

On 2011-07-08, Chris Davies <chris-usenet@roaima.co.uk> wrote:
> On Thu, 07 Jul 2011 09:37:42 +0100, Chris Davies wrote:
>> Coming back to your specific example, part of the answer to your
>> question is to quote "$SOURCE_DIR" within your 'find' construct.
>> However, the 'for' loop will still break when 'find' itself returns
>> pathnames containing spaces - and this will happen because you have
>> already said that $SOURCE_DIR does contain spaces. At that point you
>> need to invert the construct to avoid 'for' having the opportunity to
>> split on whitespace.
>
> John Stumbles <john.stumbles@ntlworld.com> wrote:
>> I tried quoting "$SOURCE_DIR" and found that the problem was that 'for' 
>> was splitting on the whitespaces, but I don't get what you mean by 
>> 'invert the construct' - ?
>
> OK. Take your original construct apart:
>
>     for f in `find $SOURCE_DIR -type f -size +0`
>     do 
>         process_file "$f"
>     done

Try
for i in "`find \"$SOURCE_DIR\" -type f -size +0`"
do
process_file "$i"
done

>
> First we run the find command and discover that because $SOURCE_DIR
> contains a directory with a space in its name, we crash and burn:
>     find $SOURCE_DIR -type f -size +0
>
> So this needs to be replaced with this:
>     find "$SOURCE_DIR" -type f -size +0
>
> Now we can interpolate that output into the for loop. However, in doing so
> we now realise that every single response will also contain whitespace,
> because $SOURCE_DIR contained whitespace. Consider a directory called
> 'one two' containing the files 'a' and 'b'. The find output will produce
> this list:
>         one two/a
>         one two/b
>
> The result is an attempt to execute this for command, which results in
> the shell complaining (if we're lucky!) that neither one nor two exist:
>     for f in one two/a one two/b; do process_file "$f"; done
>
> What you really wanted was this, but you can't have it:
>     for f in 'one two/a' 'one two/b'; do process_file "$f"; done
>
> So, there is now no sensible option but to turn the construct inside out,
> which is why the other posters suggested such things as this:
>     find "$SOURCE_DIR" -type f -size +0 | while read f; do process_file "$f"; done
>
>
> Please note, that if your process_file needs to update variables that
> are used later in your script, or if your pathnames contain embedded
> newlines and/or leading spaces, this solution WILL NOT WORK. Come back
> here for more pathologically correct answers.
>
> Chris

[toc] | [prev] | [next] | [standalone]

#1705

unruh <unruh@wormhole.physics.ubc.ca> wrote:
> for i in "`find \"$SOURCE_DIR\" -type f -size +0`"

At best that provides a single argument to the for loop.
Chris

[toc] | [prev] | [next] | [standalone]

#1707

On 2011-07-08, Chris Davies <chris-usenet@roaima.co.uk> wrote:
> unruh <unruh@wormhole.physics.ubc.ca> wrote:
>> for i in "`find \"$SOURCE_DIR\" -type f -size +0`"
>
> At best that provides a single argument to the for loop.
> Chris

OK, you are right. I did 
for i in "`find \"$S\" -type f`";do echo "$i"; done
a b/2
a b/1

(where S is "a b") and assumed I was getting two arguments to find. But
doing
for i in "`find \"$S\" -type f`";do echo hi; echo "$i"; done
hi
a b/2
a b/1
shows I am only getting one. 

So again, use xargs 
find "$S" -type f -print0|xargs -0 -n 1 echo
a b/2
a b/1

[toc] | [prev] | [next] | [standalone]

#1713

unruh <unruh@wormhole.physics.ubc.ca> wrote:
> find "$S" -type f -print0|xargs -0 -n 1 echo

Yep, that's another way of unravelling the OP's loop  :-)
Chris

[toc] | [prev] | [next] | [standalone]

#1717

On Fri, 08 Jul 2011 16:48:46 +0100, Chris Davies wrote:

> So, there is now no sensible option but to turn the construct inside
> out, which is why the other posters suggested such things as this:
>     find "$SOURCE_DIR" -type f -size +0 | while read f; do process_file
>     "$f"; done

OK, thanks, it was the "turning the construct inside out" I didn't get.
I still don't see that as turning it inside out, but I thought you were 
suggesting some yet-different solution the others hadn't.


> Please note, that if your process_file needs to update variables that
> are used later in your script, or if your pathnames contain embedded
> newlines and/or leading spaces, this solution WILL NOT WORK. Come back
> here for more pathologically correct answers.

Like "use Perl" ;-)

-- 
John Stumbles

Procrastinate now!

[toc] | [prev] | [next] | [standalone]

#1727

>> [Certain edge cases...] WILL NOT WORK. Come back here for more
>> pathologically correct answers.

John Stumbles <john.stumbles@ntlworld.com> wrote:
> Like "use Perl" ;-)

Well, maybe :-)

Actually, it is possible to do most of this using shell. One option is to
pipe the find into a temporary file, then use the while loop to read from
the temporary file. This keeps the while loop in the same shell context
as the remainder of the program, so global variables can be tweaked.

But yes, perl (or ruby, or python, or any other decent programming
language) could be an alternative. The trade off is convenience vs
perfection vs speed of solution. If I'm writing a one-liner I tend
to write quick and dirty. As soon as I realise it's got more general
possibilities I expand it out into a script and (try to) clean up the
rough edges. Or at least ensure it fails cleanly on just cases.

Chris

[toc] | [prev] | [next] | [standalone]

#1738

On Sun, 10 Jul 2011 10:45:36 +0100, Chris Davies wrote:

> But yes, perl (or ruby, or python, or any other decent programming
> language) could be an alternative. The trade off is convenience vs
> perfection vs speed of solution. If I'm writing a one-liner I tend to
> write quick and dirty. As soon as I realise it's got more general
> possibilities I expand it out into a script and (try to) clean up the
> rough edges. Or at least ensure it fails cleanly on just cases.

I stopped doing things like this in Perl a few years back and forced 
myself to learn to do them in shell, just for the educational variety. 
Now I'm so rusty I don't know if I *could* do it in Perl :-(


-- 
John Stumbles

militant pacifist

[toc] | [prev] | [standalone]

Back to top | Article view | comp.os.linux.misc

csiph-web