Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #275359 > unrolled thread

Shell function & variable usage.

Back to article view | Back to linux.debian.user

Contents

  Shell function & variable usage. peter@easthope.ca - 2024-12-07 00:40 +0100
    Re: Shell function & variable usage. Greg Wooledge <greg@wooledge.org> - 2024-12-07 00:50 +0100
      Re: Shell function & variable usage. peter@easthope.ca - 2024-12-07 15:40 +0100
        Re: Shell function & variable usage. Greg Wooledge <greg@wooledge.org> - 2024-12-07 17:10 +0100
          Re: Shell function & variable usage. peter@easthope.ca - 2024-12-08 19:50 +0100
    Re: Shell function & variable usage. eben@gmx.us - 2024-12-07 00:50 +0100
      Re: Shell function & variable usage. eben@gmx.us - 2024-12-07 01:00 +0100
      Re: Shell function & variable usage. Greg Wooledge <greg@wooledge.org> - 2024-12-07 01:00 +0100
    Re: Shell function & variable usage. Karl Vogel <vogelke@pobox.com> - 2024-12-07 12:40 +0100
    Re: Shell function & variable usage. Anssi Saari <anssi.saari@debian-user.mail.kapsi.fi> - 2024-12-11 14:30 +0100

#275359 — Shell function & variable usage.

Hypothetical shell usages (1) & (2).

(1)
Restore() { \
  source=somewhere/Backup/ ;
  echo "source is $source." ;
  destination=elsewhere/workingDirectory ;
  echo "destination is $destination." ;
  rsync \
  --exclude '*.mp3' \
   --exclude '*.mp4' \
  -anuv $source $destination ; }

(2)
exclusions='*.mp3'
exclusions+='*.mp4'
Restore() { \
  source=somewhere/Backup/ ;
  echo "source is $source." ;
  destination=elsewhere/workingDirectory ;
  echo "destination is $destination." ;
  rsync \
  -anuv --exclude=$exclusions $source $destination ; }

In a realistic case, there are more than two exclusion patterns. 
Comments or suggestions about the two possibilities?  Astonishingly 
better ideas?  =8~o

Thanks,                        ... P.

-- 
VoIP:   +1 604 670 0140
work: https://en.wikibooks.org/wiki/User:PeterEasthope

[toc] | [next] | [standalone]

#275360

On Fri, Dec 06, 2024 at 15:11:51 -0700, peter@easthope.ca wrote:
> (2)
> exclusions='*.mp3'
> exclusions+='*.mp4'
> Restore() { \
>   source=somewhere/Backup/ ;
>   echo "source is $source." ;
>   destination=elsewhere/workingDirectory ;
>   echo "destination is $destination." ;
>   rsync \
>   -anuv --exclude=$exclusions $source $destination ; }
> 
> In a realistic case, there are more than two exclusion patterns. 

Store the exclusions in an ARRAY, not in a string.  Then create a
second array which contains the spelled-out --exclude=... options.

#!/bin/bash
exclusions=( '*.mp3' '*.mp4' '*.wav' )
exclude=( "${exclusions[@]/#/--exclude=}" )
source=...
destination=...
rsync -anuv "${exclude[@]}" "$source" "$destination"

[toc] | [prev] | [next] | [standalone]

#275377

    From: Greg Wooledge <greg@wooledge.org>
    Date: Fri, 6 Dec 2024 18:44:04 -0500
> Store the exclusions in an ARRAY, not in a string.  Then create a
> second array which contains the spelled-out --exclude=... options.

Unfamiliar to me & interesting.  What benefits outweigh the additional 
complexity?

Thanks,                        ... P.

-- 
VoIP:   +1 604 670 0140
work: https://en.wikibooks.org/wiki/User:PeterEasthope

[toc] | [prev] | [next] | [standalone]

#275378

On Sat, Dec 07, 2024 at 06:19:20 -0700, peter@easthope.ca wrote:
>     From: Greg Wooledge <greg@wooledge.org>
>     Date: Fri, 6 Dec 2024 18:44:04 -0500
> > Store the exclusions in an ARRAY, not in a string.  Then create a
> > second array which contains the spelled-out --exclude=... options.
> 
> Unfamiliar to me & interesting.  What benefits outweigh the additional 
> complexity?

It's LESS complexity!

Trying to shove a list of data elements into a string is ALWAYS worse
than storing your list of data elements in an array.

Consider what happens when one of your data elements has a space in it.
What are you going to do then?

If you're storing the list in a string with spaces between elements,
then you need to add some sort of quoting or escaping *inside* the
string.

Now it's more complex than if you had simply used the array in the
first place.

Array assignments in bash also allow you to use free form syntax, and
even to add comments.

exclusions=(
    '*.mp3'
    '*.mp4'   # added 2024-12-07 due to ...
    '*[Rr]ed [Hh]ot*'
    '*.wav' '*.snd' '*.au'
)

Now, you may be thinking "Hey, you said you wouldn't need to add quoting
inside the array, but those are quotes!"  The difference between this
and a string is the quotes in the array assignment are parsed by the
shell and removed.

hobbit:~$ exclusions=(
    '*.mp3'
    '*.mp4'   # added 2024-12-07 due to ...
    '*[Rr]ed [Hh]ot*'
    '*.wav' '*.snd' '*.au'
)
hobbit:~$ declare -p exclusions
declare -a exclusions=([0]="*.mp3" [1]="*.mp4" [2]="*[Rr]ed [Hh]ot*" [3]="*.wav" [4]="*.snd" [5]="*.au")

Each array element is ready to go.  No additional parsing is needed.
If you want to do something like "convert this list of globs into a
list of --exclude=GLOB options", it's just one parameter expansion.
I've shown it already, but here it is again:

    "${exclusions[@]//#/--exclude=}"

In a string, the quotes are literally placed inside the string, and
become part of the data payload.  You have to write your *own* parsing
code to separate the data elements and remove the quotes.  The shell
won't be able to do it for you.

hobbit:~$ unset exclusions
hobbit:~$ exclusions='*.mp3 *.mp4 "*[Rr]ed [Hh]ot*" *.wav *.snd *.au'
hobbit:~$ declare -p exclusions
declare -- exclusions="*.mp3 *.mp4 \"*[Rr]ed [Hh]ot*\" *.wav *.snd *.au"

That's so much harder to deal with.  You can't do anything with it until
you parse it back out into a list.  Essentially you'll be converting
the string into an array, and then using the array operations that I've
already shown.

It's so much easier just to use an array in the first place.

[toc] | [prev] | [next] | [standalone]

#275409

    From: Greg Wooledge <greg@wooledge.org>
    Date: Sat, 7 Dec 2024 11:08:54 -0500
>     "${exclusions[@]//#/--exclude=}"

In a shell script; correct?

I'm making a shell function declared in .bashrc.  Numerous syntactical 
variations were unsuccessful.  Maybe a simple escape syntax is needed.

Stretched a test out to this.  it works.

Test() { \
  exclusions=( \
  'file' \
  '*.mp3' \
  '*.mp4' );
  echo "exclusions = ${exclusions[@]}";
  n=${#exclusions[@]};
  i=0;
  while [ "$i" -lt "$n" ]; do
    echo ${exclusions[$i]};
    exclusions[$i]="--exclude=${exclusions[$i]}";
    ((i++));
  done;
  echo "i = $i";
  echo "exclusions = ${exclusions[@]}";
}

When I'm happy with Test, will incorporate into the Backup function.

Thanks for the help,                      ... P.

-- 
VoIP:   +1 604 670 0140
work: https://en.wikibooks.org/wiki/User:PeterEasthope

[toc] | [prev] | [next] | [standalone]

#275362

On 12/6/24 17:11, peter@easthope.ca wrote:

These

>   rsync \
>   --exclude '*.mp3' \
>    --exclude '*.mp4' \

>   rsync \
>   --exclude=$exclusions

are not the same.  In the first one you have "--exclude" for each pattern,
and in the second you have it once.  I suspect the first one is correct, and
for that reason I vote for that one, even those typing "--exclude" several
times gives several opportunities for error.  You could do something
overengineered like defining an array "excludepatterns" for patterns, and
then doing something like

rsync  \
$(for pattern in ${excludepatterns[@]} ; do
  echo -- "--exclude  '$pattern'"
done)

Modulo mistakes, of course.  And make sure it deals with wildcards, quotes,
and spaces appropriately.

[toc] | [prev] | [next] | [standalone]

#275363

On 12/6/24 18:46, eben@gmx.us wrote:
> overengineered

Greg's solution is better.

[toc] | [prev] | [next] | [standalone]

#275364

On Fri, Dec 06, 2024 at 18:46:06 -0500, eben@gmx.us wrote:
> You could do something
> overengineered like defining an array "excludepatterns" for patterns, and
> then doing something like
> 
> rsync  \
> $(for pattern in ${excludepatterns[@]} ; do
>   echo -- "--exclude  '$pattern'"
> done)
> 
> Modulo mistakes, of course.

This definitely has mistakes.  You're expanding each of the patterns
based on the files in the current working directory.

hobbit:~$ exclusions=( '*.mp3' '*.mp4' '*.wav' )
hobbit:~$ echo *.mp3
christmas-peebs.mp3 inmexico.mp3 Kristie_and_John_-_Bobby_McGee.mp3
hobbit:~$ for p in ${exclusions[@]}; do echo -- "--exclude '$p'"; done
-- --exclude 'christmas-peebs.mp3'
-- --exclude 'inmexico.mp3'
-- --exclude 'Kristie_and_John_-_Bobby_McGee.mp3'
-- --exclude '*.mp4'
-- --exclude '*.wav'

That's not what you want.  (You're also writing a literal -- which is
not wanted.)

As I wrote in the previous message, you can generate the second array
using a single command that doesn't perform pathname expansions:

hobbit:~$ opts=( "${exclusions[@]/#/--exclude=}" )
hobbit:~$ declare -p opts
declare -a opts=([0]="--exclude=*.mp3" [1]="--exclude=*.mp4" [2]="--exclude=*.wav")

[toc] | [prev] | [next] | [standalone]

#275374

On Fri 06 Dec 2024 at 18:30:27 (-0500), peter@easthope.ca wrote:
> Hypothetical shell usages (1) & (2).
> 
> (1)
> Restore() { \
>   source=somewhere/Backup/ ;
>   destination=elsewhere/workingDirectory ;
>   rsync \
>   --exclude '*.mp3' \
>   --exclude '*.mp4' \
>   -anuv $source $destination ; }

  If you want to modify exclusions, you have to edit the code.

> (2)
> exclusions='*.mp3'
> exclusions+='*.mp4'
> Restore() { \
>   source=somewhere/Backup/ ;
>   destination=elsewhere/workingDirectory ;
>   rsync \
>   -anuv --exclude=$exclusions $source $destination ; }

  I would put

    *.mp[34]

  in a separate config file.  I usually name those after the script to
  make them easier to find:

    #!/usr/bin/env bash
    export PATH=/usr/local/bin:/bin:/usr/bin
    tag=${0##*/}
    ...

    excfile="/usr/local/etc/${tag}-exclude"
    test -s "$excfile" || { echo die screaming; exit 1; }
    ...

    rsync -anuv --exclude-from="$excfile" "$source" "$destination"

-- 
Karl Vogel                      I don't speak for anyone but myself

COMMENT: That's an amazing deal, would snap it up if I had the money.
REPLY:   You can eat next month.    --Reddit on buying storage, 21 Nov 2024

[toc] | [prev] | [next] | [standalone]

#275505

peter@easthope.ca writes:

> In a realistic case, there are more than two exclusion patterns. 
> Comments or suggestions about the two possibilities?  Astonishingly 
> better ideas?  =8~o

Put the exlucde patterns in a file and use --exclude-from=file? Works
for me, so far with tar and bup, rsync has the same support.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web