Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #227892 > unrolled thread

rsync --delete

Back to article view | Back to linux.debian.user

Contents

  rsync --delete Mike McClain <mike.junk.46@att.net> - 2020-10-17 00:30 +0200
    Re: rsync --delete Klaus Singvogel <deb-user-ml@singvogel.net> - 2020-10-17 00:40 +0200
      Re: rsync --delete Greg Wooledge <wooledg@eeg.ccf.org> - 2020-10-19 13:40 +0200
    Re: rsync --delete ellanios82 <ellanios82@gmail.com> - 2020-10-17 00:40 +0200
    Re: rsync --delete Charles Curley <charlescurley@charlescurley.com> - 2020-10-17 04:10 +0200
    Re: rsync --delete Will Mengarini <seldon@eskimo.com> - 2020-10-17 08:40 +0200
      Re: rsync --delete Greg Wooledge <wooledg@eeg.ccf.org> - 2020-10-19 13:50 +0200
    Re: rsync --delete <tomas@tuxteam.de> - 2020-10-17 10:40 +0200
      Re: rsync --delete David <bouncingcats@gmail.com> - 2020-10-17 14:20 +0200
        Re: rsync --delete <tomas@tuxteam.de> - 2020-10-17 15:30 +0200
      Re: rsync --delete Mike McClain <mike.junk.46@att.net> - 2020-10-17 20:30 +0200
        Re: rsync --delete Greg Wooledge <wooledg@eeg.ccf.org> - 2020-10-19 13:50 +0200
      Re: rsync --delete Andrei POPESCU <andreimpopescu@gmail.com> - 2020-10-18 12:50 +0200
    Re: rsync --delete Andy Smith <andy@strugglers.net> - 2020-10-17 17:30 +0200
    Re: rsync --delete David Christensen <dpchrist@holgerdanske.com> - 2020-10-18 00:10 +0200
      Re: rsync --delete Mike McClain <mike.junk.46@att.net> - 2020-10-19 01:40 +0200
      Re: rsync --delete Greg Wooledge <wooledg@eeg.ccf.org> - 2020-10-19 14:00 +0200
        Re: rsync --delete Mike McClain <mike.junk.46@att.net> - 2020-10-19 21:10 +0200
          Re: rsync --delete Greg Wooledge <wooledg@eeg.ccf.org> - 2020-10-19 21:20 +0200
            Re: rsync --delete Tixy <tixy@yxit.co.uk> - 2020-10-19 23:20 +0200
              Re: rsync --delete David Christensen <dpchrist@holgerdanske.com> - 2020-10-20 05:10 +0200
                Re: rsync --delete Tixy <tixy@yxit.co.uk> - 2020-10-20 09:30 +0200
                  Re: rsync --delete David <bouncingcats@gmail.com> - 2020-10-20 10:00 +0200
                    Re: rsync --delete Tixy <tixy@yxit.co.uk> - 2020-10-20 10:20 +0200
                Re: rsync --delete Andrei POPESCU <andreimpopescu@gmail.com> - 2020-10-20 11:10 +0200
          Re: rsync --delete David <bouncingcats@gmail.com> - 2020-10-19 23:20 +0200
        Re: rsync --delete David Christensen <dpchrist@holgerdanske.com> - 2020-10-20 05:20 +0200
          Re: rsync --delete David <bouncingcats@gmail.com> - 2020-10-20 06:10 +0200
            Re: rsync --delete Nicolas George <george@nsup.org> - 2020-10-20 12:10 +0200
          Re: rsync --delete Greg Wooledge <wooledg@eeg.ccf.org> - 2020-10-20 13:50 +0200
            Re: rsync --delete The Wanderer <wanderer@fastmail.fm> - 2020-10-20 14:50 +0200
              Re: rsync --delete Greg Wooledge <wooledg@eeg.ccf.org> - 2020-10-20 15:10 +0200
                Re: rsync --delete The Wanderer <wanderer@fastmail.fm> - 2020-10-20 15:50 +0200
                  Re: rsync --delete Greg Wooledge <wooledg@eeg.ccf.org> - 2020-10-20 16:10 +0200
      define (or translate, or substitute for) "interpolate": Re: rsync --delete rhkramer@gmail.com - 2020-10-19 14:30 +0200
        Re: define (or translate, or substitute for) "interpolate": Re: rsync --delete rhkramer@gmail.com - 2020-10-19 14:50 +0200
        Re: define (or translate, or substitute for) "interpolate": Re:  rsync --delete <tomas@tuxteam.de> - 2020-10-19 15:00 +0200
          Re: define (or translate, or substitute for) "interpolate": Re: rsync --delete rhkramer@gmail.com - 2020-10-20 02:10 +0200
            Re: define (or translate, or substitute for) "interpolate": Re: rsync  --delete David Christensen <dpchrist@holgerdanske.com> - 2020-10-20 05:50 +0200
    Re: rsync --delete Greg Wooledge <wooledg@eeg.ccf.org> - 2020-10-19 14:10 +0200

Page 1 of 2  [1] 2  Next page →

#227892 — rsync --delete

I've been using rsync to backup to a flash drive but it's not
performing exactly as I expected.

The man page says:
    --delete                delete extraneous files from dest dirs
A section of the backup script is so:
Params=(-a --inplace --delete);
Flash=/sda/rpi4b
cd /home/mike
[ ! -d $Flash/mike ] && mkdir $Flash/mike;

#   exclude compressed files and the contents of most of the .* directories
/mc/bin/mk_rsync_exclude.sh
echo /usr/bin/rsync $Params --exclude-from=/home/mike/.rsync_exclude . $Flash/mike
/usr/bin/rsync $Params --exclude-from=/home/mike/.rsync_exclude . $Flash/mike ||
    echo rsync $Params --exclude-from=/home/mike/.rsync_exclude . $Flash/mike    Failed $? ;

    If I delete a file from my home directory then backup over last
week's copy the deleted file stays in the backup directory and these
build up over time.
    Am I misusing rsync or am I just not understanding how it works?

Thanks,
Mike
--
"First say to yourself what you would be;
    and then do what you have to do."
    - Epictetus

[toc] | [next] | [standalone]

#227893

Mike McClain wrote:

> A section of the backup script is so:
> Params=(-a --inplace --delete);
[...]

Use instead:
Params=-a --inplace --delete

Regards,
	Klaus.
-- 
Klaus Singvogel
GnuPG-Key-ID: 1024R/5068792D  1994-06-27

[toc] | [prev] | [next] | [standalone]

#227962

On Sat, Oct 17, 2020 at 12:30:50AM +0200, Klaus Singvogel wrote:
> Mike McClain wrote:
> 
> > A section of the backup script is so:
> > Params=(-a --inplace --delete);
> [...]
> 
> Use instead:
> Params=-a --inplace --delete

Incorrect.  This command will generate an error, as it should.

Mike's command creates an array variable containing a list of 3 elements.

[toc] | [prev] | [next] | [standalone]

#227894

[Multipart message — attachments visible in raw view] — view raw

 - as stumbling newby , hesitate to express opinion

: maybe something like :

<rsync -av --delete-after /home/mike /dev/sdc>

 [ or what ever the destination is categorized]

On Sat, Oct 17, 2020 at 1:27 AM Mike McClain <mike.junk.46@att.net> wrote:

> I've been using rsync to backup to a flash drive but it's not
> performing exactly as I expected.
>
> The man page says:
>     --delete                delete extraneous files from dest dirs
> A section of the backup script is so:
> Params=(-a --inplace --delete);
> Flash=/sda/rpi4b
> cd /home/mike
> [ ! -d $Flash/mike ] && mkdir $Flash/mike;
>
> #   exclude compressed files and the contents of most of the .* directories
> /mc/bin/mk_rsync_exclude.sh
> echo /usr/bin/rsync $Params --exclude-from=/home/mike/.rsync_exclude .
> $Flash/mike
> /usr/bin/rsync $Params --exclude-from=/home/mike/.rsync_exclude .
> $Flash/mike ||
>     echo rsync $Params --exclude-from=/home/mike/.rsync_exclude .
> $Flash/mike    Failed $? ;
>
>     If I delete a file from my home directory then backup over last
> week's copy the deleted file stays in the backup directory and these
> build up over time.
>     Am I misusing rsync or am I just not understanding how it works?
>
> Thanks,
> Mike
> --
> "First say to yourself what you would be;
>     and then do what you have to do."
>     - Epictetus
>
>

[toc] | [prev] | [next] | [standalone]

#227895

On Fri, 16 Oct 2020 17:09:42 -0500
Mike McClain <mike.junk.46@att.net> wrote:

> I've been using rsync to backup to a flash drive but it's not
> performing exactly as I expected.

You might look into rsnapshot.

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

[toc] | [prev] | [next] | [standalone]

#227899

* Mike McClain <mike.junk.46@att.net> [20-10/16=Fr 17:09 -0500]:
> [...] A section of the backup script is so:
> Params=(-a --inplace --delete);
> [...] 
> echo /usr/bin/rsync $Params --exclude-from=/home/mike/.rsync_exclude . $Flash/mike

Try this to be sure your shell is doing what you think:
debian/pts/14 bash ~ 23:20 0$a=(x y z)
debian/pts/14 bash ~ 23:20 0$echo $a
x
debian/pts/14 bash ~ 23:20 0$echo "$a"
x
debian/pts/14 bash ~ 23:21 0$echo "${a[@]}"
x y z
debian/pts/14 bash ~ 23:21 0$a=x y z # Suggested in another reply
bash: y: command not found
debian/pts/14 bash ~ 23:21 127$echo "$a"
x
debian/pts/14 bash ~ 23:25 0$a="x y z"
debian/pts/14 bash ~ 23:25 0$echo $a
x y z
debian/pts/14 bash ~ 23:26 0$echo "$a" # preferred
x y z
debian/pts/14 bash ~ 23:26 0$# When $a is embedded, quote *outer* string:
debian/pts/14 bash ~ 23:27 0$echo "foo $a bar"
foo x y z bar
debian/pts/14 bash ~ 23:27 0$

[toc] | [prev] | [next] | [standalone]

#227963

On Fri, Oct 16, 2020 at 11:29:34PM -0700, Will Mengarini wrote:
> debian/pts/14 bash ~ 23:25 0$a="x y z"
> debian/pts/14 bash ~ 23:25 0$echo $a
> x y z
> debian/pts/14 bash ~ 23:26 0$echo "$a" # preferred
> x y z
> debian/pts/14 bash ~ 23:26 0$# When $a is embedded, quote *outer* string:
> debian/pts/14 bash ~ 23:27 0$echo "foo $a bar"
> foo x y z bar
> debian/pts/14 bash ~ 23:27 0$

You're so close here, but you drew the wrong conclusion.

When creating an array variable that contains a list of arguments,
you want each argument to be passed separately.  That's the entire
purpose and point of the array variable.

unicorn:~$ args=(-v CFLAGS="-g -O" -q)
unicorn:~$ printf '{%s} ' "${args[@]}"; echo
{-v} {CFLAGS=-g -O} {-q} 

Use an array variable, not a string variable, to hold a list of
arguments.  Use the quoted [@] expansion to expand the array variable.

args=(... your arguments here ...)
somecmd "${args[@]}"

Any other usage is wrong, and will cause bugs eventually.  See also
<https://mywiki.wooledge.org/BashFAQ/050>.

[toc] | [prev] | [next] | [standalone]

#227903

[Multipart message — attachments visible in raw view] — view raw

On Fri, Oct 16, 2020 at 05:09:42PM -0500, Mike McClain wrote:
> I've been using rsync to backup to a flash drive but it's not
> performing exactly as I expected.

I think Will nailed it. Your problem is not an rsync problem,
but a shell (presumably bash) problem:

[...]

> A section of the backup script is so:
> Params=(-a --inplace --delete);

The above is setting a shell array (I guess it is a bashism
(i.e. a bash specific idiom), but am too lazy to look up now).

[...]

> echo /usr/bin/rsync $Params --exclude-from=/home/mike/.rsync_exclude . $Flash/mike
> /usr/bin/rsync $Params --exclude-from=/home/mike/.rsync_exclude . $Flash/mike ||
>     echo rsync $Params --exclude-from=/home/mike/.rsync_exclude . $Flash/mike    Failed $? ;

And the above expands $Params to just the first element of the
array (i.e. the "-a"). That's how bash arrays work. This is
unfortunate, but that's how things are.

Actually, the first of both lines above is an "echo": that's for you
to see the actual rsync command as it's going to happen. If you look
at your log file, you'll see that the --inplace and --delete are
missing.

If you want to have the whole array (space-separated) expanded, you'll
have to do ${Params[@]}, as Will notes.

Or -- much better -- don't use arrays here. Whoever wrote that script
comes from Virtual Basic or Java or something similar. Arrays in shells
may have their places, but this ain't one of those.

Simply do:

  Params="-a --inplace --delete"

then

  /usr/bin/rsync $Params [...]

There is one thing I still don't understand about this script. Why does
it invoke /usr/bin/rsync? Does the script writer know better where the
"right" rsync lives? Or the sysadmin/user, who is in control of $PATH?

Cheers
 - t

[toc] | [prev] | [next] | [standalone]

#227904

On Sat, 17 Oct 2020 at 19:30, <tomas@tuxteam.de> wrote:
> On Fri, Oct 16, 2020 at 05:09:42PM -0500, Mike McClain wrote:

> > A section of the backup script is so:
> > Params=(-a --inplace --delete);

Using an array here is actually best practice. It will handle most
requirements in a less fragile manner than using a string.

> If you want to have the whole array (space-separated) expanded, you'll
> have to do ${Params[@]}, as Will notes.

Correct. Except it should be quoted:
"${Params[@]}"               see [1] below

> Or -- much better -- don't use arrays here. Whoever wrote that script
> comes from Virtual Basic or Java or something similar. Arrays in shells
> may have their places, but this ain't one of those.

Actually, it is an ideal place to use an array. Whoever wrote
that script shows some knowledge of best practice.

The thorough explanation is found here:
http://mywiki.wooledge.org/BashGuide/Arrays

To take a selection of relevant quotes from that page:
"""
Strings are without a doubt the most used parameter type.
But they are also the most misused parameter type.
It is important to remember that a string holds just one element.
[...]
when you put multiple items in a single string, these multiple
items must be somehow delimited from each other.
[...]
The only safe way to represent multiple string elements
in Bash is through the use of arrays.
"""

It's obvious that in the OP example, the option arguments
are multiple, individual elements.

If they are all smashed together into a string, there's an assumption
that they are separated by whitespace, and that the shell's
whitespace processing will magically separate them.
Well, sometimes that works, and sometimes it doesn't.

For example, it won't work with the rsync option
--exclude=PATTERN
if the desired PATTERN contains whitespace.

Using an array is capable of preserving individual
arguments, without relying on whitespace to do so.

> Simply do:
>   Params="-a --inplace --delete"
> then
>   /usr/bin/rsync $Params [...]

[1] Testing that code at shellcheck.net:
  #!/bin/bash
  Params="-a --inplace --delete"; echo $Params

gives the recommendation regarding $Params:
"Double quote to prevent globbing and word splitting".

Explained in detail there, or at:
http://mywiki.wooledge.org/BashGuide/Practices#Quoting

And here:
http://mywiki.wooledge.org/BashGuide/Parameters
"""
You should always keep parameter expansions properly quoted.
The only good PE, is a quoted PE.
"""

This advice is given by shell-scripting experts because
it handles almost all of the required cases, instead of
just most of them, fingers crossed.
A lot of suboptimal advice on shell scripting bounces
around the net, even amongst smart developers.
The people who are dedicated, expert shell scripters wrote
shellcheck, and write the best guides that I have linked.
I am just reproducing the advice that they offer, in the hope
that it helps people. Of course there are various ways to do
things, and some are easier and less fragile than others,
but if you spend time and take an interest in their
wisdom and experience, this is what they will recommend.

The familiar construct "$@" is an array, widely used, to keep
all the command line arguments separate, even if they
contain whitespace. Arrays like "${Params[@]}" provide
exactly the same benefits.

[toc] | [prev] | [next] | [standalone]

#227907

[Multipart message — attachments visible in raw view] — view raw

On Sat, Oct 17, 2020 at 11:18:57PM +1100, David wrote:
> On Sat, 17 Oct 2020 at 19:30, <tomas@tuxteam.de> wrote:

[...]

> > Or -- much better -- don't use arrays here [...]

> Actually, it is an ideal place to use an array. Whoever wrote
> that script shows some knowledge of best practice.

I still disagree. But I haven't got the time to discuss it
thoroughly. So let's agree to differ :)

Cheers
 - t

[toc] | [prev] | [next] | [standalone]

#227917

On Sat, Oct 17, 2020 at 10:30:04AM +0200, tomas@tuxteam.de wrote:
> On Fri, Oct 16, 2020 at 05:09:42PM -0500, Mike McClain wrote:
> > I've been using rsync to backup to a flash drive but it's not
> > performing exactly as I expected.
>
> I think Will nailed it. Your problem is not an rsync problem,
> but a shell (presumably bash) problem:
>
> Simply do:
>   Params="-a --inplace --delete"
>
> then
>   /usr/bin/rsync $Params [...]
>
> There is one thing I still don't understand about this script. Why does
> it invoke /usr/bin/rsync? Does the script writer know better where the
> "right" rsync lives? Or the sysadmin/user, who is in control of $PATH?
>
> Cheers
>  - t

Tom & Will,
You hit right on the head.
I realized it when seeing Klaus post "Params=-a --inplace --delete".

I know better but write bash scripts so seldom that I forget the
intricacies and switching back and forth between bash, perl and ruby
fogs my mind.

As for your last question, the script is called from cron and I'm
never sure whether cron is going to be able to find things so have
just gotten into the habit of putting the path in.

Thanks for the help,
Mike
--
If a Communist mole got elected as President of the United States,
    how would he act?    -   MM

[toc] | [prev] | [next] | [standalone]

#227964

On Sat, Oct 17, 2020 at 12:58:04PM -0500, Mike McClain wrote:
> As for your last question, the script is called from cron and I'm
> never sure whether cron is going to be able to find things so have
> just gotten into the habit of putting the path in.

The line that actually appears *in* the crontab is interpreted by sh,
so you should avoid bashisms there.  You also need to escape % signs.

Your script, however, is executed according to the shebang at the top
of it.  So, if you use bashisms in your script, you should have a
#!/bin/bash shebang on it (assuming the script only needs to run on
Debian systems).

[toc] | [prev] | [next] | [standalone]

#227940

[Multipart message — attachments visible in raw view] — view raw

On Sb, 17 oct 20, 10:30:04, tomas@tuxteam.de wrote:
> 
> There is one thing I still don't understand about this script. Why does
> it invoke /usr/bin/rsync? Does the script writer know better where the
> "right" rsync lives? Or the sysadmin/user, who is in control of $PATH?

On the other hand it prevents unexpected behaviour due to stuff in paths 
that have precedence over /usr/bin. Besides the environment the script 
runs in might not even have a properly configured $PATH.

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser

[toc] | [prev] | [next] | [standalone]

#227913

Hi Mike,

On Fri, Oct 16, 2020 at 05:09:42PM -0500, Mike McClain wrote:
> A section of the backup script is so:
> Params=(-a --inplace --delete);
> Flash=/sda/rpi4b
> cd /home/mike
> [ ! -d $Flash/mike ] && mkdir $Flash/mike;
> 
> #   exclude compressed files and the contents of most of the .* directories
> /mc/bin/mk_rsync_exclude.sh
> echo /usr/bin/rsync $Params --exclude-from=/home/mike/.rsync_exclude . $Flash/mike

shellcheck has this to say:

$ shellcheck ./foo.sh 

In ./foo.sh line 6:
echo /usr/bin/rsync $Params --exclude-from=/home/mike/.rsync_exclude . $Flash/mike
                    ^-- SC2128: Expanding an array without an index only gives the first element.

It's worth using shellcheck.

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]

#227926

On 2020-10-16 15:09, Mike McClain wrote:
> I've been using rsync to backup to a flash drive but it's not
> performing exactly as I expected.
> 
> The man page says:
>      --delete                delete extraneous files from dest dirs
> A section of the backup script is so:
> Params=(-a --inplace --delete);

I try to use all lower case letters for variable names and all upper 
case letters for constants.


'$Params' -- Variable, function, media, source, destination, etc., names 
are critical to understanding.  I don't like 'Params' -- it's too 
generic.  I would prefer 'rsync_opts'.


I use braces whenever evaluating a variable -- '${Params}'.  I forget 
the details why, but I do recall that this practice is important.


I don't use Bourne arrays, and I barely understand how the shell 
interpolates lists and preserves items containing whitespace.  When I 
can't figure it out, I switch to Perl.


-a -- I use this option for backups.


--inplace -- If your backup media cannot hold the last backup with 
enough room for the next backup, get bigger backup media.


--delete -- I use this option for backups.


Additional options I use in my rsync(1) backup script:

--one-file-system -- I use this option for backups.  The bottom-level 
backup script is driven by a higher level script that backs up multiple 
filesystems on multiple hosts via configuration files.

-e ${SSH} -- where:

     SSH=/usr/bin/ssh


(To be pendantic, I should put single parentheses around the RHS?)


When doing rsync(1) backups over SSH, I seem to recall that rsync(1) 
will use whatever login shell is specified for that account.  Some of my 
machines are FreeBSD, and the root login shell is tcsh(1).  Strange 
things were happening before I discovered the '-e' / '--rsh' option.


> Flash=/sda/rpi4b

Mixed case variable name -- as above.


Flash -- I would prefer 'backup_media'.


Is /sda the mount point for your backup media?  If so, that is confusing 
-- 'sda' implies '/dev/sda', which should be your system drive (e.g. 
root).  I would label the backup filesystem 'backup-rpi4b' and mount it 
at '/mnt/backup-rpi4b' or '/media/backup-rpi4b' (your desktop might be 
able to do this for you).


> cd /home/mike

I prefer scripts that I can run from anywhere.  The script should know 
where to do its work, either from built-in variables, environment 
variables, resource script, configuration file, command-line arguments/ 
options, etc..  Sophisticated scripts can draw this information from 
multiple resources, with some hierarchy of precedence.  (I would switch 
to Perl if I wanted to get fancy with options.)


If the script must change the working directory, I would display that -- 
'set -x', 'cd ...', and 'set +x'.


> [ ! -d $Flash/mike ] && mkdir $Flash/mike;

I would set a variable for the destination directory.  Something like:

     dst="${Flash}/mike"


I would do an old-school 'if' block and display that a directory is 
being created -- 'set -x', 'mkdir ...', 'set +x'.


> #   exclude compressed files and the contents of most of the .* directories
> /mc/bin/mk_rsync_exclude.sh

What is /mc?


mk_rsync_exclude.sh creates '.rsync_exclude' in the current working 
directory?


A problem with dynamically generated code/ arguments/ options is: how do 
you know what code was actually run in the past?.  Now you need a log 
file.  I prefer to use a static configuration files and static scripts, 
and check everything into a version control system.  My previous backup 
scripts generated log files, but my current rewrite does not have that 
feature (yet).


I prefer to keep my backup scripts, include files, etc., outside of the 
backup source and destination paths (except when backing up the 
filesystem that contains the backup stuff).  This reduces confusion, and 
might prevent a chicken-and-egg situation.


> echo /usr/bin/rsync $Params --exclude-from=/home/mike/.rsync_exclude . $Flash/mike
> /usr/bin/rsync $Params --exclude-from=/home/mike/.rsync_exclude . $Flash/mike ||
>      echo rsync $Params --exclude-from=/home/mike/.rsync_exclude . $Flash/mike    Failed $? ;


You cut and pasted the following code three times:

    /usr/bin/rsync $Params --exclude-from=/home/mike/.rsync_exclude . 
$Flash/mike


DRY:

     https://en.wikipedia.org/wiki/Don%27t_repeat_yourself


I prefer 'set -x', 'command ...', and 'set +x' when I want to see what 
the shell is actually doing (which might not be the same output as 'echo 
...').


I use 'set -e' at the top of my scripts so that the shell will stop and 
display an error message if a script command fails.


/usr/bin/rsync -- I also use absolute paths for tools.  But, I put them 
into upper-case variables at the top of my script.


--exclude-from -- It is too easy to screw up exclude specifications and 
exclude a file you need.  Therefore, I backup entire filesystems.


When invoking rsync(1), I make sure that SRC and DEST are directories, 
that their paths are absolute, and that their paths end with '/'.  This 
prevents confusion and works as I expect.


>      If I delete a file from my home directory then backup over last
> week's copy the deleted file stays in the backup directory and these
> build up over time.

That is another good reason not to use 'exclude' options.


>      Am I misusing rsync or am I just not understanding how it works?

rsync(1) is a very flexible tool.  I use only a small subset of its 
features, and I do so in a consistent manner.  Thus, I can obtain the 
results I need and I have confidence that they are correct.


David

[toc] | [prev] | [next] | [standalone]

#227952

On Sat, Oct 17, 2020 at 03:01:13PM -0700, David Christensen wrote:
>
> Is /sda the mount point for your backup media?  If so, that is confusing --
> 'sda' implies '/dev/sda', which should be your system drive (e.g. root).  I
> would label the backup filesystem 'backup-rpi4b' and mount it at
> '/mnt/backup-rpi4b' or '/media/backup-rpi4b' (your desktop might be able to
> do this for you).

I'm a lousy/lazy typist so mount a USB flash drive at /dev/sda1 on /sda.

> If the script must change the working directory, I would display that --
> 'set -x', 'cd ...', and 'set +x'.

I did adopt this suggestion.

> I would do an old-school 'if' block and display that a directory is being
> created -- 'set -x', 'mkdir ...', 'set +x'.


> What is /mc?

/mc is simply a directory I put docs/scripts I create and/ or collect
that are not part of any installation.
There is an /mc/docs/, an /mc/bin/ and a couple of others, sometimes.

> mk_rsync_exclude.sh creates '.rsync_exclude' in the current working
> directory?

Yes.

> > echo /usr/bin/rsync $Params --exclude-from=/home/mike/.rsync_exclude . $Flash/mike
> > /usr/bin/rsync $Params --exclude-from=/home/mike/.rsync_exclude . $Flash/mike ||
> >      echo rsync $Params --exclude-from=/home/mike/.rsync_exclude . $Flash/mike    Failed $? ;

> You cut and pasted the following code three times:
>
>    /usr/bin/rsync $Params --exclude-from=/home/mike/.rsync_exclude . $Flash/mike
>
> DRY:     https://en.wikipedia.org/wiki/Don%27t_repeat_yourself

I've seen this just haven't it ingrained yet.

> I prefer 'set -x', 'command ...', and 'set +x' when I want to see what the
> shell is actually doing (which might not be the same output as 'echo ..').

and this one.

>
> I use 'set -e' at the top of my scripts so that the shell will stop and
> display an error message if a script command fails.
>
>
> /usr/bin/rsync -- I also use absolute paths for tools.  But, I put them into
> upper-case variables at the top of my script.
>
> --exclude-from -- It is too easy to screw up exclude specifications and
> exclude a file you need.  Therefore, I backup entire filesystems.

We have different needs.

> When invoking rsync(1), I make sure that SRC and DEST are directories, that
> their paths are absolute, and that their paths end with '/'.  This prevents
> confusion and works as I expect.

> David

I've taken several of your suggestions.
Thanks for the feedback.
Be well,
Mike
--
    If everything seems to be going well, you have obviously overlooked something........

[toc] | [prev] | [next] | [standalone]

#227967

On Sat, Oct 17, 2020 at 03:01:13PM -0700, David Christensen wrote:
> I try to use all lower case letters for variable names and all upper case
> letters for constants.

ALL_UPPER_CASE is reserved for internal shell variables, and environment
variables.

If you abuse it for "constants" as well, on your head be it.  You must
take care to ensure that your ALL_CAPS variable does not collide with
any internal shell variables, or environment variables.

> I use braces whenever evaluating a variable -- '${Params}'.  I forget the
> details why, but I do recall that this practice is important.

It isn't.  It's entirely stylistic.

> I don't use Bourne arrays, and I barely understand how the shell

Bourne shell does not have arrays.  Neither does POSIX shell.

> interpolates lists and preserves items containing whitespace.  When I can't
> figure it out, I switch to Perl.

OK.

> -e ${SSH} -- where:

The stylistic curly braces are NOT a substitute for proper quoting.

>     SSH=/usr/bin/ssh
> 
> 
> (To be pendantic, I should put single parentheses around the RHS?)

No.  foo=(...) creates an array variable, and you are not using this
variable in an array manner.

> > cd /home/mike
> 
> I prefer scripts that I can run from anywhere.

... hence, the cd inside the script.  Doesn't matter where you call it
from, as long as the script cd's to the right location to do its work,
**AND VERIFIES THAT THE cd ACTUALLY WORKED**.

> If the script must change the working directory, I would display that --
> 'set -x', 'cd ...', and 'set +x'.

For god's sake, why?

> I use 'set -e'

NOOOOOOOOOOO!!!!

[toc] | [prev] | [next] | [standalone]

#227985

On Mon, Oct 19, 2020 at 07:55:27AM -0400, Greg Wooledge wrote:
>
> > I use 'set -e'
>
> NOOOOOOOOOOO!!!!

While interesting this response is not very informative.
I can only tell that you have a problem with it.
I spent a while searching your wiki trying to find your objections
without luck, so would you plaese tell this poor heathen what your
objection to 'set -e' is?

On a different subject, my guess is that your insistence on quoting
variables and using arrays for multi-part parameters is that doing so
as a habit covers the times when a string variable will not expand as
expected while an array will.
Please correct me if I'm mis-reading things.

Thanks,
Mike
--
"God answers prayer on His own way, not ours."
    - Ghandi

[toc] | [prev] | [next] | [standalone]

#227986

On Mon, Oct 19, 2020 at 01:51:04PM -0500, Mike McClain wrote:
> I spent a while searching your wiki trying to find your objections
> without luck, so would you plaese tell this poor heathen what your
> objection to 'set -e' is?

https://mywiki.wooledge.org/BashFAQ/105

> On a different subject, my guess is that your insistence on quoting
> variables and using arrays for multi-part parameters is that doing so
> as a habit covers the times when a string variable will not expand as
> expected while an array will.
> Please correct me if I'm mis-reading things.

https://mywiki.wooledge.org/Quotes

If a string variable contains whitespace (more technically, any
character in IFS), or if it contains globbing characters (* ? [ ]),
the results of an unquoted expansion can be surprising, in a bad way.

Failing to quote the special [@] array expansion is exactly the same
as failing to quote "$@" when using the positional parameters.  Without
the quotews, it will fail to expand each element to a separate word,
which is the entire point of using an array.

unicorn:~$ array=("this array" has four elements)
unicorn:~$ printf '[%s] ' "${array[@]}"  ; echo    # right
[this array] [has] [four] [elements] 
unicorn:~$ printf '[%s] ' ${array[@]}    ; echo    # wrong
[this] [array] [has] [four] [elements] 

[toc] | [prev] | [next] | [standalone]

#227990

On Mon, 2020-10-19 at 15:16 -0400, Greg Wooledge wrote:
> On Mon, Oct 19, 2020 at 01:51:04PM -0500, Mike McClain wrote:
> > I spent a while searching your wiki trying to find your objections
> > without luck, so would you plaese tell this poor heathen what your
> > objection to 'set -e' is?
> 
> https://mywiki.wooledge.org/BashFAQ/105

NOOOOOOOOOOO!!!! ;-)

Excuse me while I go grep 'set -e' -R ...

-- 
Tixy 

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.user

csiph-web