Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #172336 > unrolled thread

Longstanding rsync script no longer excludes

Back to article view | Back to linux.debian.user

Contents

  Longstanding rsync script no longer excludes Clive Menzies <clive@clivemenzies.co.uk> - 2016-09-17 12:50 +0200
    Re: Longstanding rsync script no longer excludes Eduardo M KALINOWSKI <eduardo@kalinowski.com.br> - 2016-09-17 14:30 +0200
    Re: Longstanding rsync script no longer excludes - SOLVED Clive Menzies <clive@clivemenzies.co.uk> - 2016-09-17 18:20 +0200
      Crontab doesn't complete a script - was Re: Longstanding rsync script  no longer excludes Clive Menzies <clive@clivemenzies.co.uk> - 2016-09-19 13:20 +0200
        Re: Crontab doesn't complete a script "Thomas Schmitt" <scdbackup@gmx.net> - 2016-09-19 13:40 +0200
          Re: Crontab doesn't complete a script Clive Menzies <clive@clivemenzies.co.uk> - 2016-09-19 23:20 +0200
            Re: Crontab doesn't complete a script - SOLVED Clive Menzies <clive@clivemenzies.co.uk> - 2016-09-20 00:00 +0200
            Re: Crontab doesn't complete a script "Thomas Schmitt" <scdbackup@gmx.net> - 2016-09-20 00:10 +0200
              Re: Crontab doesn't complete a script Clive Menzies <clive@clivemenzies.co.uk> - 2016-09-20 01:10 +0200

#172336 — Longstanding rsync script no longer excludes

[Multipart message — attachments visible in raw view] — view raw

Hi

One of the issues we've encountered following a recent Jessie upgrade, 
(around 5th September) is an rsync script seems to have changed behaviour.

We've been using rsync for a daily automated incremental backup and have 
three scripts to maintain an archive of daily snapshots for a week; 
weekly snapshots for a month; and 6 monthly snapshots (files are 
irretrievable 6months after they been deleted from the backed up server).

We've two pairs of FileServer and BackupServer, rsyncd runs on 
FileServer and the following daily script on BackupServer

#!/bin/bash

# This script is called daily from cron to perform overnight backups
# The full paths of the programs used in this script
     rm=/bin/rm
     mv=/bin/mv
     cp=/bin/cp
     rsync=/usr/bin/rsync

# Good rsync options for FileServer_backups.
     rsync_opts="-av --delete --delete-excluded"

# The name of the file containing the rsync connection password
     password="--password-file=/etc/.rs_pass"

# A list of files and directories that do not need to be backed up
      exclude_list="music funstuff cma/TransitStuff lost+found/"
      excludes=""
      for exclude in $exclude_list; do
      excludes="$excludes --exclude=$exclude"
  done

# define NOW for adding date to rsync log filename eg. FileServer.date
     NOW=$(date +"%F.%H:%M")

# Move all other backups up a level. Copy previous backup to /backup/daily.
# Backup FileServer according to the [rsync] sections of the rsyncd.conf 
files
# on FileServer. Use the password given in /etc/.rs_pass. Dump any 
output and
# error messages to /var/rsync/FileServer
     mv /home/backup/snapshot/day.5 /home/backup/snapshot/day.6
     mv /home/backup/snapshot/day.4 /home/backup/snapshot/day.5
     mv /home/backup/snapshot/day.3 /home/backup/snapshot/day.4
     mv /home/backup/snapshot/day.2 /home/backup/snapshot/day.3
     mv /home/backup/snapshot/day.1 /home/backup/snapshot/day.2
     cp -al /home/backup/home /home/backup/snapshot/day.1
     $rsync $rsync_opts $password FileServer::rsync \
/home/backup/home/ > /var/rsync/FileServer.$NOW.log

The incremental backups have worked seamlessly for over 10 years but 
recently, possibly related to the upgrade of Debian Jessie, the exclude 
options have ceased to have effect.

I've dug around the docs but haven't found anything relevant. Any 
suggestions as to what may be wrong?

Thanks

Clive


-- 
Clive Menzies
http://freecriticalthinking.org

[toc] | [next] | [standalone]

#172340

[Multipart message — attachments visible in raw view] — view raw

On 17-09-2016 07:42, Clive Menzies wrote:
>
> Hi
>
> One of the issues we've encountered following a recent Jessie upgrade,
> (around 5th September) is an rsync script seems to have changed behaviour.
>
> We've been using rsync for a daily automated incremental backup and
> have three scripts to maintain an archive of daily snapshots for a
> week; weekly snapshots for a month; and 6 monthly snapshots (files are
> irretrievable 6months after they been deleted from the backed up server).
>
> We've two pairs of FileServer and BackupServer, rsyncd runs on
> FileServer and the following daily script on BackupServer
>
> #!/bin/bash
>
> # This script is called daily from cron to perform overnight backups
> # The full paths of the programs used in this script
>     rm=/bin/rm
>     mv=/bin/mv
>     cp=/bin/cp
>     rsync=/usr/bin/rsync
>
> # Good rsync options for FileServer_backups.
>     rsync_opts="-av --delete --delete-excluded"
>
> # The name of the file containing the rsync connection password
>     password="--password-file=/etc/.rs_pass"
>
> # A list of files and directories that do not need to be backed up
>      exclude_list="music funstuff cma/TransitStuff lost+found/"
>      excludes=""
>      for exclude in $exclude_list; do
>      excludes="$excludes --exclude=$exclude"
>  done
>
> # define NOW for adding date to rsync log filename eg. FileServer.date
>     NOW=$(date +"%F.%H:%M")
>
> # Move all other backups up a level. Copy previous backup to
> /backup/daily.
> # Backup FileServer according to the [rsync] sections of the
> rsyncd.conf files
> # on FileServer. Use the password given in /etc/.rs_pass. Dump any
> output and
> # error messages to /var/rsync/FileServer
>     mv /home/backup/snapshot/day.5 /home/backup/snapshot/day.6
>     mv /home/backup/snapshot/day.4 /home/backup/snapshot/day.5
>     mv /home/backup/snapshot/day.3 /home/backup/snapshot/day.4
>     mv /home/backup/snapshot/day.2 /home/backup/snapshot/day.3
>     mv /home/backup/snapshot/day.1 /home/backup/snapshot/day.2
>     cp -al /home/backup/home /home/backup/snapshot/day.1
>     $rsync $rsync_opts $password FileServer::rsync \
>     /home/backup/home/ > /var/rsync/FileServer.$NOW.log
>
> The incremental backups have worked seamlessly for over 10 years but
> recently, possibly related to the upgrade of Debian Jessie, the
> exclude options have ceased to have effect.
>
> I've dug around the docs but haven't found anything relevant. Any
> suggestions as to what may be wrong?
>

You define $excludes, but you don't seem to use it anywhere.

BTW, your rotating backups are basically what rsnapshot does, you might
want to look at that.

-- 
Eduardo M KALINOWSKI
eduardo@kalinowski.com.br

[toc] | [prev] | [next] | [standalone]

#172346 — Re: Longstanding rsync script no longer excludes - SOLVED

On 17/09/16 13:32, Mike Bird wrote

>>       $rsync $rsync_opts $password FileServer::rsync \ 
>> /home/backup/home/ > /var/rsync/FileServer.$NOW.log
>> I don't see where the $excludes you built is actually used.
>>
>

On 17/09/16 13:44, Clive Menzies wrote:
> The script wasn't changed and I've always assumed the $rsync-opts took 
> care of this. The script was copied from someone else and adapted.
>
> # Good rsync options for FileServer_backups.
>     rsync_opts="-av --delete --delete-excluded"

Thanks to prompting by Mike, I've changed the way excludes are handled 
and it works :-)

OLD version:
# Good rsync options for uhuru_backups.
     rsync_opts="-av --delete --delete-excluded"

<snip>

# A list of files and directories that do not need to be backed up
  exclude_list="music funstuff cma/TransitStuff lost+found/"
      excludes=""
      for exclude in $exclude_list; do
      excludes="$excludes --exclude=$exclude"

WORKING version:
# Good rsync options for uhuru_backups.
     rsync_opts="-av --exclude-from=exclude_list --delete --delete-excluded"

The exclude_list is a file with one excluded directory per line with an 
asterisk at the end directory name.

Regards

Clive


-- 
Clive Menzies
http://freecriticalthinking.org

[toc] | [prev] | [next] | [standalone]

#172440 — Crontab doesn't complete a script - was Re: Longstanding rsync script no longer excludes

On 17/09/16 17:12, Clive Menzies wrote:
> WORKING version:
> # Good rsync options for uhuru_backups.
>     rsync_opts="-av --exclude-from=exclude_list --delete 
> --delete-excluded"
>
> The exclude_list is a file with one excluded directory per line with 
> an asterisk at the end directory name.

      $rsync $rsync_opts $password Uhuru::rsync \
       /root/uhuru_backup/uhuru/home/ > /var/rsync/uhuru.$NOW.log

The daily backup script works fine when run manually as sudo but doesn't 
complete when run as a root cronjob. I know the script starts because it 
creates a log. When successful it gives a list of files copied but when 
run as a cronjob it's blank (0 bytes)

$ sudo crontab -l
20 21 1 * * /root/uhuru_backup/uhuru.monthly.sh > /dev/null
25 21 * * 7 /root/uhuru_backup/uhuru.weekly.sh > /dev/null
30 21 * * * /root/uhuru_backup/uhuru.daily.sh > /dev/null

This should be impossible but I've got the same issue on two backup servers.

Regards

Clive

-- 
Clive Menzies
http://freecriticalthinking.org

[toc] | [prev] | [next] | [standalone]

#172442 — Re: Crontab doesn't complete a script

Hi,

> The daily backup script works fine when run manually as sudo but doesn't
> complete when run as a root cronjob.

The classic reason for this is difference in environment variables.
I.e. the cron job could die from a non set variable.

The classic remedy would be to set the missing variables inside the
cronjob script.


> /root/uhuru_backup/uhuru.daily.sh

Consider to put some "echo" commands into the script and direct them to
a log file:

   log_file=/tmp/uhuru_daily_sh.log
   ...
   echo "Currently at: ..." >>$log_file 
   ...

This should give you an idea what part of the script causes the premature
end.


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#172504 — Re: Crontab doesn't complete a script

On 19/09/16 12:31, Thomas Schmitt wrote:
> The classic remedy would be to set the missing variables inside the
> cronjob script.
>
>
>> /root/uhuru_backup/uhuru.daily.sh
> Consider to put some "echo" commands into the script and direct them to
> a log file:
>
>     log_file=/tmp/uhuru_daily_sh.log
>     ...
>     echo "Currently at: ..." >>$log_file
>     ...
>
> This should give you an idea what part of the script causes the premature
> end.
>
>
Thanks Thomas

I confess, I've not originated scripts before and so am on a steep 
learning curve. Your suggestion was very useful but my implementation is 
poor. It did yield useful information though.

It didn't find the exclude_list I created. I modified the script to 
point to it but clearly the syntax is not right

# Good rsync options for uhuru_backups.
     rsync_opts="-av --exclude-from=exclude_list --delete --delete-excluded"

# exclude list location:
         exclude_list=/root/uhuru_backup/exclude_list

echo "Currently at line 17">>$log_file

The log reported thus:
/root/uhuru_backup/uhuru.daily.sh: line 17: $log_file: ambiguous redirect
rsync: failed to open exclude file exclude_list: No such file or 
directory (2)
rsync error: error in file IO (code 11) at exclude.c(1179) [client=3.1.1]

Regards

Clive



-- 
Clive Menzies
http://freecriticalthinking.org

[toc] | [prev] | [next] | [standalone]

#172505 — Re: Crontab doesn't complete a script - SOLVED

On 19/09/16 22:10, Clive Menzies wrote:
> It didn't find the exclude_list I created. I modified the script to 
> point to it but clearly the syntax is not right
>
> # Good rsync options for uhuru_backups.
>     rsync_opts="-av --exclude-from=exclude_list --delete 
> --delete-excluded"
>
> # exclude list location:
>         exclude_list=/root/uhuru_backup/exclude_list
>
> echo "Currently at line 17">>$log_file
>
> The log reported thus:
> /root/uhuru_backup/uhuru.daily.sh: line 17: $log_file: ambiguous redirect
> rsync: failed to open exclude file exclude_list: No such file or 
> directory (2)
> rsync error: error in file IO (code 11) at exclude.c(1179) [client=3.1.1]
>
D'oh! I searched around a bit and realised I'd over-complicated it.

This works :-)

# Good rsync options for uhuru_backups.
     rsync_opts="-av --exclude-from=/root/uhuru_backup/exclude_list \
          --delete --delete-excluded"

Thanks Thomas, I'd still be going round in circles.

Regards

Clive

-- 
Clive Menzies
http://freecriticalthinking.org

[toc] | [prev] | [next] | [standalone]

#172508 — Re: Crontab doesn't complete a script

Hi,

Clive Menzies wrote:
>     rsync_opts="-av --exclude-from=exclude_list --delete --delete-excluded"
>         exclude_list=/root/uhuru_backup/exclude_list
> rsync: failed to open exclude file exclude_list: No such file or directory (2)

The decisive difference between dialog and cron could be the
current working directory. 

Try with an absolute path as parameter to --exclude-from= :

  rsync_opts="-av --exclude-from=/root/uhuru_backup/exclude_list --delete --delete-excluded"

Or if you want to use the variable exclude_list, set it first.
Then evaluate it by $exclude_list in the definition of variable rsync_opts:

  exclude_list=/root/uhuru_backup/exclude_list
  rsync_opts="-av --exclude-from=$exclude_list --delete --delete-excluded"

This should yield the same rsync_opts content as above line.


> echo "Currently at line 17">>$log_file
> /root/uhuru_backup/uhuru.daily.sh: line 17: $log_file: ambiguous redirect

Hm. To what value did you set the variable log_file ?

If i set it to "/tmp/log_file", then the echo works in dash and bash.
(We have to expect that cron uses the shell dash, whereas the dialog
 shell is probably bash.)
 
If i set it to empty text, then dash complains
  dash: 2: Syntax error: newline unexpected
bash says
  -bash: syntax error near unexpected token `newline'

A blank would do it in bash but not in dash:

  $ log_file="/tmp/log file"
  $ echo "Currently at line 17">>$log_file
  -bash: $log_file: ambiguous redirect

dash is brutal enough to really create a file "/tmp/log file".


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#172511 — Re: Crontab doesn't complete a script

Hi Thomas

The absolute path was the answer.

Your explanation has added to my limited knowledge :-)

Thanks

Clive

On 19/09/16 23:04, Thomas Schmitt wrote:
> Hi,
>
> Clive Menzies wrote:
>>      rsync_opts="-av --exclude-from=exclude_list --delete --delete-excluded"
>>          exclude_list=/root/uhuru_backup/exclude_list
>> rsync: failed to open exclude file exclude_list: No such file or directory (2)
> The decisive difference between dialog and cron could be the
> current working directory.
>
> Try with an absolute path as parameter to --exclude-from= :
>
>    rsync_opts="-av --exclude-from=/root/uhuru_backup/exclude_list --delete --delete-excluded"
>
> Or if you want to use the variable exclude_list, set it first.
> Then evaluate it by $exclude_list in the definition of variable rsync_opts:
>
>    exclude_list=/root/uhuru_backup/exclude_list
>    rsync_opts="-av --exclude-from=$exclude_list --delete --delete-excluded"
>
> This should yield the same rsync_opts content as above line.
>
>
>> echo "Currently at line 17">>$log_file
>> /root/uhuru_backup/uhuru.daily.sh: line 17: $log_file: ambiguous redirect
> Hm. To what value did you set the variable log_file ?
>
> If i set it to "/tmp/log_file", then the echo works in dash and bash.
> (We have to expect that cron uses the shell dash, whereas the dialog
>   shell is probably bash.)
>   
> If i set it to empty text, then dash complains
>    dash: 2: Syntax error: newline unexpected
> bash says
>    -bash: syntax error near unexpected token `newline'
>
> A blank would do it in bash but not in dash:
>
>    $ log_file="/tmp/log file"
>    $ echo "Currently at line 17">>$log_file
>    -bash: $log_file: ambiguous redirect
>
> dash is brutal enough to really create a file "/tmp/log file".
>
>
> Have a nice day :)
>
> Thomas
>

-- 
Clive Menzies
http://freecriticalthinking.org

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web