Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #196677 > unrolled thread

rsnapshot is Throwing an Intermittent Error.

Back to article view | Back to linux.debian.user

Contents

  rsnapshot is Throwing an Intermittent Error. "Martin McCormick" <martin.m@suddenlink.net> - 2018-06-14 17:40 +0200
    Re: rsnapshot is Throwing an Intermittent Error. Greg Wooledge <wooledg@eeg.ccf.org> - 2018-06-14 17:50 +0200
      Re: rsnapshot is Throwing an Intermittent Error. "Martin McCormick" <martin.m@suddenlink.net> - 2018-06-14 19:30 +0200

#196677 — rsnapshot is Throwing an Intermittent Error.

On most days or nights, this runs perfectly and then there is the following:

From:    root@wb5agz (Cron Daemon)
Subject: Cron <root@wb5agz> /usr/local/etc/daily_backup


/bin/rm: cannot remove '/var/cache/rsnapshot/halfday.1/wb5agz/home/usr/lib/grub
/i386-pc': Transport endpoint is not connected
/bin/rm: cannot remove '/var/cache/rsnapshot/halfday.1/wb5agz/home/usr/lib/libb
ind9.so.80.0.7': Transport endpoint is not connected

This can go on for hundreds of lines and always references a
different set of directories and files.

	There can also be weeks of error-free backups in which
everything just works.

I think I am creating this mess by the way I do the backups since
it smells like some sort of race condition.  The daily backup
script follows:

#!/bin/sh
#Do the halfday backup first.
#The halfday mounts and unmounts the backup media.
/usr/bin/rsnapshot halfday
#Mount backup media first since this next is just a rotation.
mount /rsnapshot1 >/dev/null 2>&1 ||exit 1
mount /rsnapshot2 >/dev/null 2>&1 ||exit 1
    mhddfs /rsnapshot1,/rsnapshot2 /var/cache/rsnapshot -o mlimit=100M >/dev/null 2>&1 
/usr/bin/rsnapshot daily
umount /var/cache/rsnapshot /rsnapshot2 /rsnapshot1 
exit 0


As far as I know, both mount and umount block until they reach
some sort of resolution, be it success or failure.  I have even
seen umount hang for a perceptible amount of time when one has
changed a large number of blocks and sync hasn't had time to
catch up so I am curious as to what may be happening.

	I was able to simply re-run the very same command later
for the daily backup and it worked without so much as a peep out
of anything. It just ran.

	The two backup drives in question passed fsck -f -y without
a single issue.  In most cases of one of these big error spews,
the files are in places that aren't changing on a regular basis
so it's not as if I caught a log file just as it was backing up.
I don't ever remember seeing log files in the spew.

	I always leave the backup drives unmounted unless I need
something off of them or the cron job is running since it would
seem to make sense to not mount them continuously.  A couple of
weeks ago, we got 4 electrical power blinks in one day so you
don't want your backup media mounted any more than necessary.

	When I do pull something off of the backups, it is good
and not corrupted so far so I am really curious as to what is
happening.

Martin WB5AGZ

[toc] | [next] | [standalone]

#196679

On Thu, Jun 14, 2018 at 10:36:01AM -0500, Martin McCormick wrote:
> /bin/rm: cannot remove '/var/cache/rsnapshot/halfday.1/wb5agz/home/usr/lib/grub

> #Mount backup media first since this next is just a rotation.
> mount /rsnapshot1 >/dev/null 2>&1 ||exit 1
> mount /rsnapshot2 >/dev/null 2>&1 ||exit 1
>     mhddfs /rsnapshot1,/rsnapshot2 /var/cache/rsnapshot -o mlimit=100M >/dev/null 2>&1 

My immediate thought is to stop throwing away the output of these
commands, so that if one of them is failing or giving warnings, you can
actually see what's wrong.

At least get rid of the 2>&1 on all 3 lines, so you get the stderr output.
Or log the stdout and stderr to a file, if you absolutely can't stand
the thought that you might ever possibly get it mailed to you.

[toc] | [prev] | [next] | [standalone]

#196685

Greg Wooledge <wooledg@eeg.ccf.org> writes:
> My immediate thought is to stop throwing away the output of these
> commands, so that if one of them is failing or giving warnings, you can
> actually see what's wrong.
> 
> At least get rid of the 2>&1 on all 3 lines, so you get the stderr output.
> Or log the stdout and stderr to a file, if you absolutely can't stand
> the thought that you might ever possibly get it mailed to you.

	Syslog shows:

Jun 14 00:01:01 wb5agz CRON[5701]: (root) CMD (/usr/local/etc/daily_backup)
Jun 14 00:01:02 wb5agz kernel: [572848.771718] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
Jun 14 00:01:02 wb5agz kernel: [572848.941297] EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null)
Jun 14 00:27:04 wb5agz rsnapshot[5703]: /usr/bin/rsnapshot halfday: completed successfully

That was actually where the error originated which was mailed to
me.

This next entry is after the drives were unmounted for the
halfday and shows their remount for the daily rotation

Jun 14 00:27:05 wb5agz kernel: [574412.532306] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
Jun 14 00:27:06 wb5agz kernel: [574412.792220] EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null)
Jun 14 00:27:07 wb5agz rsnapshot[5887]: /usr/bin/rsnapshot daily: completed successfully

	The squawks are coming from rsnapshot, itself and it's output
is not blocked from being mailed. What you saw was the beginning
of the body of the mail message indicating a problem

	I did remove the tossing of the standard error output on
the mount commands and mhddfs but syslog shows all those commands
as happening normally.
	rsnapshot is telling syslog it completed successfully at
the same time I got the spew and rsnapshot basically failed
safely and did not save a new backup

Martin

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web