Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #160649 > unrolled thread

dear lazyweb: looking for a special backup tool

Back to article view | Back to linux.debian.user

Contents

  dear lazyweb: looking for a special backup tool Christoph Anton Mitterer <calestyo@scientia.net> - 2015-11-17 01:30 +0100
    Re: dear lazyweb: looking for a special backup tool "Thomas Schmitt" <scdbackup@gmx.net> - 2015-11-17 09:20 +0100
      Re: dear lazyweb: looking for a special backup tool Gene Heskett <gheskett@shentel.net> - 2015-11-17 13:30 +0100
    Re: dear lazyweb: looking for a special backup tool Jonathan Dowland <jmtd@debian.org> - 2015-11-17 12:50 +0100
      Re: dear lazyweb: looking for a special backup tool Christoph Anton Mitterer <calestyo@scientia.net> - 2015-11-18 03:40 +0100

#160649 — dear lazyweb: looking for a special backup tool

[Multipart message — attachments visible in raw view] — view raw

Hey.

Maybe someone can help me with this...

I'm looking for a backup solution with quite some specific needs,...
the idea is basically to backup my main data vaults (which are already
on RAID and regularly btrfs-sent/received to other HDDs) to optical
media (or smaller HDDs with e.g. another fs).


The following goals/features should be met:
the easier parts:
- regular files, symlinks directories must be retained
  for special files it would be nice, but not a must
- hard links must be retained
- file times, owners (ideally as IDs and names), permissions, XATTRS,
  ACLs must all be retained

- it must be possible to backup to split media (e.g. mutliple CDs) or
  multiple smaller filesystems (e.g. which I dm-(en)crypt and then
  burn to CD).
- I want always *full* files to be backuped (in both cases, when
  splitting and when doing incremental backups), so
  - a single file shouldn't be split over multiple backup media
    (unless this isn't possible otherweise, because all targets are
    smaller than the file size)
  - I don't want deltas to be stored (i.e. deltas to the last
    incremental backup)
  The reason for both is simply resilience and recoverability in case
  of loosing single mediums.
- ideally the program would offer two modes:
  - either trying to keep "neighbouring" files (i.e. those that are
    close to each other in the directory hierarchy) closely on the 
    split target mediums
  - or trying to be as space efficient as possible (i.e. place files so
    that space is used most efficiently
  with the first mode being more important.
- catalogues should be made, of both, all files and the files on a
  certain medium, also as a help in the disaster case

here it gets tricky:
- as I've said, incremental backups should be possible,... but that
  should also work when I move files around
  There are basically two ways, either the program sets it's own IDs on
  the files as XATTRs and identifies files based on that, while always
  assuring that IDs are unique. The disadvantage on that would be that
  it's not so uncommon that XATTRs are lost on files, when those get
  edited.
  The other would be to use hashsums, which would be nice, because I
  already store hashsums of each file in an xattr... but I would want
  the program to really verify (i.e. byte for byte) when two different
  files have a matching hash sum. I know, finding collisions is
  unlikely but not impossible.


Well I guess it wouldn't be too difficult to script most of this, but
if there's already a tool which does the job, I'd be happy to hear
about it.


Cheers,
Chris

[toc] | [next] | [standalone]

#160659

Hi,

Christoph Anton Mitterer wrote:
> I'm looking for a backup solution with quite some specific needs,...

My scdbackup with xorriso as backend might nearly do what
you want:
  http://scdbackup.sourceforge.net/main_eng.html

It initially demands some configuration effort
  http://scdbackup.sourceforge.net/examples.html
  http://scdbackup.sourceforge.net/README

xorriso is available as Debian package, or as source tarball
  https://www.gnu.org/software/xorriso/xorriso-1.4.0.tar.gz


> - hard links must be retained
> - it must be possible to backup to split media (e.g. mutliple CDs)

This is not easy to achieve.
Consider one hardlink sibling in one fileystem and the other
in the next filesystem.


> - file times, owners (ideally as IDs and names), permissions, XATTRS,
>  ACLs must all be retained

scdbackup records them in scripts with setfattr and setfacl,
which it stores in data files in the backup.
xorriso records them too, but restoring those will need xorriso,
because Linux does not interpret AAIP (which is not astounding
because it is a libisofs extension to ISO 9660).

ISO 9660 has the advantage of being readable nearly everywhere
as long as you restrict yourself to single-session and data
files smaller than 4 GiB. (BSD and Solaris ended development of
their ISO 9660 readers long ago. Meanwhile one can smell their
fermentation state.)


> - I want always *full* files to be backuped
>   - a single file shouldn't be split over multiple backup media
>     (unless this isn't possible otherweise, because all targets are
>     smaller than the file size)

Putting together the pieces of large files is of course a pain
at restore time.


> - ideally the program would offer two modes:
>   - either trying to keep "neighbouring" files (i.e. those that are
>     close to each other in the directory hierarchy) closely on the 
>     split target mediums

You mean those with neighboring names, i assume.
(More or less the same as alphabetic ordering by name.)


>  - or trying to be as space efficient as possible (i.e. place files so
>    that space is used most efficiently

After a few years of cramming files, i switched to alphabetic.


> - catalogues should be made, of both, all files and the files on a
>   certain medium, also as a help in the disaster case

Not to forget checksums.
In most dangerous storage environments one should consider to
make several identical copies of the media and have means to
recognize good blocks.


> - as I've said, incremental backups should be possible,... but that
>  should also work when I move files around

So timestamps are not enough.
scdbackup has two ways of incremental backups:
- If inode numbers are persistent and ( devide numbers are persistent
  or mount points always lead to the same filesystems), then it is
  possible to decide by the outcome of stat(2).
- Much slower is decision on base of MD5 of file content.


> Well I guess it wouldn't be too difficult to script most of this,

scdbackup needed about seven years of try-and-error.
It can be done faster if you have a clear plan.
Beware of being sucked into neighboring topics like DVD burning
or backup-grade ISO 9660 production.


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#160665

On Tuesday 17 November 2015 03:14:50 Thomas Schmitt wrote:

> Hi,
>
> Christoph Anton Mitterer wrote:
> > I'm looking for a backup solution with quite some specific needs,...
>
> My scdbackup with xorriso as backend might nearly do what
> you want:
>   http://scdbackup.sourceforge.net/main_eng.html
>
> It initially demands some configuration effort
>   http://scdbackup.sourceforge.net/examples.html
>   http://scdbackup.sourceforge.net/README
>
> xorriso is available as Debian package, or as source tarball
>   https://www.gnu.org/software/xorriso/xorriso-1.4.0.tar.gz
>
> > - hard links must be retained
> > - it must be possible to backup to split media (e.g. mutliple CDs)
>
> This is not easy to achieve.
> Consider one hardlink sibling in one fileystem and the other
> in the next filesystem.
>
> > - file times, owners (ideally as IDs and names), permissions,
> > XATTRS, ACLs must all be retained
>
> scdbackup records them in scripts with setfattr and setfacl,
> which it stores in data files in the backup.
> xorriso records them too, but restoring those will need xorriso,
> because Linux does not interpret AAIP (which is not astounding
> because it is a libisofs extension to ISO 9660).
>
> ISO 9660 has the advantage of being readable nearly everywhere
> as long as you restrict yourself to single-session and data
> files smaller than 4 GiB. (BSD and Solaris ended development of
> their ISO 9660 readers long ago. Meanwhile one can smell their
> fermentation state.)
>
> > - I want always *full* files to be backuped
> >   - a single file shouldn't be split over multiple backup media
> >     (unless this isn't possible otherweise, because all targets are
> >     smaller than the file size)
>
> Putting together the pieces of large files is of course a pain
> at restore time.
>
> > - ideally the program would offer two modes:
> >   - either trying to keep "neighbouring" files (i.e. those that are
> >     close to each other in the directory hierarchy) closely on the 
> >     split target mediums
>
> You mean those with neighboring names, i assume.
> (More or less the same as alphabetic ordering by name.)
>
> >  - or trying to be as space efficient as possible (i.e. place files
> > so that space is used most efficiently
>
> After a few years of cramming files, i switched to alphabetic.
>
> > - catalogues should be made, of both, all files and the files on a
> >   certain medium, also as a help in the disaster case
>
> Not to forget checksums.
> In most dangerous storage environments one should consider to
> make several identical copies of the media and have means to
> recognize good blocks.
>
> > - as I've said, incremental backups should be possible,... but that
> >  should also work when I move files around
>
> So timestamps are not enough.
> scdbackup has two ways of incremental backups:
> - If inode numbers are persistent and ( devide numbers are persistent
>   or mount points always lead to the same filesystems), then it is
>   possible to decide by the outcome of stat(2).
> - Much slower is decision on base of MD5 of file content.
>
> > Well I guess it wouldn't be too difficult to script most of this,
>
> scdbackup needed about seven years of try-and-error.
> It can be done faster if you have a clear plan.
> Beware of being sucked into neighboring topics like DVD burning
> or backup-grade ISO 9660 production.
>
>
> Have a nice day :)
>
> Thomas

I have been using amanda (Advanced Maryland Automatic Network Disk 
Archiver) since 1998 or 99.  Setup here to use files on a BIG HD that 
are its "tape library", I have found such a setup to be both much faster 
when it comes to recovery, AND considerably more dependable than any 
tape format I can afford. Amanda has a different basic format, mixing 
backup levels in an attempt to use about the same amount of media per 
run.  A typical run over the 4 machines live on my home network will run 
to about 40Gb, which it does in about 3.5 hours in the middle of the 
night.  I do a 4 day cycle so I get a level0 full backup of every 
disklist entry at some point in those 4 days, mixed with incremental, 
only whats changed, in between times.  And I can reach back as far as 30 
days before that part of the disk is reused. So if I find something 
valuable has been erased, I have 30 days to discover it and recover it.

There are tools which can make offsite, forever storage for amanda users. 
But I'm not recording business transactions I might have to recover for 
the Infernal Revenoo Service, I'm just a retired hobbiest making a stick 
or two of furniture when the mood strikes me.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#160663

I *think* you should be able to achieve this via Bacula and a lot of time
configuring it, but I haven't tried myself. I've been considering whether
to add some optical discs to my backup system as a 3rd backup after local
HDD and remote HDD, and Bacula is where I would start if I ever get to it.

[toc] | [prev] | [next] | [standalone]

#160714

[Multipart message — attachments visible in raw view] — view raw

Hey guys.

Thanks for the suggestions... I'll have a look at them :)

Cheers,
Chris.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web