Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #173016 > unrolled thread

Recommendation: Backup system

Back to article view | Back to linux.debian.user

Contents

  Recommendation: Backup system mo <mo777@gmx.net> - 2016-10-01 11:40 +0200
    Re: Recommendation: Backup system Mark Fletcher <mark27q1@gmail.com> - 2016-10-01 14:50 +0200
      Re: Recommendation: Backup system mo <mo777@gmx.net> - 2016-10-01 15:10 +0200
      Re: Recommendation: Backup system Gene Heskett <gheskett@shentel.net> - 2016-10-01 18:30 +0200
        Re: Recommendation: Backup system Clive Menzies <clive@clivemenzies.co.uk> - 2016-10-01 18:50 +0200
          Re: Recommendation: Backup system Gene Heskett <gheskett@shentel.net> - 2016-10-01 19:50 +0200
            Re: Recommendation: Backup system Clive Menzies <clive@clivemenzies.co.uk> - 2016-10-01 20:00 +0200
              Re: Recommendation: Backup system Glenn English <ghe2001@gmail.com> - 2016-10-01 22:10 +0200
              Re: Recommendation: Backup system Gene Heskett <gheskett@shentel.net> - 2016-10-02 06:20 +0200
        Re: Recommendation: Backup system Glenn English <ghe2001@gmail.com> - 2016-10-01 22:00 +0200
          Re: Recommendation: Backup system mo <mo777@gmx.net> - 2016-10-01 22:50 +0200
        Re: Recommendation: Backup system mo <mo777@gmx.net> - 2016-10-01 22:40 +0200
    Re: Recommendation: Backup system Bob Weber <bobrweber@gmail.com> - 2016-10-01 16:50 +0200
      Re: Recommendation: Backup system mo <mo777@gmx.net> - 2016-10-01 17:10 +0200
      Re: Recommendation: Backup system Mark Fletcher <mark27q1@gmail.com> - 2016-10-01 17:40 +0200
    Re: Recommendation: Backup system mo <mo777@gmx.net> - 2016-10-01 17:20 +0200
    Re: Recommendation: Backup system Dan Purgert <dan@djph.net> - 2016-10-01 19:10 +0200
      Re: Recommendation: Backup system mo <mo777@gmx.net> - 2016-10-01 22:50 +0200
    Re: Recommendation: Backup system mo <mo777@gmx.net> - 2016-10-01 22:50 +0200
      Re: Recommendation: Backup system Bob Weber <bobrweber@gmail.com> - 2016-10-01 23:10 +0200
        Re: Recommendation: Backup system mo <mo777@gmx.net> - 2016-10-02 10:50 +0200
          Re: Recommendation: Backup system rhkramer@gmail.com - 2016-10-02 15:00 +0200
            Re: Recommendation: Backup system Bob Weber <bobrweber@gmail.com> - 2016-10-03 01:20 +0200
      Re: Recommendation: Backup system Dan Purgert <dan@djph.net> - 2016-10-02 03:00 +0200
        Re: Recommendation: Backup system mo <mo777@gmx.net> - 2016-10-02 11:00 +0200
          Re: Recommendation: Backup system Dan Purgert <dan@djph.net> - 2016-10-02 13:10 +0200
            Re: Recommendation: Backup system mo <mo777@gmx.net> - 2016-10-02 14:40 +0200
    Re: Recommendation: Backup system Jonathan Dowland <jmtd@debian.org> - 2016-10-03 11:50 +0200
      Re: Recommendation: Backup system Teemu Likonen <tlikonen@iki.fi> - 2016-10-03 14:40 +0200
        Re: Recommendation: Backup system Markus Grunwald <markus@the-grue.de> - 2016-10-03 14:50 +0200
          Re: Recommendation: Backup system Richard Hector <richard@walnut.gen.nz> - 2016-10-05 04:10 +0200
            Re: Recommendation: Backup system Daniel Bareiro <daniel-listas@gmx.net> - 2016-10-05 05:10 +0200
              Re: Recommendation: Backup system Richard Hector <richard@walnut.gen.nz> - 2016-10-05 23:40 +0200
            Re: Recommendation: Backup system Jonathan Dowland <jmtd@debian.org> - 2016-10-07 12:20 +0200
              Re: Recommendation: Backup system Richard Hector <richard@walnut.gen.nz> - 2016-10-12 12:10 +0200
        Re: Recommendation: Backup system Jonathan Dowland <jmtd@debian.org> - 2016-10-03 17:10 +0200
      Re: Recommendation: Backup system Celejar <celejar@gmail.com> - 2016-10-10 19:30 +0200
        Re: Recommendation: Backup system Jonathan Dowland <jmtd@debian.org> - 2016-10-11 00:00 +0200
          Re: Recommendation: Backup system Dave Thayer <debian1628420.dmthayer@recursor.net> - 2016-10-11 05:40 +0200
    Re: Recommendation: Backup system Jarle Aase <jgaa@jgaa.com> - 2016-10-04 20:50 +0200

Page 2 of 2 — ← Prev page 1 [2]

#173107

Am 01.10.2016 um 23:06 schrieb Bob Weber:
> Like I said backuppc uses incremental and full backups.  The web
> interface lets you browse any backup (inc or full) and you see all the
> files backed up.  I set the incremental for each day up to a week.  So I
> have up to 7 of them.  The full can kept for for however long you want.
> I currently keep 12 weekly, 8 bi-weekly and 4 monthly full backups so
> that covers almost a year.

That would serve my needs well also :) (For the moment it is just my 
private data - But either way, data should always be backed up ;) )
Thanks for the info ;)

> There is another solution you might like called rsnapshot.  I use it to
> backup just my root directory on my desktop before I do updates.  That
> way if something goes wrong I can boot into a rescue cd and restore the
> system to the state before the update.  I just can't afford to have my
> desktop to break.  rsnapshot uses rsync so it can backup any computer
> that has rsync.  It uses hard links so duplicate files are only stored
> once.  You specify how many backups you want to keep and rsnapshot
> deletes older ones over that max before adding the new one.  That way
> you always have backups (assuming you set the count greater that 1) that
> will be there even if there is a transfer error.  This is similar to
> your script but is very versatile.

I heard about it but never used it so far, i should give that a try for 
sure :)

>
> *...Bob*
> On 10/01/2016 04:42 PM, mo wrote:
>> Maybe this is a little OT, but what kind of backup strategy would you
>> guys recommend? (Any advice Gene? :) )
>>
>>
>

Greets

mo

[toc] | [prev] | [next] | [standalone]

#173119

> Am 01.10.2016 um 23:06 schrieb Bob Weber:
> > Like I said backuppc uses incremental and full backups.  The web
> > interface lets you browse any backup (inc or full) and you see all the
> > files backed up.  I set the incremental for each day up to a week.  So I
> > have up to 7 of them.  The full can kept for for however long you want.
> > I currently keep 12 weekly, 8 bi-weekly and 4 monthly full backups so
> > that covers almost a year.

I am not the op, but backuppc sounded pretty nice, so yesterday I tried to 
install it on both my Wheezy and Jessie systems.  It didn't work (with 
different failures) on either system--I won't give much detail for now, but I'd 
just ask a few questions:

   * what system (what version of Debian) are you using?

   * should I expect that it will properly configure a web server (on the 
Wheezy system it talked about apache2, iirc), or must I have a properly 
configured web server before installing backuppc?

Some cryptic notes on the failures:

On wheezy, I thought the installation completed successfully--it ran something 
it called a script, and, in or after the script it gave me a url to log in to 
manage backuppc along with a username and password.  When I tried to go to 
that URL, using either http or https on either of my browsers, it gave me a 
404 error.

On jessie, it apparently did not complete the installation, it told me it 
could not run that initial script.

Suggestions?

[toc] | [prev] | [next] | [standalone]

#173139

On 10/02/2016 08:50 AM, rhkramer@gmail.com wrote:
>> Am 01.10.2016 um 23:06 schrieb Bob Weber:
>>> Like I said backuppc uses incremental and full backups.  The web
>>> interface lets you browse any backup (inc or full) and you see all the
>>> files backed up.  I set the incremental for each day up to a week.  So I
>>> have up to 7 of them.  The full can kept for for however long you want.
>>> I currently keep 12 weekly, 8 bi-weekly and 4 monthly full backups so
>>> that covers almost a year.
> I am not the op, but backuppc sounded pretty nice, so yesterday I tried to 
> install it on both my Wheezy and Jessie systems.  It didn't work (with 
> different failures) on either system--I won't give much detail for now, but I'd 
> just ask a few questions:
>
>    * what system (what version of Debian) are you using?
>
>    * should I expect that it will properly configure a web server (on the 
> Wheezy system it talked about apache2, iirc), or must I have a properly 
> configured web server before installing backuppc?
>
> Some cryptic notes on the failures:
>
> On wheezy, I thought the installation completed successfully--it ran something 
> it called a script, and, in or after the script it gave me a url to log in to 
> manage backuppc along with a username and password.  When I tried to go to 
> that URL, using either http or https on either of my browsers, it gave me a 
> 404 error.
>
> On jessie, it apparently did not complete the installation, it told me it 
> could not run that initial script.
>
> Suggestions?
>
>

I am running stable 8.5 on my backup machine.  It is a 386 install.  Backuppc is
version backuppc/stable,now 3.3.0-2 i386 [installed]

It uses apache2 (dont know if it will use other web servers) and has put the
following in the apache  /etc/apache2/conf-enabled/backuppc.conf file:

Alias /backuppc /usr/share/backuppc/cgi-bin/

<Directory /usr/share/backuppc/cgi-bin/>
        AllowOverride None
        Allow from all

        # Uncomment the line below to ensure that nobody can sniff importanti
        # info from network traffic during editing of the BackupPC config or
        # when browsing/restoring backups.
        # Requires that you have your webserver set up for SSL (https) access.
        #SSLRequireSSL

        Options ExecCGI FollowSymlinks
        AddHandler cgi-script .cgi
        DirectoryIndex index.cgi

        AuthUserFile /etc/backuppc/htpasswd
        AuthType basic
        AuthName "BackupPC admin"
        require valid-user

</Directory>

So I would assume that there should be a working install of apache2 before
backuppc is installed.  The only install problem I remember (this was quite a
while ago) was that I wanted the backups put in a directory mounted on a
separate partition.  Even though there was a setting for the backup directory in
backuppc the directory is hard coded to  "/var/lib/backuppc" in some of the
installed backuppc programs.  So to use another location you have to symbolic
link /var/lib/backuppc to that directory before install (or mount the partition
on /var/lib/backuppc).  So if you want to use another directory delete
/var/lib/backuppc and make the link (link -s /var/lib/backuppc  /somewhere-else
  or    mount /dev/sdxx /var/lib/backuppc).  Then run "apt-get install
--reinstall backuppc" and hopefully things will be setup correctly at that new
location.  Since I access the backup server on my local net I don't use https. 

I use rsync over ssh to connect to linux servers so the data transferred is over
a secure tunnel. So my backup of a remote vm is secure.  The backuppc user
should have public key that is placed in the authorized_keys file of the clients
that are to be backed up. http://backuppc.sourceforge.net/faq/ssh.html explains
this procedure.

Hope this helps.  I have run backuppc over 10 years at several locations where I
have worked and at home.  It just seems to run.

...Bob

[toc] | [prev] | [next] | [standalone]

#173096

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

mo wrote:
> Maybe this is a little OT, but what kind of backup strategy would you 
> guys recommend? (Any advice Gene? :) )

If it *must* survive, 3-2-1 is the way to go.

3 copies (Original, Backup, and Backup of the Backup)
2 different media types (such as HDD(Orig) + HDD(Bk1) + Optical(Bk2))
1 stored offsite

That being said, most of my stuff isn't of the "absolutely must survive"
nature, so I just have it on external HDDs.  If something is
particularly important, it's on HDDs and also somewhere like google
drive / dropbox (although, encrypted when at those places).

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJX8FjGAAoJEI4R3fMSeaKBX9gH/i6qw2GOY/LSE5haz8uv6u/X
GNT6m44c6olrnzGtKFzEPae3ELTTAT/t47yet2noAR+AS9aJzhtR86l3JfQfu4ug
0XplxY/GkkrEpIeMFrqKy2hc3u++PoEHEBkQut46x6QAw/85ieKs2tsbmfeyUuF7
y5gFbGlowcy3NtSdGIMR6qzn67I+DaO2veTMV3Z+/aR+fAegZtvv/t3CCi1dsmRW
kEWIuPyGR//Foy/vkQsakpHJnl9AvJREB5/T+zRX3pkBXKkzQrIfe80eJYAm5Z8w
rhvZc9lLbO/bVcSvLwRKqYtDmWRY4AeVrqmCitiuXf54KjwZY5Lh16LVq+KfCiQ=
=YzZ4
-----END PGP SIGNATURE-----

-- 
|_|O|_| Registered Linux user #585947
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: 05CA 9A50 3F2E 1335 4DC5  4AEE 8E11 DDF3 1279 A281

[toc] | [prev] | [next] | [standalone]

#173109

Am 02.10.2016 um 02:47 schrieb Dan Purgert:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> mo wrote:
>> Maybe this is a little OT, but what kind of backup strategy would you
>> guys recommend? (Any advice Gene? :) )
>
> If it *must* survive, 3-2-1 is the way to go.
>
> 3 copies (Original, Backup, and Backup of the Backup)
> 2 different media types (such as HDD(Orig) + HDD(Bk1) + Optical(Bk2))
> 1 stored offsite

Except for the offsite storage (Just for my private data) this sounds 
perfect for my needs. Good thing is that i have two free HDD's here with 
2TB each, so this would work ;)

> That being said, most of my stuff isn't of the "absolutely must survive"
> nature, so I just have it on external HDDs.  If something is
> particularly important, it's on HDDs and also somewhere like google
> drive / dropbox (although, encrypted when at those places).

I currently use dropbox myself to store my conf files (gzipped and then 
encrypted with aes) in case something bad should ever happen to my NAS, 
so that i can at least restore my servers quickly ;)

One question Dan: What do u use to encrypt your files? also openssl?

> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
>
> iQEcBAEBAgAGBQJX8FjGAAoJEI4R3fMSeaKBX9gH/i6qw2GOY/LSE5haz8uv6u/X
> GNT6m44c6olrnzGtKFzEPae3ELTTAT/t47yet2noAR+AS9aJzhtR86l3JfQfu4ug
> 0XplxY/GkkrEpIeMFrqKy2hc3u++PoEHEBkQut46x6QAw/85ieKs2tsbmfeyUuF7
> y5gFbGlowcy3NtSdGIMR6qzn67I+DaO2veTMV3Z+/aR+fAegZtvv/t3CCi1dsmRW
> kEWIuPyGR//Foy/vkQsakpHJnl9AvJREB5/T+zRX3pkBXKkzQrIfe80eJYAm5Z8w
> rhvZc9lLbO/bVcSvLwRKqYtDmWRY4AeVrqmCitiuXf54KjwZY5Lh16LVq+KfCiQ=
> =YzZ4
> -----END PGP SIGNATURE-----
>

Greets

mo

[toc] | [prev] | [next] | [standalone]

#173112

mo wrote:
> Am 02.10.2016 um 02:47 schrieb Dan Purgert:
>> mo wrote:
>>> Maybe this is a little OT, but what kind of backup strategy would you
>>> guys recommend? (Any advice Gene? :) )
>>
>> If it *must* survive, 3-2-1 is the way to go.
>>
>> 3 copies (Original, Backup, and Backup of the Backup)
>> 2 different media types (such as HDD(Orig) + HDD(Bk1) + Optical(Bk2))
>> 1 stored offsite
>
> Except for the offsite storage (Just for my private data) this sounds 
> perfect for my needs. Good thing is that i have two free HDD's here with 
> 2TB each, so this would work ;)

Yeah, I don't have a spare copy offsite either -- but I don't
(currently) have any electronic data that would be absolutely
devastating if I lost it.  If / when I have kids or something, then baby
pics among other things will likely have that status.

> [snip]
> One question Dan: What do u use to encrypt your files? also openssl?

I just use PGP.

-- 
|_|O|_| Registered Linux user #585947
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: 05CA 9A50 3F2E 1335 4DC5  4AEE 8E11 DDF3 1279 A281

[toc] | [prev] | [next] | [standalone]

#173116

Am 02.10.2016 um 12:55 schrieb Dan Purgert:
> mo wrote:
>> Am 02.10.2016 um 02:47 schrieb Dan Purgert:
>>> mo wrote:
>>>> Maybe this is a little OT, but what kind of backup strategy would you
>>>> guys recommend? (Any advice Gene? :) )
>>>
>>> If it *must* survive, 3-2-1 is the way to go.
>>>
>>> 3 copies (Original, Backup, and Backup of the Backup)
>>> 2 different media types (such as HDD(Orig) + HDD(Bk1) + Optical(Bk2))
>>> 1 stored offsite
>>
>> Except for the offsite storage (Just for my private data) this sounds
>> perfect for my needs. Good thing is that i have two free HDD's here with
>> 2TB each, so this would work ;)
>
> Yeah, I don't have a spare copy offsite either -- but I don't
> (currently) have any electronic data that would be absolutely
> devastating if I lost it.  If / when I have kids or something, then baby
> pics among other things will likely have that status.

But i guess i should consider it, loosing all the data is quite hard 
(Had that a while ago).

>> [snip]
>> One question Dan: What do u use to encrypt your files? also openssl?
>
> I just use PGP.
>

I need to delve into PGP a little more ;) (Currently i use the openssl 
command to do my encrpytion on the backup archives)

[toc] | [prev] | [next] | [standalone]

#173145

[Multipart message — attachments visible in raw view] — view raw

On Sat, Oct 01, 2016 at 11:37:31AM +0200, mo wrote:
> Make a long story short:
> Have you guys a recommendation for me?
> Is there a specific application you use for your backups guys?

rdiff-backup[1]. I don't know what your NAS is, whether it's an off-the-shelf
thing or a DIY system, but I tend to initiate the backups on my NAS (which is
a Debian-powered mini itx PC). You can initiate from the clients instead if
you prefer.

rdiff-backup is, conceptually, quite similar to the level of detail you are
working at with your script, so it should be easy to get working. I'd also
consider looking at Obnam[2], and I'd avoid rsnapshot if I were you (it
falls over with large quantities of files, such as mailboxes, and rdiff-backup
basically does the same job but better).

rsync, whilst an awesome piece of software, is not, on its own, a backup
system.

When evaluating or building a backup system, consider all of the points in
"The Tao of Backup"[3]. Finally, remember that RAID is not backup[4]. (JWZ's
advice about a 3rd backup disk here is particularly good. I presently use
rsync for my 3rd backup drive.)

Penultimately, I'll echo what another poster has said, backuppc[5] is very good,
especially if you have windows clients (I realise you don't), although the
backend storage (last time I looked) was a somewhat awkward format that I
think would be painful to extract stuff from in an emergency situation should
the BackupPC software itself fail. As for Bacula, I *think* it's more geared
towards an enterprise-level, perhaps using tape backups, like Amanda. I know
you can use Bacula w/o tapes (Amanda too) but that's kind-of where it came
from and it still "feels" like a tape-oriented system to me (others are welcome
to jump to Bacula's defence here)

Finally, and although Tao of Backup covers this I'll repeat it: test restores.
Test how easy restores are for any tool of choice. Restores, restore performance
and restore UI is infinitely more important than the backup process.

[1] http://rdiff-backup.nongnu.org
[2] http://obnam.org/
[3] http://www.taobackup.com
[4] https://www.jwz.org/doc/backups.html
[5] http://backuppc.sourceforge.net


-- 
Jonathan Dowland
Please do not CC me, I am subscribed to the list.

[toc] | [prev] | [next] | [standalone]

#173147

[Multipart message — attachments visible in raw view] — view raw

Jonathan Dowland [2016-10-03 10:48:57+01] wrote:

> rsync, whilst an awesome piece of software, is not, on its own, a
> backup system.

Yes. With some scripting I think "rsync" with "--link-dest" is quite
ideal for incremental backups. Unchanged files are created as hard links
for the previous backup files. Every backup generation is just a normal
and complete file system tree.

-- 
/// Teemu Likonen   - .-..   <https://github.com/tlikonen> //
// PGP: 4E10 55DC 84E9 DFF6 13D7 8557 719D 69D3 2453 9450 ///

[toc] | [prev] | [next] | [standalone]

#173148

Hello Teemu,

> > rsync, whilst an awesome piece of software, is not, on its own, a
> > backup system.
>
> Yes. With some scripting I think "rsync" with "--link-dest" is quite
> ideal for incremental backups. Unchanged files are created as hard links
> for the previous backup files. Every backup generation is just a normal
> and complete file system tree.

I haven't followed this thread closely, but:

To everybody who ponders using rsync for backup, I strongly suggest a
closer look at "dirvish". I'm using it to backup servers, laptops,
raspberries. It is not hard to configure, uses rsync with hard links
for "incremental" backups and keeps older versions of the backups.

--
Markus Grunwald

Fragen zur Mail? http://www.the-grue.de/mail_und_co
http://www.the-grue.de/~markus/markus_grunwald.gpg

[toc] | [prev] | [next] | [standalone]

#173195

On 04/10/16 01:45, Markus Grunwald wrote:
> Hello Teemu,
>
>>> rsync, whilst an awesome piece of software, is not, on its own, a
>>> backup system.
>>
>> Yes. With some scripting I think "rsync" with "--link-dest" is quite
>> ideal for incremental backups. Unchanged files are created as hard
>> links for the previous backup files. Every backup generation is just
>> a normal and complete file system tree.
>
> I haven't followed this thread closely, but:
>
> To everybody who ponders using rsync for backup, I strongly suggest a
> closer look at "dirvish". I'm using it to backup servers, laptops,
> raspberries. It is not hard to configure, uses rsync with hard links
> for "incremental" backups and keeps older versions of the backups.

That's what I use too. I use it with rsync over ssh as root, with the
remote set up for root to run forced commands only, and in order to back
up different trees independently, I use different ssh keys, each with
its own forced command. That's a bit fiddly to set up, but seems to work ok.

On 04/10/16 04:06, Jonathan Dowland wrote:
> That's basically a less convenient rsnapshot, with all the caveats
> (bad performance for large trees of files like mailboxes)

Can you elaborate on the performance issues? I'm using dirvish for my
maildirs (dovecot imap server), without noticeable problems.

One change I do make is to enable 'dateext' in my logrotate config, so I
don't end up with endless duplicates of my logfiles due to the names
changing.

My current challenge is to back up windows boxes - if I can get rsync to
work (maybe DeltaCopy? Not sure if that will work how I want), I guess
I'll be stuck doing a local rsync of a smbfs mount ... unless someone
has a better suggestion.

Richard

[toc] | [prev] | [next] | [standalone]

#173196

[Multipart message — attachments visible in raw view] — view raw

Hi, Richard.

On 04/10/16 23:06, Richard Hector wrote:

> My current challenge is to back up windows boxes - if I can get rsync to
> work (maybe DeltaCopy? Not sure if that will work how I want), I guess
> I'll be stuck doing a local rsync of a smbfs mount ... unless someone
> has a better suggestion.

Some time ago I started a thread [1] on the list for this topic. The
Cygwin SSH server seems to run smoothly although I have not yet
implemented it on a daily basis.


Kind regards,
Daniel

[1] https://lists.debian.org/debian-user/2016/08/threads.html#00040

[toc] | [prev] | [next] | [standalone]

#173218

On 05/10/16 16:03, Daniel Bareiro wrote:
> Hi, Richard.
> 
> On 04/10/16 23:06, Richard Hector wrote:
> 
>> My current challenge is to back up windows boxes - if I can get
>> rsync to work (maybe DeltaCopy? Not sure if that will work how I
>> want), I guess I'll be stuck doing a local rsync of a smbfs mount
>> ... unless someone has a better suggestion.
> 
> Some time ago I started a thread [1] on the list for this topic.
> The Cygwin SSH server seems to run smoothly although I have not
> yet implemented it on a daily basis.

Ah, thanks. I think I even read it at the time, but didn't have the
immediate need, so didn't take so much in :-)

It depends a bit on how involved the setup is - the Windows machines
aren't mine (they're my parents'), and I don't want anything too
confusing for their regular Windows helper ... or for me, for that
matter; I don't touch Windows very often :-)

Richard

[toc] | [prev] | [next] | [standalone]

#173265

[Multipart message — attachments visible in raw view] — view raw

On Wed, Oct 05, 2016 at 03:06:58PM +1300, Richard Hector wrote:
> Can you elaborate on the performance issues? I'm using dirvish for my
> maildirs (dovecot imap server), without noticeable problems.

I don't know whether dirvish does something to improve matters, but with
hard link trees, if you have lots of little files (such as Maildir archives
of busy mailing lists like LKML), the amount of space consumed by the file
system metadata to represent the trees is very high, getting towards the
size used for the files themselves. This is compounded on some filesystems
(the ext family amongst them) that only ever increase the amount of space
for storing dirent metadata, never decrease it, so if you have a Maildir
which has high churn, and you move from having a large amount of mail to
a small amount, the storage allocated for metadata for the directory will
remain large.

On a low-powered device like an embedded NAS, performing operations like
delete (such as during a backup rotation) on a large hardlink tree can be
very slow and require a lot of CPU time.

> My current challenge is to back up windows boxes - if I can get rsync to
> work (maybe DeltaCopy? Not sure if that will work how I want), I guess
> I'll be stuck doing a local rsync of a smbfs mount ... unless someone
> has a better suggestion.

Windows has an rsync-like tool called Robocopy, but I'd skip straight to
Backup PC for backing up Windows machines.

-- 
Jonathan Dowland
Please do not CC me, I am subscribed to the list.

[toc] | [prev] | [next] | [standalone]

#173439

[Multipart message — attachments visible in raw view] — view raw

On 07/10/16 23:19, Jonathan Dowland wrote:
> I don't know whether dirvish does something to improve matters, but with
> hard link trees, if you have lots of little files (such as Maildir archives
> of busy mailing lists like LKML), the amount of space consumed by the file
> system metadata to represent the trees is very high, getting towards the
> size used for the files themselves. This is compounded on some filesystems
> (the ext family amongst them) that only ever increase the amount of space
> for storing dirent metadata, never decrease it, so if you have a Maildir
> which has high churn, and you move from having a large amount of mail to
> a small amount, the storage allocated for metadata for the directory will
> remain large.

The metadata you're talking about is merely the directory entries,
right? I'm still searching, but haven't yet found info on how much space
a directory entry takes up. Any pointers would be most welcome :-)

I'm on xfs, btw.

Regardless, in the trivial case where nothing has changed between
backups, the directories in the new backup must take up exactly the same
space as the ones in the old backup, where both point to the same files.
So whether there's one set of links or several, it's still the case that
for small files, the directory entries will take proportionately more
space relative to the file - that's inevitable.

I think I'm right in saying that in the dirvish (or similar) use case,
the number of directory entries in a given directory will never decrease
anyway - except in the odd case where I discover I've backed up
something I shouldn't have (eg a huge or confidential file), and gone
through purging it from all the backups.

Richard

[toc] | [prev] | [next] | [standalone]

#173151

[Multipart message — attachments visible in raw view] — view raw

On Mon, Oct 03, 2016 at 02:59:19PM +0300, Teemu Likonen wrote:
> Yes. With some scripting I think "rsync" with "--link-dest" is quite
> ideal for incremental backups. Unchanged files are created as hard links
> for the previous backup files. Every backup generation is just a normal
> and complete file system tree.

That's basically a less convenient rsnapshot, with all the caveats (bad
performance for large trees of files like mailboxes)


-- 
Jonathan Dowland
Please do not CC me, I am subscribed to the list.

[toc] | [prev] | [next] | [standalone]

#173406

On Mon, 3 Oct 2016 10:48:57 +0100
Jonathan Dowland <jmtd@debian.org> wrote:

> On Sat, Oct 01, 2016 at 11:37:31AM +0200, mo wrote:
> > Make a long story short:
> > Have you guys a recommendation for me?
> > Is there a specific application you use for your backups guys?
> 
> rdiff-backup[1]. I don't know what your NAS is, whether it's an off-the-shelf
> thing or a DIY system, but I tend to initiate the backups on my NAS (which is
> a Debian-powered mini itx PC). You can initiate from the clients instead if
> you prefer.
> 
> rdiff-backup is, conceptually, quite similar to the level of detail you are
> working at with your script, so it should be easy to get working. I'd also
> consider looking at Obnam[2], and I'd avoid rsnapshot if I were you (it
> falls over with large quantities of files, such as mailboxes, and rdiff-backup
> basically does the same job but better).

Interesting, thanks. I've been using rsnapshot for years, and am
basically satisfied with it, although the performance when run on my
T61 laptop (backing up to a (slow) USB external disk) is indeed painful
(I do have largeish Maildirs). [Interestingly, when run on my ARM
(Kirkland) NAS, pulling from the laptop over ethernet / wifi,
performance seems much better ...]

One thing I really like about rsnapshot is how the backups are all
stored just as the files themselves, without any special formats, and
can therefore be inspected / restored from using just the ordinary
filesystem tools.

A number of years ago I looked at rdiff-backup, but dropped it due to
suspicions of the code quality: the project seemed to be dead, and even
significant bugs were being ignored:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=623336
https://lists.debian.org/debian-user/2012/10/msg00182.html

I see that you, too, have a report that's been ignored for more than 5
years ;)

Am I being unreasonable? You are certainly more of an expert than I - I
suppose you find that it is quality software, and better than
rsnapshot, despite basically being dead?

Celejar

[toc] | [prev] | [next] | [standalone]

#173411

[Multipart message — attachments visible in raw view] — view raw

On Mon, Oct 10, 2016 at 09:32:02AM -0400, Celejar wrote:
> Interesting, thanks. I've been using rsnapshot for years, and am
> basically satisfied with it, although the performance when run on my
> T61 laptop (backing up to a (slow) USB external disk) is indeed painful
> (I do have largeish Maildirs). [Interestingly, when run on my ARM
> (Kirkland) NAS, pulling from the laptop over ethernet / wifi,
> performance seems much better ...]

The first machine where I really had performance problems
with rsnapshot was an ARM-powered Thecus N2100, but that was a pretty
weak CPU from what I recall.

> One thing I really like about rsnapshot is how the backups are all
> stored just as the files themselves, without any special formats, and
> can therefore be inspected / restored from using just the ordinary
> filesystem tools.

That is a nice property, yes. the rdiff-backup format is not quite this,
but is well specified outside of the code (from the last time I looked)
to give me confidence I could pull my files out by hand if I needed to.

I'm not sure if I mentioned it in this thread or not, but I actually began a
third party tool to parse rdiff-backup format backups[1], and there exists
another third party tool to do the same thing[2].

> A number of years ago I looked at rdiff-backup, but dropped it due to
> suspicions of the code quality: the project seemed to be dead, and even
> significant bugs were being ignored:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=623336
> https://lists.debian.org/debian-user/2012/10/msg00182.html
> 
> I see that you, too, have a report that's been ignored for more than 5
> years ;)

Yes, that's true. The bug that I remember reporting (haven't looked it back up
to be sure) was that reverting a partial (failed) backup requires some disk
space and so fails if the backup device is full - this is a big problem in
theory, but in practice I'm running my backup jobs as non-root, so there's
always the 5% or so reserved space for root users. So in this situation I
can have the roll-back occur as root.

> Am I being unreasonable? You are certainly more of an expert than I - I
> suppose you find that it is quality software, and better than
> rsnapshot, despite basically being dead?
 
Lack of bug and development activity is certainly a worrying sign of a program
with problems. But development activity naturally dies off if a program does
what it is supposed to do, so it's not always a certainty that it is doomed.
It remains a concern for me, but regardless the tool has worked very well.

[1] https://jmtd.net/software/rdifffs/
[2] https://github.com/rbrito/rdiff-backup-fs

-- 
Jonathan Dowland
Please do not CC me, I am subscribed to the list.

[toc] | [prev] | [next] | [standalone]

#173415

On Mon, Oct 10, 2016 at 10:55:44PM +0100, Jonathan Dowland wrote:
> On Mon, Oct 10, 2016 at 09:32:02AM -0400, Celejar wrote:
> > Am I being unreasonable? You are certainly more of an expert than I - I
> > suppose you find that it is quality software, and better than
> > rsnapshot, despite basically being dead?
>  
> Lack of bug and development activity is certainly a worrying sign of a program
> with problems. But development activity naturally dies off if a program does
> what it is supposed to do, so it's not always a certainty that it is doomed.
> It remains a concern for me, but regardless the tool has worked very well.
> 

It looks like rdiff-backup has a new maintainer as of February 2016, sol1. See
<http://rdiff-backup.org/>. There may be some progress forthcoming
after all. 

dt

-- 
Dave Thayer / Denver, Colorado USA / dave@thayer-boyle.com 
 Whenever you read a good book, it's like the author is right there, in
 the room talking to you, which is why I don't like to read good books.
 - Jack Handey "Deep Thoughts"

[toc] | [prev] | [next] | [standalone]

#173187

I have moved from simple scripts to simple scripts with zbackup in them :)

Then I rsync the zbackup directories from different machines to my 
central  backup disks (and distribute from there to cloud storage and 
off-site disks).

zbackup supports deduplication and encryption, and is really a nice 
piece of software.


Jarle


Den 01. okt. 2016 12:37, skrev mo:
> Hi Debian users :)
>
> Information:
> Distributor ID:    Debian
> Description:    Debian GNU/Linux 8.6 (jessie)
> Release:    8.6
> Codename:    jessie
>
> As the title say i'm in search for a backup application/system.
> Currently i manage my backups with a little script that i wrote... but 
> it does not really serve my needs anymore.
> I want to be able to make backups on my main PC and also on my server, 
> the backups i would then store on my NAS.
>
> Make a long story short:
> Have you guys a recommendation for me?
> Is there a specific application you use for your backups guys?
>
> Btw: I dont mind configuring or playing around with new applications, 
> every recommendation is welcome ;)
>
>
> Here is my current backup script (Which is run by cron daily):
> #!/bin/bash
>
> TO_BACKUP="/home /etc /var/log"
> BACKUP_DIR="/var/backup"
> BACKUP_ARCHIVE="backup-`date +%d_%m_%Y-%H:%M`.tar"
> TAR_OPTIONS='-cpf'
>
> delete_old_backup() {
>     if [ -f ${BACKUP_DIR}/backup*.tar ]; then
>         rm -rf $BACKUP_DIR/backup*
>     fi
> }
>
> create_new_backup() {
>     tar $TAR_OPTIONS ${BACKUP_DIR}/$BACKUP_ARCHIVE $TO_BACKUP
> }
>
> main() {
>     delete_old_backup
>     create_new_backup
> }
>
> main
>
> Greets
> mo
>

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.debian.user

csiph-web