Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #159540 > unrolled thread

Backup solutions without reinventig the wheel these days

Back to article view | Back to linux.debian.user

Contents

  Backup solutions without reinventig the wheel these days Ondřej Grover <ondrej.grover@gmail.com> - 2015-10-20 19:00 +0200
    Re: Backup solutions without reinventig the wheel these days Mario Castelán Castro  <marioxcc.MT@yandex.com> - 2015-10-20 19:50 +0200
      Re: Backup solutions without reinventig the wheel these days Teemu Likonen <tlikonen@iki.fi> - 2015-10-20 20:20 +0200
        Re: Backup solutions without reinventig the wheel these days Jonathan Dowland <jmtd@debian.org> - 2015-10-27 12:40 +0100
          Re: Backup solutions without reinventig the wheel these days Teemu Likonen <tlikonen@iki.fi> - 2015-10-27 16:40 +0100
    Re: Backup solutions without reinventig the wheel these days Sarunas Burdulis <sarunas@math.dartmouth.edu> - 2015-10-20 21:10 +0200
      Re: Backup solutions without reinventig the wheel these days Ondřej Grover <ondrej.grover@gmail.com> - 2015-10-21 11:40 +0200
        Re: Backup solutions without reinventig the wheel these days mourik jan heupink <heupink@merit.unu.edu> - 2015-10-21 13:30 +0200
          Re: Backup solutions without reinventig the wheel these days mourik jan heupink <heupink@merit.unu.edu> - 2015-10-21 13:40 +0200
        Re: Backup solutions without reinventig the wheel these days Eduardo M KALINOWSKI <eduardo@kalinowski.com.br> - 2015-10-21 13:30 +0200
        Re: Backup solutions without reinventig the wheel these days Dan Ritter <dsr@randomstring.org> - 2015-10-21 16:10 +0200
        Re: Backup solutions without reinventig the wheel these days Sarunas Burdulis <sarunas@math.dartmouth.edu> - 2015-10-21 16:30 +0200
          Re: Backup solutions without reinventig the wheel these days Ondřej Grover <ondrej.grover@gmail.com> - 2015-10-21 16:50 +0200
            Re: Backup solutions without reinventig the wheel these days Sarunas Burdulis <sarunas@math.dartmouth.edu> - 2015-10-21 17:10 +0200
              Re: Backup solutions without reinventig the wheel these days Ondřej Grover <ondrej.grover@gmail.com> - 2015-10-22 08:50 +0200
                Re: Backup solutions without reinventig the wheel these days Sarunas Burdulis <sarunas@math.dartmouth.edu> - 2015-10-22 16:10 +0200
    Re: Backup solutions without reinventig the wheel these days Glenn English <ghe@slsware.net> - 2015-10-22 18:30 +0200

#159540 — Backup solutions without reinventig the wheel these days

[Multipart message — attachments visible in raw view] — view raw

Hello,

I'm looking for recommendations for backup solutions that don't reinvent
the wheel and are reliable and used. I want to backup two servers to a
backup server. The main data content is several hundred GB in many very
small files.

I really like the idea behind backupninja, because it provides a
centralized solution to the cron + ssh transfer (rsync) + mail paradigm and
elevates the need to write one's own elaborate scripts. It also provides
the most common backup helper scripts with sensible defaults. The mail
reporting part isn't that great (does not offer consistent logging of data
transfer solutions), but that can be fixed with a few custom shell scripts.

However, I found that for my use-case rdiff-backup runs out of memory on
the backup server (1GB RAM + 1GB swap) and duplicity creates an over 50 GB
signature file. I could use just simple rsync, but incremental +
compression would be a nice feature as data corruption may not become
apparent immediately.

I've also looked at the new kids on the block like obnam, attick and
borgbackup. They look interesting, but I prefer time-tested SW for backups.
After realizing that these new backup programs pretty much try to replicate
features of btrfs or ZFS (incremental snapshots, block-level compression
and deduplication) I started thinking that I could perhaps just send the
data to the backup server via rsync and save them to a btrfs or ZFS (but
the backup server may not have enough RAM for ZFS) and create daily
snaphosts on the server. If memory will permit (if I optimize it), I'd go
with ZFS as it should be more reliable. Does anybody use such a solution?

I also had a look at Bacula, but it seemed that it does not offer
block-level deduplication and compression at the moment.

I'm looking forward to your recommendations.

Kind regards,
Ondřej Grover

[toc] | [next] | [standalone]

#159542

Take a look at the "--link-dest" option of rsync; it provides 
deduplication at the file level. That may work well for storing 
snapshots more efficiently in your use case given that your data 
consists of many very small files. Since it uses hardlinks, it can't 
deduplicate among files that have the same content but different 
filesystem level metadata for those attributes that rsync is instructed 
to keep (for a backup those usually include permissions, user, group, 
modification date).

You can create one directory for each snapshot within a directory for 
backups, then each time you add a new snapshot, pass the previous 
snapshot to "--link-dest".

For example, if you store your backups under directory "/backups", and 
the previous backup is under directory "/backups/2015-10-13", then to 
make a new backup for today, use "rsync [OTHER-OPTIONS] 
--link-dest=/backups/2015-10-13 [SOURCE] /backups/2015-10-20". You may 
use "--link-dest" several times with different directories.

To be able to detect corruption of the backup (as opposed to corruption 
of the live data), compute hashes of the files. You can use "find 
-print0 -type f | xargs -0 sha256sum > [HASHES-FILE]". Take a hash of 
this list of hasehes and store it in at least 2 places, so that you will 
be able to detect and distinguish corruption of the list of hashes from 
corruption of the snapshots.

Regards.

El 20/10/15 a las 11:57, Ondřej Grover escribió:
> Hello,
>
> I'm looking for recommendations for backup solutions that don't reinvent
> the wheel and are reliable and used. I want to backup two servers to a
> backup server. The main data content is several hundred GB in many very
> small files.
>
> I really like the idea behind backupninja, because it provides a
> centralized solution to the cron + ssh transfer (rsync) + mail paradigm and
> elevates the need to write one's own elaborate scripts. It also provides
> the most common backup helper scripts with sensible defaults. The mail
> reporting part isn't that great (does not offer consistent logging of data
> transfer solutions), but that can be fixed with a few custom shell scripts.
>
> However, I found that for my use-case rdiff-backup runs out of memory on
> the backup server (1GB RAM + 1GB swap) and duplicity creates an over 50 GB
> signature file. I could use just simple rsync, but incremental +
> compression would be a nice feature as data corruption may not become
> apparent immediately.
>
> I've also looked at the new kids on the block like obnam, attick and
> borgbackup. They look interesting, but I prefer time-tested SW for backups.
> After realizing that these new backup programs pretty much try to replicate
> features of btrfs or ZFS (incremental snapshots, block-level compression
> and deduplication) I started thinking that I could perhaps just send the
> data to the backup server via rsync and save them to a btrfs or ZFS (but
> the backup server may not have enough RAM for ZFS) and create daily
> snaphosts on the server. If memory will permit (if I optimize it), I'd go
> with ZFS as it should be more reliable. Does anybody use such a solution?
>
> I also had a look at Bacula, but it seemed that it does not offer
> block-level deduplication and compression at the moment.
>
> I'm looking forward to your recommendations.
>
> Kind regards,
> Ondřej Grover
>

[toc] | [prev] | [next] | [standalone]

#159544

[Multipart message — attachments visible in raw view] — view raw

Mario Castelán Castro [2015-10-20 12:39:59-05] wrote:

> Take a look at the "--link-dest" option of rsync; it provides
> deduplication at the file level.

> You can create one directory for each snapshot within a directory for
> backups, then each time you add a new snapshot, pass the previous
> snapshot to "--link-dest".

I suggest "rsync --link-dest" too. Symbolic links are a nice way to
store the previous (and currently newest) backup destination directory.
So maybe like this in the backup destination machine:


time_stamp=$(date +%Y%m%dT%H%M%S%z)
destination_base_dir=some_directory

rsync -aH --link-dest="$destination_base_dir/newest" \
    [backup source] "$destination_base_dir/$time_stamp"

if [ $? -eq 0 ]; then
	# Update the symbolic link to point the just created backup dir:
    ln -sfn -- "$time_stamp" "$destination_base_dir/newest"
    # Delete old backups:
    find "$destination_base_dir" -mindepth 1 -maxdepth 1 -type d \
        -mtime +90 -exec /bin/rm -fr -- {} +
fi

[toc] | [prev] | [next] | [standalone]

#159849

On Tue, Oct 20, 2015 at 09:10:54PM +0300, Teemu Likonen wrote:
> I suggest "rsync --link-dest" too. Symbolic links are a nice way to
> store the previous (and currently newest) backup destination directory.
> So maybe like this in the backup destination machine:

--link-dest doesn't create symbolic links, it uses hard links.

If you have a lot of file metadata (due to lots of files), and you want to
store lots of increments, and you want those increments to be fully expanded on
the filesystem, such as via rsnapshot or --link-dest, then your filesystem
metadata (dentries, inodes etc) are going to be large.

[toc] | [prev] | [next] | [standalone]

#159859

[Multipart message — attachments visible in raw view] — view raw

Jonathan Dowland [2015-10-27 10:53:25Z] wrote:

> On Tue, Oct 20, 2015 at 09:10:54PM +0300, Teemu Likonen wrote:
>> I suggest "rsync --link-dest" too. Symbolic links are a nice way to
>> store the previous (and currently newest) backup destination
>> directory. So maybe like this in the backup destination machine:
>
> --link-dest doesn't create symbolic links, it uses hard links.

I know. In my message I explained how we, script-writers, can use
symbolic links to maintain pointer to the newest backup directory made
by "rsync --link-dest". That's how the next backup (rsync invocation)
can find its --link-dest target, for example.

[toc] | [prev] | [next] | [standalone]

#159547

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 10/20/2015 12:57 PM, Ondřej Grover wrote:
> Hello,
> 
> I'm looking for recommendations for backup solutions that don't reinvent
> the wheel and are reliable and used. I want to backup two servers to a
> backup server. The main data content is several hundred GB in many very
> small files.
> 
> [...]
> 
> I've also looked at the new kids on the block like obnam, attick and
> borgbackup. They look interesting, but I prefer time-tested SW for backups.
> After realizing that these new backup programs pretty much try to
> replicate features of btrfs or ZFS (incremental snapshots, block-level
> compression and deduplication) I started thinking that I could perhaps
> just send the data to the backup server via rsync and save them to a
> btrfs or ZFS (but the backup server may not have enough RAM for ZFS) and
> create daily snaphosts on the server. If memory will permit (if I
> optimize it), I'd go with ZFS as it should be more reliable. Does
> anybody use such a solution?

Yes, we do use pretty much the setup you describe, i.e. rsync push to
backup server, btrfs volume with daily snapshot subvolumes on the
server. This particular setup has been in use for more than two years by
now, and is working well with 8-10 servers (totals for all backed-up
filesystems is about 1.2TiB/2.5M files of varying size; transferred
daily delta is of course just a fraction of this). The backup server is
a rather modest self-built Mini-ITX Core i5 with 8GiB RAM and 6TiB
hardware (3ware) RAID-10 storage volume.

Using btrfs volume in btrfs-raid configuration (raid-1 for data and
metadata) would provide protection against data bit-rot. We use several
such volumes on some of our severs, though not in combination with btrfs
snapshot subvolumes.

- -- 
Sarunas Burdulis
http://math.dartmouth.edu/~sarunas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWJollAAoJEAcPjabqyGA9iD4H/2Hxa4IqYxRRLi55kg8c/tOS
UafxgMMnUB498HHVGMldDZdjqmzBd4djHuVOeJPguu7RaSVKOIBBdpmZdI44LXoj
y3P0m7Ig1cg3kGB7tZTgG6mzsgZQln6F8uW/lNox9EikmWUQ2eQZYIDVbLTEOLXt
pMKVTKQeR1Jdl9HvbIqlPBpniqHn+JvesWUeTDdo47i45jcfRonYfPay0/ua7ql0
n80ObcOLlYSCw/rKZbkspHz7k9PMEEdeosbkvUkw2ZiGq13E1j6DFwSUTxcrMPhR
Ffd+bL5c0CtKOTVrvtZdahJZ5U/Kl/QG1c78TsJagCN7YTSXRWnDSn6acdDRPmM=
=Edp3
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#159563

[Multipart message — attachments visible in raw view] — view raw

--lnk-dest is a nice feature of rsync, however, it would require me to
write yet another custom script that so many people already have and
reinvent the wheel again. I'm trying to find a ready and time-tested and
maintained solution before I go down that path. The rsynch handler in
backupninja does pretty much the same thing, but uses cp -l as --link-dest
did not exist back when it was written. Nevertheless, I'd prefer
block-level deduplication and compression.

Your btrfs setup sounds interesting, Sarunas. Are you running those btrfs
volumes on Jessie? Any stability issues? Do you think it could work with
just 1GB RAM + 1 GB swap? I was thinking about doing out-of-band
deduplication to conserve RAM. Do you use both compression and
deduplication (file or block level)? I heard they don't play nicely
together. Could you share your mount options?

Kind regards,
Ondřej Grover

On Tue, Oct 20, 2015 at 8:35 PM, Sarunas Burdulis <
sarunas@math.dartmouth.edu> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 10/20/2015 12:57 PM, Ondřej Grover wrote:
> > Hello,
> >
> > I'm looking for recommendations for backup solutions that don't reinvent
> > the wheel and are reliable and used. I want to backup two servers to a
> > backup server. The main data content is several hundred GB in many very
> > small files.
> >
> > [...]
> >
> > I've also looked at the new kids on the block like obnam, attick and
> > borgbackup. They look interesting, but I prefer time-tested SW for
> backups.
> > After realizing that these new backup programs pretty much try to
> > replicate features of btrfs or ZFS (incremental snapshots, block-level
> > compression and deduplication) I started thinking that I could perhaps
> > just send the data to the backup server via rsync and save them to a
> > btrfs or ZFS (but the backup server may not have enough RAM for ZFS) and
> > create daily snaphosts on the server. If memory will permit (if I
> > optimize it), I'd go with ZFS as it should be more reliable. Does
> > anybody use such a solution?
>
> Yes, we do use pretty much the setup you describe, i.e. rsync push to
> backup server, btrfs volume with daily snapshot subvolumes on the
> server. This particular setup has been in use for more than two years by
> now, and is working well with 8-10 servers (totals for all backed-up
> filesystems is about 1.2TiB/2.5M files of varying size; transferred
> daily delta is of course just a fraction of this). The backup server is
> a rather modest self-built Mini-ITX Core i5 with 8GiB RAM and 6TiB
> hardware (3ware) RAID-10 storage volume.
>
> Using btrfs volume in btrfs-raid configuration (raid-1 for data and
> metadata) would provide protection against data bit-rot. We use several
> such volumes on some of our severs, though not in combination with btrfs
> snapshot subvolumes.
>
> - --
> Sarunas Burdulis
> http://math.dartmouth.edu/~sarunas
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iQEcBAEBCAAGBQJWJollAAoJEAcPjabqyGA9iD4H/2Hxa4IqYxRRLi55kg8c/tOS
> UafxgMMnUB498HHVGMldDZdjqmzBd4djHuVOeJPguu7RaSVKOIBBdpmZdI44LXoj
> y3P0m7Ig1cg3kGB7tZTgG6mzsgZQln6F8uW/lNox9EikmWUQ2eQZYIDVbLTEOLXt
> pMKVTKQeR1Jdl9HvbIqlPBpniqHn+JvesWUeTDdo47i45jcfRonYfPay0/ua7ql0
> n80ObcOLlYSCw/rKZbkspHz7k9PMEEdeosbkvUkw2ZiGq13E1j6DFwSUTxcrMPhR
> Ffd+bL5c0CtKOTVrvtZdahJZ5U/Kl/QG1c78TsJagCN7YTSXRWnDSn6acdDRPmM=
> =Edp3
> -----END PGP SIGNATURE-----
>
>

[toc] | [prev] | [next] | [standalone]

#159564

>     Yes, we do use pretty much the setup you describe, i.e. rsync push to
>     backup server, btrfs volume with daily snapshot subvolumes on the
>     server. This particular setup has been in use for more than two years by
>     now, and is working well with 8-10 servers (totals for all backed-up
>     filesystems is about 1.2TiB/2.5M files of varying size; transferred

This sounds exactly as the solution we're using. Only we backup up to a 
freenas device, using rsync, zfs and it's snapshots. Using 4*6TB WD 
disks in a raidz2 config. Allowing for two disks to die at the same time 
without data loss. The freenas gui makes configuring everything simple.

I must say: zfs is absolutely super.

[toc] | [prev] | [next] | [standalone]

#159566

On 10/21/2015 01:17 PM, mourik jan heupink wrote:
> This sounds exactly as the solution we're using. Only we backup up to
> a freenas device, using rsync, zfs and it's snapshots. Using 4*6TB WD
>  disks in a raidz2 config. Allowing for two disks to die at the same
> time without data loss. The freenas gui makes configuring everything
> simple.
>
> I must say: zfs is absolutely super.
We also use the freenas/zfs replication option to replicate our backups
to another location. (so this setup gives us on- and offsite backups)

[toc] | [prev] | [next] | [standalone]

#159565

On Qua, 21 Out 2015, Ondřej Grover wrote:

> --lnk-dest is a nice feature of rsync, however, it would require me to
> write yet another custom script that so many people already have and
> reinvent the wheel again. I'm trying to find a ready and time-tested and
> maintained solution before I go down that path.

Take a look at rsnapshot. It's a wrapper for rsync with --lnk-dest.

-- 
Eduardo M KALINOWSKI
eduardo@kalinowski.com.br

[toc] | [prev] | [next] | [standalone]

#159573

On Wed, Oct 21, 2015 at 11:32:06AM +0200, Ondřej Grover wrote:
> --lnk-dest is a nice feature of rsync, however, it would require me to
> write yet another custom script that so many people already have and
> reinvent the wheel again. I'm trying to find a ready and time-tested and
> maintained solution before I go down that path. The rsynch handler in
> backupninja does pretty much the same thing, but uses cp -l as --link-dest
> did not exist back when it was written. Nevertheless, I'd prefer
> block-level deduplication and compression.
> 
> Your btrfs setup sounds interesting, Sarunas. Are you running those btrfs
> volumes on Jessie? Any stability issues? Do you think it could work with
> just 1GB RAM + 1 GB swap? I was thinking about doing out-of-band
> deduplication to conserve RAM. Do you use both compression and
> deduplication (file or block level)? I heard they don't play nicely
> together. Could you share your mount options?

Deduplication is a RAM hog on zfs and I don't know anybody using
it on btrfs.

Why does your backup server only have 1 GB RAM?

-dsr-

[toc] | [prev] | [next] | [standalone]

#159577

On 10/21/2015 05:32 AM, Ondřej Grover wrote:
> [...]
> Your btrfs setup sounds interesting, Sarunas. Are you running those
> btrfs volumes on Jessie? Any stability issues? Do you think it could
> work with just 1GB RAM + 1 GB swap? I was thinking about doing
> out-of-band deduplication to conserve RAM. Do you use both compression
> and deduplication (file or block level)? I heard they don't play nicely
> together. Could you share your mount options?

Our backup server is actually running Linux 3.13.0-66-generic, Btrfs
v3.12 — Ubuntu 14.04.3 LTS, which is behind Jessie in software versions.
It has been very stable — for the past too years it's running as a
headless box in an off-site location.

We don't use compression or deduplication. In-band dedup is not in btrfs
3.*, and I haven't tried any out-of-band tools. Nothing particular in
Btrfs mount, the only non-default options in 'noatime':

LABEL=BACKUPS /backups btrfs noatime 0 0

1GB RAM sounds low by today's habits. That doesn't mean it won't work,
but just begs for RAM upgrade.

Sarunas

> On Tue, Oct 20, 2015 at 8:35 PM, Sarunas Burdulis
> <sarunas@math.dartmouth.edu <mailto:sarunas@math.dartmouth.edu>> wrote:
> 
> On 10/20/2015 12:57 PM, Ondřej Grover wrote:
>> Hello,
> 
>> I'm looking for recommendations for backup solutions that don't reinvent
>> the wheel and are reliable and used. I want to backup two servers to a
>> backup server. The main data content is several hundred GB in many very
>> small files.
> 
>> [...]
> 
>> I've also looked at the new kids on the block like obnam, attick and
>> borgbackup. They look interesting, but I prefer time-tested SW for backups.
>> After realizing that these new backup programs pretty much try to
>> replicate features of btrfs or ZFS (incremental snapshots, block-level
>> compression and deduplication) I started thinking that I could perhaps
>> just send the data to the backup server via rsync and save them to a
>> btrfs or ZFS (but the backup server may not have enough RAM for ZFS) and
>> create daily snaphosts on the server. If memory will permit (if I
>> optimize it), I'd go with ZFS as it should be more reliable. Does
>> anybody use such a solution?
> 
> Yes, we do use pretty much the setup you describe, i.e. rsync push to
> backup server, btrfs volume with daily snapshot subvolumes on the
> server. This particular setup has been in use for more than two years by
> now, and is working well with 8-10 servers (totals for all backed-up
> filesystems is about 1.2TiB/2.5M files of varying size; transferred
> daily delta is of course just a fraction of this). The backup server is
> a rather modest self-built Mini-ITX Core i5 with 8GiB RAM and 6TiB
> hardware (3ware) RAID-10 storage volume.
> 
> Using btrfs volume in btrfs-raid configuration (raid-1 for data and
> metadata) would provide protection against data bit-rot. We use several
> such volumes on some of our severs, though not in combination with btrfs
> snapshot subvolumes.
> 
> --
> Sarunas Burdulis
> http://math.dartmouth.edu/~sarunas

[toc] | [prev] | [next] | [standalone]

#159581

[Multipart message — attachments visible in raw view] — view raw

I could try to request a RAM upgrade (not super easy in uni environment,
but doable), but it goes against my understanding of what a storage server
is. In my book, a storage server needs fast IO capabilities (storage drives
and NIC) and possibly CPU for compression. Deduplication is the only reason
I could think of higher RAM requirements on a storage server.

Sarunas, did you have good experiences with actually using the backups for
restoring?

On Wed, Oct 21, 2015 at 4:28 PM, Sarunas Burdulis <
sarunas@math.dartmouth.edu> wrote:

> On 10/21/2015 05:32 AM, Ondřej Grover wrote:
> > [...]
> > Your btrfs setup sounds interesting, Sarunas. Are you running those
> > btrfs volumes on Jessie? Any stability issues? Do you think it could
> > work with just 1GB RAM + 1 GB swap? I was thinking about doing
> > out-of-band deduplication to conserve RAM. Do you use both compression
> > and deduplication (file or block level)? I heard they don't play nicely
> > together. Could you share your mount options?
>
> Our backup server is actually running Linux 3.13.0-66-generic, Btrfs
> v3.12 — Ubuntu 14.04.3 LTS, which is behind Jessie in software versions.
> It has been very stable — for the past too years it's running as a
> headless box in an off-site location.
>
> We don't use compression or deduplication. In-band dedup is not in btrfs
> 3.*, and I haven't tried any out-of-band tools. Nothing particular in
> Btrfs mount, the only non-default options in 'noatime':
>
> LABEL=BACKUPS /backups btrfs noatime 0 0
>
> 1GB RAM sounds low by today's habits. That doesn't mean it won't work,
> but just begs for RAM upgrade.
>
> Sarunas
>
> > On Tue, Oct 20, 2015 at 8:35 PM, Sarunas Burdulis
> > <sarunas@math.dartmouth.edu <mailto:sarunas@math.dartmouth.edu>> wrote:
> >
> > On 10/20/2015 12:57 PM, Ondřej Grover wrote:
> >> Hello,
> >
> >> I'm looking for recommendations for backup solutions that don't reinvent
> >> the wheel and are reliable and used. I want to backup two servers to a
> >> backup server. The main data content is several hundred GB in many very
> >> small files.
> >
> >> [...]
> >
> >> I've also looked at the new kids on the block like obnam, attick and
> >> borgbackup. They look interesting, but I prefer time-tested SW for
> backups.
> >> After realizing that these new backup programs pretty much try to
> >> replicate features of btrfs or ZFS (incremental snapshots, block-level
> >> compression and deduplication) I started thinking that I could perhaps
> >> just send the data to the backup server via rsync and save them to a
> >> btrfs or ZFS (but the backup server may not have enough RAM for ZFS) and
> >> create daily snaphosts on the server. If memory will permit (if I
> >> optimize it), I'd go with ZFS as it should be more reliable. Does
> >> anybody use such a solution?
> >
> > Yes, we do use pretty much the setup you describe, i.e. rsync push to
> > backup server, btrfs volume with daily snapshot subvolumes on the
> > server. This particular setup has been in use for more than two years by
> > now, and is working well with 8-10 servers (totals for all backed-up
> > filesystems is about 1.2TiB/2.5M files of varying size; transferred
> > daily delta is of course just a fraction of this). The backup server is
> > a rather modest self-built Mini-ITX Core i5 with 8GiB RAM and 6TiB
> > hardware (3ware) RAID-10 storage volume.
> >
> > Using btrfs volume in btrfs-raid configuration (raid-1 for data and
> > metadata) would provide protection against data bit-rot. We use several
> > such volumes on some of our severs, though not in combination with btrfs
> > snapshot subvolumes.
> >
> > --
> > Sarunas Burdulis
> > http://math.dartmouth.edu/~sarunas
>
>

[toc] | [prev] | [next] | [standalone]

#159583

On 10/21/2015 10:47 AM, Ondřej Grover wrote:
> I could try to request a RAM upgrade (not super easy in uni environment,
> but doable), but it goes against my understanding of what a storage
> server is. In my book, a storage server needs fast IO capabilities
> (storage drives and NIC) and possibly CPU for compression. Deduplication
> is the only reason I could think of higher RAM requirements on a storage
> server.

This morning I added a 'memory' plugin to the Munin node on the server I
described. Here is a snaphot so far, for what it's worth. The bump at
9:50 is an rsync from one of the biggest filesystems backed-up (~600GiB,
about a million files to sync).

https://math.dartmouth.edu/owncloud/index.php/s/E3VYxhWlCqUusn4

> Sarunas, did you have good experiences with actually using the backups
> for restoring? 

Whenever I needed them, backup copies were available. This is more of an
archival storage. The main point for using btrfs was instantaneous
snapshot creation (and removal of older ones, something that takes
'forever' in rsync --link-dest scenario, using rsnapshot, for example).

Sarunas

> On Wed, Oct 21, 2015 at 4:28 PM, Sarunas Burdulis
> <sarunas@math.dartmouth.edu <mailto:sarunas@math.dartmouth.edu>> wrote:
> 
>     On 10/21/2015 05:32 AM, Ondřej Grover wrote:
>     > [...]
>     > Your btrfs setup sounds interesting, Sarunas. Are you running those
>     > btrfs volumes on Jessie? Any stability issues? Do you think it could
>     > work with just 1GB RAM + 1 GB swap? I was thinking about doing
>     > out-of-band deduplication to conserve RAM. Do you use both compression
>     > and deduplication (file or block level)? I heard they don't play nicely
>     > together. Could you share your mount options?
> 
>     Our backup server is actually running Linux 3.13.0-66-generic, Btrfs
>     v3.12 — Ubuntu 14.04.3 LTS, which is behind Jessie in software versions.
>     It has been very stable — for the past too years it's running as a
>     headless box in an off-site location.
> 
>     We don't use compression or deduplication. In-band dedup is not in btrfs
>     3.*, and I haven't tried any out-of-band tools. Nothing particular in
>     Btrfs mount, the only non-default options in 'noatime':
> 
>     LABEL=BACKUPS /backups btrfs noatime 0 0
> 
>     1GB RAM sounds low by today's habits. That doesn't mean it won't work,
>     but just begs for RAM upgrade.
> 
>     Sarunas
> 
>     > On Tue, Oct 20, 2015 at 8:35 PM, Sarunas Burdulis
>     > <sarunas@math.dartmouth.edu <mailto:sarunas@math.dartmouth.edu>
>     <mailto:sarunas@math.dartmouth.edu
>     <mailto:sarunas@math.dartmouth.edu>>> wrote:
>     >
>     > On 10/20/2015 12:57 PM, Ondřej Grover wrote:
>     >> Hello,
>     >
>     >> I'm looking for recommendations for backup solutions that don't
>     reinvent
>     >> the wheel and are reliable and used. I want to backup two servers
>     to a
>     >> backup server. The main data content is several hundred GB in
>     many very
>     >> small files.
>     >
>     >> [...]
>     >
>     >> I've also looked at the new kids on the block like obnam, attick and
>     >> borgbackup. They look interesting, but I prefer time-tested SW
>     for backups.
>     >> After realizing that these new backup programs pretty much try to
>     >> replicate features of btrfs or ZFS (incremental snapshots,
>     block-level
>     >> compression and deduplication) I started thinking that I could
>     perhaps
>     >> just send the data to the backup server via rsync and save them to a
>     >> btrfs or ZFS (but the backup server may not have enough RAM for
>     ZFS) and
>     >> create daily snaphosts on the server. If memory will permit (if I
>     >> optimize it), I'd go with ZFS as it should be more reliable. Does
>     >> anybody use such a solution?
>     >
>     > Yes, we do use pretty much the setup you describe, i.e. rsync push to
>     > backup server, btrfs volume with daily snapshot subvolumes on the
>     > server. This particular setup has been in use for more than two
>     years by
>     > now, and is working well with 8-10 servers (totals for all backed-up
>     > filesystems is about 1.2TiB/2.5M files of varying size; transferred
>     > daily delta is of course just a fraction of this). The backup
>     server is
>     > a rather modest self-built Mini-ITX Core i5 with 8GiB RAM and 6TiB
>     > hardware (3ware) RAID-10 storage volume.
>     >
>     > Using btrfs volume in btrfs-raid configuration (raid-1 for data and
>     > metadata) would provide protection against data bit-rot. We use
>     several
>     > such volumes on some of our severs, though not in combination with
>     btrfs
>     > snapshot subvolumes.
>     >
>     > --
>     > Sarunas Burdulis
>     > http://math.dartmouth.edu/~sarunas
> 
> 


-- 
Sarunas Burdulis
Systems Administrator
Department of Mathematics, Dartmouth College
http://math.dartmouth.edu/~sarunas

[toc] | [prev] | [next] | [standalone]

#159623

[Multipart message — attachments visible in raw view] — view raw

Thank you Sarunas for the graphs, it is most illuminating. That's pretty
much how I imagine a storage server should behave.
Atomic snapshots are very appealing indeed. In my case cp -lr or
--link-dest solutions would take a long time as the FS hierarchy is both
very broad and deep.

So far it seems I will go with backupninja with simple rsync transport and
a few custom shell scripts to create and delete btrfs snapshots on the
remote server. This means I will need root access on the backup server, but
I can restrict that with authorized_keys or sudo.

Is there some other framework similar to backupninja that provides the
common boilerplate code for the cron + command + mail report paradigma?
I also like the various backupninja helper scripts.
I was thinking about Bacula, but I'm not sure if it has block-level
deduplication, from the docks it seems it has file-level dedup. It seems
like a bit of an overkill for my case.


On Wed, Oct 21, 2015 at 5:07 PM, Sarunas Burdulis <
sarunas@math.dartmouth.edu> wrote:

> On 10/21/2015 10:47 AM, Ondřej Grover wrote:
> > I could try to request a RAM upgrade (not super easy in uni environment,
> > but doable), but it goes against my understanding of what a storage
> > server is. In my book, a storage server needs fast IO capabilities
> > (storage drives and NIC) and possibly CPU for compression. Deduplication
> > is the only reason I could think of higher RAM requirements on a storage
> > server.
>
> This morning I added a 'memory' plugin to the Munin node on the server I
> described. Here is a snaphot so far, for what it's worth. The bump at
> 9:50 is an rsync from one of the biggest filesystems backed-up (~600GiB,
> about a million files to sync).
>
> https://math.dartmouth.edu/owncloud/index.php/s/E3VYxhWlCqUusn4
>
> > Sarunas, did you have good experiences with actually using the backups
> > for restoring?
>
> Whenever I needed them, backup copies were available. This is more of an
> archival storage. The main point for using btrfs was instantaneous
> snapshot creation (and removal of older ones, something that takes
> 'forever' in rsync --link-dest scenario, using rsnapshot, for example).
>
> Sarunas
>
> > On Wed, Oct 21, 2015 at 4:28 PM, Sarunas Burdulis
> > <sarunas@math.dartmouth.edu <mailto:sarunas@math.dartmouth.edu>> wrote:
> >
> >     On 10/21/2015 05:32 AM, Ondřej Grover wrote:
> >     > [...]
> >     > Your btrfs setup sounds interesting, Sarunas. Are you running those
> >     > btrfs volumes on Jessie? Any stability issues? Do you think it
> could
> >     > work with just 1GB RAM + 1 GB swap? I was thinking about doing
> >     > out-of-band deduplication to conserve RAM. Do you use both
> compression
> >     > and deduplication (file or block level)? I heard they don't play
> nicely
> >     > together. Could you share your mount options?
> >
> >     Our backup server is actually running Linux 3.13.0-66-generic, Btrfs
> >     v3.12 — Ubuntu 14.04.3 LTS, which is behind Jessie in software
> versions.
> >     It has been very stable — for the past too years it's running as a
> >     headless box in an off-site location.
> >
> >     We don't use compression or deduplication. In-band dedup is not in
> btrfs
> >     3.*, and I haven't tried any out-of-band tools. Nothing particular in
> >     Btrfs mount, the only non-default options in 'noatime':
> >
> >     LABEL=BACKUPS /backups btrfs noatime 0 0
> >
> >     1GB RAM sounds low by today's habits. That doesn't mean it won't
> work,
> >     but just begs for RAM upgrade.
> >
> >     Sarunas
> >
> >     > On Tue, Oct 20, 2015 at 8:35 PM, Sarunas Burdulis
> >     > <sarunas@math.dartmouth.edu <mailto:sarunas@math.dartmouth.edu>
> >     <mailto:sarunas@math.dartmouth.edu
> >     <mailto:sarunas@math.dartmouth.edu>>> wrote:
> >     >
> >     > On 10/20/2015 12:57 PM, Ondřej Grover wrote:
> >     >> Hello,
> >     >
> >     >> I'm looking for recommendations for backup solutions that don't
> >     reinvent
> >     >> the wheel and are reliable and used. I want to backup two servers
> >     to a
> >     >> backup server. The main data content is several hundred GB in
> >     many very
> >     >> small files.
> >     >
> >     >> [...]
> >     >
> >     >> I've also looked at the new kids on the block like obnam, attick
> and
> >     >> borgbackup. They look interesting, but I prefer time-tested SW
> >     for backups.
> >     >> After realizing that these new backup programs pretty much try to
> >     >> replicate features of btrfs or ZFS (incremental snapshots,
> >     block-level
> >     >> compression and deduplication) I started thinking that I could
> >     perhaps
> >     >> just send the data to the backup server via rsync and save them
> to a
> >     >> btrfs or ZFS (but the backup server may not have enough RAM for
> >     ZFS) and
> >     >> create daily snaphosts on the server. If memory will permit (if I
> >     >> optimize it), I'd go with ZFS as it should be more reliable. Does
> >     >> anybody use such a solution?
> >     >
> >     > Yes, we do use pretty much the setup you describe, i.e. rsync push
> to
> >     > backup server, btrfs volume with daily snapshot subvolumes on the
> >     > server. This particular setup has been in use for more than two
> >     years by
> >     > now, and is working well with 8-10 servers (totals for all
> backed-up
> >     > filesystems is about 1.2TiB/2.5M files of varying size; transferred
> >     > daily delta is of course just a fraction of this). The backup
> >     server is
> >     > a rather modest self-built Mini-ITX Core i5 with 8GiB RAM and 6TiB
> >     > hardware (3ware) RAID-10 storage volume.
> >     >
> >     > Using btrfs volume in btrfs-raid configuration (raid-1 for data and
> >     > metadata) would provide protection against data bit-rot. We use
> >     several
> >     > such volumes on some of our severs, though not in combination with
> >     btrfs
> >     > snapshot subvolumes.
> >     >
> >     > --
> >     > Sarunas Burdulis
> >     > http://math.dartmouth.edu/~sarunas
> >
> >
>
>
> --
> Sarunas Burdulis
> Systems Administrator
> Department of Mathematics, Dartmouth College
> http://math.dartmouth.edu/~sarunas
>
>

[toc] | [prev] | [next] | [standalone]

#159638

On 10/22/2015 02:47 AM, Ondřej Grover wrote:
> Thank you Sarunas for the graphs, it is most illuminating. That's pretty
> much how I imagine a storage server should behave.
> Atomic snapshots are very appealing indeed. In my case cp -lr or
> --link-dest solutions would take a long time as the FS hierarchy is both
> very broad and deep.

Ondřej, you're are welcome. “For the sake of science” I restarted the
backups' server yesterday with a kernel boot parameter mem=1G. All the
nightly rsync jobs and btrfs snapshot manipulations went as usual, and
the server is running fine:

https://math.dartmouth.edu/owncloud/index.php/s/fYO81CfmuV5yrpZ

So 1G RAM might still do just fine for such basic tasks.

-- 
Sarunas Burdulis
Systems Administrator
Department of Mathematics, Dartmouth College
http://math.dartmouth.edu/~sarunas

[toc] | [prev] | [next] | [standalone]

#159645

On Oct 20, 2015, at 10:57 AM, Ondřej Grover <ondrej.grover@gmail.com> wrote:

> I'm looking for recommendations for backup solutions that don't reinvent the wheel and are reliable and used.

Have you considered Amanda? It's been around for quite a long time, it's worked very reliably for me for the past 15 or so years, it'll do an entire net, it's free, and apt-get will install it for you. It uses tar or dump (your choice) to write the backup, so if you lose the program, you can recover with standard *nix utilities.

I don't know what you use for backup media. I use tape, and that's what Amanda was originally designed for, but I hear it does disks now as well.

> I want to backup two servers to a backup server. The main data content is several hundred GB in many very small files.

Amanda doesn't require another box; one of the two can do it -- lots of free CPU cycles if you're using tape. But it does require a big hunk of disk space for a cache. 

-- 
Glenn English

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web