Re: Backup solutions without reinventig the wheel these days

From	Ondřej Grover <ondrej.grover@gmail.com>
Newsgroups	linux.debian.user
Subject	Re: Backup solutions without reinventig the wheel these days
Date	2015-10-22 08:50 +0200
Message-ID	<qmhRf-2Ly-5@gated-at.bofh.it> (permalink)
References	(1 earlier) <qlKsi-4o0-11@gated-at.bofh.it> <qlY2d-7mq-11@gated-at.bofh.it> <qm2yT-5Ck-25@gated-at.bofh.it> <qm2Sd-606-19@gated-at.bofh.it> <qm3bA-6Fw-27@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

[Multipart message — attachments visible in raw view] - view raw

Thank you Sarunas for the graphs, it is most illuminating. That's pretty
much how I imagine a storage server should behave.
Atomic snapshots are very appealing indeed. In my case cp -lr or
--link-dest solutions would take a long time as the FS hierarchy is both
very broad and deep.

So far it seems I will go with backupninja with simple rsync transport and
a few custom shell scripts to create and delete btrfs snapshots on the
remote server. This means I will need root access on the backup server, but
I can restrict that with authorized_keys or sudo.

Is there some other framework similar to backupninja that provides the
common boilerplate code for the cron + command + mail report paradigma?
I also like the various backupninja helper scripts.
I was thinking about Bacula, but I'm not sure if it has block-level
deduplication, from the docks it seems it has file-level dedup. It seems
like a bit of an overkill for my case.


On Wed, Oct 21, 2015 at 5:07 PM, Sarunas Burdulis <
sarunas@math.dartmouth.edu> wrote:

> On 10/21/2015 10:47 AM, Ondřej Grover wrote:
> > I could try to request a RAM upgrade (not super easy in uni environment,
> > but doable), but it goes against my understanding of what a storage
> > server is. In my book, a storage server needs fast IO capabilities
> > (storage drives and NIC) and possibly CPU for compression. Deduplication
> > is the only reason I could think of higher RAM requirements on a storage
> > server.
>
> This morning I added a 'memory' plugin to the Munin node on the server I
> described. Here is a snaphot so far, for what it's worth. The bump at
> 9:50 is an rsync from one of the biggest filesystems backed-up (~600GiB,
> about a million files to sync).
>
> https://math.dartmouth.edu/owncloud/index.php/s/E3VYxhWlCqUusn4
>
> > Sarunas, did you have good experiences with actually using the backups
> > for restoring?
>
> Whenever I needed them, backup copies were available. This is more of an
> archival storage. The main point for using btrfs was instantaneous
> snapshot creation (and removal of older ones, something that takes
> 'forever' in rsync --link-dest scenario, using rsnapshot, for example).
>
> Sarunas
>
> > On Wed, Oct 21, 2015 at 4:28 PM, Sarunas Burdulis
> > <sarunas@math.dartmouth.edu <mailto:sarunas@math.dartmouth.edu>> wrote:
> >
> >     On 10/21/2015 05:32 AM, Ondřej Grover wrote:
> >     > [...]
> >     > Your btrfs setup sounds interesting, Sarunas. Are you running those
> >     > btrfs volumes on Jessie? Any stability issues? Do you think it
> could
> >     > work with just 1GB RAM + 1 GB swap? I was thinking about doing
> >     > out-of-band deduplication to conserve RAM. Do you use both
> compression
> >     > and deduplication (file or block level)? I heard they don't play
> nicely
> >     > together. Could you share your mount options?
> >
> >     Our backup server is actually running Linux 3.13.0-66-generic, Btrfs
> >     v3.12 — Ubuntu 14.04.3 LTS, which is behind Jessie in software
> versions.
> >     It has been very stable — for the past too years it's running as a
> >     headless box in an off-site location.
> >
> >     We don't use compression or deduplication. In-band dedup is not in
> btrfs
> >     3.*, and I haven't tried any out-of-band tools. Nothing particular in
> >     Btrfs mount, the only non-default options in 'noatime':
> >
> >     LABEL=BACKUPS /backups btrfs noatime 0 0
> >
> >     1GB RAM sounds low by today's habits. That doesn't mean it won't
> work,
> >     but just begs for RAM upgrade.
> >
> >     Sarunas
> >
> >     > On Tue, Oct 20, 2015 at 8:35 PM, Sarunas Burdulis
> >     > <sarunas@math.dartmouth.edu <mailto:sarunas@math.dartmouth.edu>
> >     <mailto:sarunas@math.dartmouth.edu
> >     <mailto:sarunas@math.dartmouth.edu>>> wrote:
> >     >
> >     > On 10/20/2015 12:57 PM, Ondřej Grover wrote:
> >     >> Hello,
> >     >
> >     >> I'm looking for recommendations for backup solutions that don't
> >     reinvent
> >     >> the wheel and are reliable and used. I want to backup two servers
> >     to a
> >     >> backup server. The main data content is several hundred GB in
> >     many very
> >     >> small files.
> >     >
> >     >> [...]
> >     >
> >     >> I've also looked at the new kids on the block like obnam, attick
> and
> >     >> borgbackup. They look interesting, but I prefer time-tested SW
> >     for backups.
> >     >> After realizing that these new backup programs pretty much try to
> >     >> replicate features of btrfs or ZFS (incremental snapshots,
> >     block-level
> >     >> compression and deduplication) I started thinking that I could
> >     perhaps
> >     >> just send the data to the backup server via rsync and save them
> to a
> >     >> btrfs or ZFS (but the backup server may not have enough RAM for
> >     ZFS) and
> >     >> create daily snaphosts on the server. If memory will permit (if I
> >     >> optimize it), I'd go with ZFS as it should be more reliable. Does
> >     >> anybody use such a solution?
> >     >
> >     > Yes, we do use pretty much the setup you describe, i.e. rsync push
> to
> >     > backup server, btrfs volume with daily snapshot subvolumes on the
> >     > server. This particular setup has been in use for more than two
> >     years by
> >     > now, and is working well with 8-10 servers (totals for all
> backed-up
> >     > filesystems is about 1.2TiB/2.5M files of varying size; transferred
> >     > daily delta is of course just a fraction of this). The backup
> >     server is
> >     > a rather modest self-built Mini-ITX Core i5 with 8GiB RAM and 6TiB
> >     > hardware (3ware) RAID-10 storage volume.
> >     >
> >     > Using btrfs volume in btrfs-raid configuration (raid-1 for data and
> >     > metadata) would provide protection against data bit-rot. We use
> >     several
> >     > such volumes on some of our severs, though not in combination with
> >     btrfs
> >     > snapshot subvolumes.
> >     >
> >     > --
> >     > Sarunas Burdulis
> >     > http://math.dartmouth.edu/~sarunas
> >
> >
>
>
> --
> Sarunas Burdulis
> Systems Administrator
> Department of Mathematics, Dartmouth College
> http://math.dartmouth.edu/~sarunas
>
>

Thread

Backup solutions without reinventig the wheel these days Ondřej Grover <ondrej.grover@gmail.com> - 2015-10-20 19:00 +0200
  Re: Backup solutions without reinventig the wheel these days Mario Castelán Castro  <marioxcc.MT@yandex.com> - 2015-10-20 19:50 +0200
    Re: Backup solutions without reinventig the wheel these days Teemu Likonen <tlikonen@iki.fi> - 2015-10-20 20:20 +0200
      Re: Backup solutions without reinventig the wheel these days Jonathan Dowland <jmtd@debian.org> - 2015-10-27 12:40 +0100
        Re: Backup solutions without reinventig the wheel these days Teemu Likonen <tlikonen@iki.fi> - 2015-10-27 16:40 +0100
  Re: Backup solutions without reinventig the wheel these days Sarunas Burdulis <sarunas@math.dartmouth.edu> - 2015-10-20 21:10 +0200
    Re: Backup solutions without reinventig the wheel these days Ondřej Grover <ondrej.grover@gmail.com> - 2015-10-21 11:40 +0200
      Re: Backup solutions without reinventig the wheel these days mourik jan heupink <heupink@merit.unu.edu> - 2015-10-21 13:30 +0200
        Re: Backup solutions without reinventig the wheel these days mourik jan heupink <heupink@merit.unu.edu> - 2015-10-21 13:40 +0200
      Re: Backup solutions without reinventig the wheel these days Eduardo M KALINOWSKI <eduardo@kalinowski.com.br> - 2015-10-21 13:30 +0200
      Re: Backup solutions without reinventig the wheel these days Dan Ritter <dsr@randomstring.org> - 2015-10-21 16:10 +0200
      Re: Backup solutions without reinventig the wheel these days Sarunas Burdulis <sarunas@math.dartmouth.edu> - 2015-10-21 16:30 +0200
        Re: Backup solutions without reinventig the wheel these days Ondřej Grover <ondrej.grover@gmail.com> - 2015-10-21 16:50 +0200
          Re: Backup solutions without reinventig the wheel these days Sarunas Burdulis <sarunas@math.dartmouth.edu> - 2015-10-21 17:10 +0200
            Re: Backup solutions without reinventig the wheel these days Ondřej Grover <ondrej.grover@gmail.com> - 2015-10-22 08:50 +0200
              Re: Backup solutions without reinventig the wheel these days Sarunas Burdulis <sarunas@math.dartmouth.edu> - 2015-10-22 16:10 +0200
  Re: Backup solutions without reinventig the wheel these days Glenn English <ghe@slsware.net> - 2015-10-22 18:30 +0200

csiph-web