Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #261395 > unrolled thread

I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsync

Back to article view | Back to linux.debian.user

Contents

  I uninstalled OpenMediaVault (because totally overkill for me) and  replaced it with borgbackup and rsync Jason <jason@7748229.xyz> - 2023-09-01 06:50 +0200
    Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsync mlnl <mlnl@mailbox.org> - 2023-09-01 08:20 +0200
    Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsync jeremy ardley <jeremy.ardley@gmail.com> - 2023-09-01 08:40 +0200
    Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsync Manphiz <manphiz@gmail.com> - 2023-09-01 12:40 +0200
    Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq John Hasler <john@sugarbit.com> - 2023-09-01 14:30 +0200
      Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq Default User <hunguponcontent@gmail.com> - 2023-09-01 21:30 +0200
        Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq Michel Verdier <mv524@free.fr> - 2023-09-01 22:50 +0200
          Re: I uninstalled OpenMediaVault (because totally overkill for me)          and replaced it with borgbackup and rsyncq Linux-Fan <Ma_Sys.ma@web.de> - 2023-09-01 23:30 +0200
            Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq Michel Verdier <mv524@free.fr> - 2023-09-02 00:10 +0200
              Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq Michael Kjörling <2695bd53d63c@ewoof.net> - 2023-09-02 11:20 +0200
                Backup systems (was: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq) Stefan Monnier <monnier@iro.umontreal.ca> - 2023-09-02 19:30 +0200
                  Re: Backup systems Michel Verdier <mv524@free.fr> - 2023-09-02 21:20 +0200
                    Re: Backup systems David Christensen <dpchrist@holgerdanske.com> - 2023-09-02 23:50 +0200
                      Re: Backup systems Michel Verdier <mv524@free.fr> - 2023-09-03 00:30 +0200
                        Re: Backup systems David Christensen <dpchrist@holgerdanske.com> - 2023-09-03 02:00 +0200
                          Re: Backup systems Michel Verdier <mv524@free.fr> - 2023-09-03 11:00 +0200
                      Re: Backup systems Michael Kjörling <2695bd53d63c@ewoof.net> - 2023-09-03 12:10 +0200
                        Re: Backup systems David Christensen <dpchrist@holgerdanske.com> - 2023-09-03 23:30 +0200
                          Re: Backup systems Michael Kjörling <2695bd53d63c@ewoof.net> - 2023-09-04 10:00 +0200
                            Re: Backup systems David Christensen <dpchrist@holgerdanske.com> - 2023-09-04 23:00 +0200
                              Re: Backup systems Michael Kjörling <2695bd53d63c@ewoof.net> - 2023-09-05 16:40 +0200
                                Re: Backup systems David Christensen <dpchrist@holgerdanske.com> - 2023-09-06 02:20 +0200
                Re: I uninstalled OpenMediaVault (because totally overkill for me)          and replaced it with borgbackup and rsyncq Linux-Fan <Ma_Sys.ma@web.de> - 2023-09-03 00:10 +0200
                  Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq David <david.g_jones@ntlworld.com> - 2023-09-03 20:20 +0200
                    Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq Gareth Evans <donotspam@fastmail.fm> - 2023-09-03 21:40 +0200
          Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq Default User <hunguponcontent@gmail.com> - 2023-09-06 02:10 +0200
            Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq Default User <hunguponcontent@gmail.com> - 2023-09-06 02:40 +0200
              Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq David Christensen <dpchrist@holgerdanske.com> - 2023-09-06 07:40 +0200
        Re: I uninstalled OpenMediaVault (because totally overkill for me)          and replaced it with borgbackup and rsyncq Linux-Fan <Ma_Sys.ma@web.de> - 2023-09-01 23:20 +0200
          Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq Default User <hunguponcontent@gmail.com> - 2023-09-06 03:40 +0200
            Re: I uninstalled OpenMediaVault (because totally overkill for me)          and replaced it with borgbackup and rsyncq Linux-Fan <Ma_Sys.ma@web.de> - 2023-09-09 19:00 +0200

Page 1 of 2  [1] 2  Next page →

#261395 — I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsync

[Multipart message — attachments visible in raw view] — view raw

Hi

I was a user of OpenMediaVault for several years. I even donated money 
to the developer.

But very provocatively OpenMediaVault is bloatware, way too big. The 
only thing I need is a reliable backup.

I had pure Debian (minimal installation, very few packages) installed 
with borgbackup and rsync.

I am now very satisfied, much more streamlined than OpenMediaVault.

Am I wrong, are my statements completely off?

Or how does your backup look like?

cheers
Jason

[toc] | [next] | [standalone]

#261396 — Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsync

Hi Jason,

Jason <jason@7748229.xyz> wrote:

>I had pure Debian (minimal installation, very few packages) installed 
>with borgbackup and rsync.
>
>I am now very satisfied, much more streamlined than OpenMediaVault.
>
>Am I wrong, are my statements completely off?
>
>Or how does your backup look like?

I have three SSDs with two for storage space, one for the system and one
USB drive for backup. All drives are lvm/luks encrypted. Many
backup solutions exist and i don't know OpenMediaVault, but since few
years i'm using borgbackup and rsync & i'm happy with it.

-- 
mlnl

GPG:1FC05426F87FA623

[toc] | [prev] | [next] | [standalone]

#261397 — Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsync

On 1/9/23 12:44, Jason wrote:
>
> Or how does your backup look like?


I had a QNAP NAS but it became so incredibly slow I replaced it with 
Debian using Samba and SSH.

The backups are managed by the clients, but periodically I save part of 
the NAS to Amazon S3.

I also have a remote Nextcloud server which my clients use to share a 
small working set of documents and files. It's on Akami (nee Linode) and 
is very fast in my region and pretty cheap. I used to do this on AWS but 
Linode is cheaper and simpler for my applications

I'm also in the process of converting some of my storage from Dropbox to 
Nextcloud

[toc] | [prev] | [next] | [standalone]

#261400 — Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsync

Jason <jason@7748229.xyz> writes:

> Hi
>
> I was a user of OpenMediaVault for several years. I even donated money to the
> developer.
>
> But very provocatively OpenMediaVault is bloatware, way too big. The only thing
> I need is a reliable backup.
>
> I had pure Debian (minimal installation, very few packages) installed with
> borgbackup and rsync.
>
> I am now very satisfied, much more streamlined than OpenMediaVault.
>
> Am I wrong, are my statements completely off?
>
> Or how does your backup look like?
>
> cheers
> Jason
>
>

Another OMV user for several years and have been happy about it.  It's a
NAS system based on Debian (latest version based on Buster), on top of
which it provides its NAS management layer and you can control what to
install or not.  There is no ads, no telemetry, and it manages my ZFS
raidz1 cluster without any issue for several years, and the system
itself takes less than 5GB space, so I'm not sure why you'd call it
bloatware.

-- 
Manphiz

[toc] | [prev] | [next] | [standalone]

#261405 — Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq

Jason writes:
> Or how does your backup look like?

Just rsync.
-- 
John Hasler 
john@sugarbit.com
Elmwood, WI USA

[toc] | [prev] | [next] | [standalone]

#261410 — Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq

On Fri, 2023-09-01 at 07:25 -0500, John Hasler wrote:
> Jason writes:
> > Or how does your backup look like?
> 
> Just rsync.


Sorry, I just couldn't resist chiming in here.

I have never used OpenMediaVault.  

I HAVE used a number of other backup methodologies, including
Borgbackup, for which I had high hopes, but was highly disappointed.  

In the end, I currently have settled upon using rsnapshot to back up my
single-machine, single-user setup to external external usb hard drive
A, which is then copied to external usb hard drive B, using rsync.  If
you can do rsync, you can do rsnapshot.  

It's easy, especially when it comes to restoring, verifying, and
impromptu access to data, to use random stuff, or even to just "check
on" your data occasionally, to reassure yourself that it is still
there. 

Yes, it does require considerable space (no data de-duplication), and
the rsync of the backup drives does take considerable time.  But to me,
it is worth it, to avoid the methodological equivalent of "vendor lock-
in". 

INB4:  No, I don't do online backups. If people or organizations with
nose problems want my data they are going to have to make at least a
little effort to get it. And yes, I do know the 1-2-3 backup
philosophy, which does seem like a good idea for many (most?) users. 

Also, I really like Clonezilla for occasional brute-force, scorched-
earth backups, such as when preparing for a complete reinstall or
release upgrade, or when switching to a new computer. 

(Note: everything above applies to backing up my data: (/home/[user]
only). For backing up the actual system: (/ except for /home/[user]), I
use (and like) Timeshift. It has saved my donkey more than once!)

Well, that's what works for me.  Feel free to disregard this. It may
not work for you.  And, "if it breaks, you get to keep both pieces"!

[toc] | [prev] | [next] | [standalone]

#261413 — Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq

On 2023-09-01, Default User wrote:

> Yes, it does require considerable space (no data de-duplication), and
> the rsync of the backup drives does take considerable time.  But to me,
> it is worth it, to avoid the methodological equivalent of "vendor lock-
> in". 

You must have a bad configuration : rsnaphot de-duplicate using hard
links so you never have duplicated files. Keeping 52 weekly and 7 daily
and 24 hourly I need only 130% of original space. And it takes minimal
time as it transfers only changes and can use ssh compression.

[toc] | [prev] | [next] | [standalone]

#261420 — Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq

[Multipart message — attachments visible in raw view] — view raw

Michel Verdier writes:

> On 2023-09-01, Default User wrote:
>
> > Yes, it does require considerable space (no data de-duplication), and
> > the rsync of the backup drives does take considerable time.  But to me,
> > it is worth it, to avoid the methodological equivalent of "vendor lock-
> > in".
>
> You must have a bad configuration : rsnaphot de-duplicate using hard
> links so you never have duplicated files. Keeping 52 weekly and 7 daily
> and 24 hourly I need only 130% of original space. And it takes minimal
> time as it transfers only changes and can use ssh compression.

It highly depends on the type of data that is being backed up.

For my regular user files, I think a file-based deduplication works OK. But  
for my VM images, hardlinks would only save space for those VMs which did  
not run between the current and the preceding backup.

Btw.: I am personally not using any hard-link based approach, mostly due to  
the missing encryption and integrity protection of data and metadata.

HTH
Linux-Fan

öö

[toc] | [prev] | [next] | [standalone]

#261424 — Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq

On 2023-09-01, Linux-Fan wrote:

> For my regular user files, I think a file-based deduplication works OK. But
> for my VM images, hardlinks would only save space for those VMs which did not
> run between the current and the preceding backup.

rsnapshot de-duplicate files not blocks. If you backup files from your VM
you can de-duplicate. If you want to backup whole VM images of course you
need a tool working on physical blocks.

> Btw.: I am personally not using any hard-link based approach, mostly due to
> the missing encryption and integrity protection of data and metadata.

rsnapshot use hard links on the backup filesystem. I encrypt this
filesystem. Integrity is the same for all backup systems. Doing 2 backups
on remote places cover this.

[toc] | [prev] | [next] | [standalone]

#261435 — Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq

On 2 Sep 2023 00:04 +0200, from mv524@free.fr (Michel Verdier):
> rsnapshot use hard links on the backup filesystem.

More accurately, rsnapshot (which is basically a frontend to rsync)
tells rsync to do that; IIRC by passing --link-dest pointing at the
previous backup target directory.

And this is not an argument against rsnapshot/rsync; I use the
combination myself, plus a home-grown script to prune old backups
based on the amount of free space remaining on the backup disks rather
than a fixed backup count.

The one big downside of rsnapshot + rsync at least for me is that it
has no real concept of whether a backup run completed or was aborted;
the latter, for example, due to a system shutdown or out-of-memory
condition while it's running. That really shouldn't happen often or
even at all, but I've had it happen a few times over many years, and
it's a bit of a pain when it does happen because you pretty much have
to go in and delete the most recent backup and then renumber all the
previous ones to get back into a sane state on the backup target. Yes,
that can be added with another piece of a wrapper script, and I have
on occasion contemplated doing just that; but it happens sufficiently
rarely, and is noisy enough when it does happen, that it isn't really
worth the effort in my particular situation.

The biggest issue for me is ensuring that I am not dependent on
_anything_ on the backed-up system itself to start restoring that
system from a backup. In other words, enabling bare-metal restoration.
I figure that I can always download a Debian live ISO, put that on a
USB stick, set up an environment to access the (encrypted) backup
drive, set up partitions on new disks, and start copying; if I were
using backup software that uses some kind of custom format, that would
include keeping a copy of an installation package of that and whatever
else it needs for installing and running within a particular
distribution version, and making sure to specifically test that,
ideally without Internet access, so that I can get to the point of
starting to copy things back. (I figure that the boot loader is the
easy part to all this.)

-- 
Michael Kjörling                     🔗 https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

[toc] | [prev] | [next] | [standalone]

#261448 — Backup systems (was: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq)

> More accurately, rsnapshot (which is basically a frontend to rsync)
> tells rsync to do that; IIRC by passing --link-dest pointing at the
> previous backup target directory.

I've used a similar (tho hand-cooked) script running `rsync`.
I switched to Bup a few years ago and saw a significant reduction in the
size of my backups that is partly due to the deduplication *between*
machines (I backup several Debian machines to the same backup
repository) as well as because the deduplication occurs even when I move
files around (most obvious when I move directories filled with large
files like videos or music).

Another advantage I've notice with Bup is that my backups are done much
more quickly (because more of the work is done locally on the client
machines which have SSDs, there's a lot less network communication, and
much less work is done on the backup server which is a low-power machine
with spinning rust).

The great advantage of `rsync/rsnapshot` is that your backup is really
"live": you can go `grep` and `find` through it with ease.  With Bup you
can theoretically do that as well (via `bup fuse`) but the performance
is significantly lower :-(


        Stefan

[toc] | [prev] | [next] | [standalone]

#261451 — Re: Backup systems

On 2023-09-02, Stefan Monnier wrote:

> I switched to Bup a few years ago and saw a significant reduction in the
> size of my backups that is partly due to the deduplication *between*
> machines (I backup several Debian machines to the same backup
> repository) as well as because the deduplication occurs even when I move
> files around (most obvious when I move directories filled with large
> files like videos or music).

I setup deduplication between hosts with rsnapshot as you do. But it was
a small gain in my case as the larger part was users data, logs and the
like. So always different between hosts. I gain only on system
files. Mainly /etc as I don't backup binaries and libs.
I almost never move large directories. But if needed it's easy to move it
also in rsnapshot directories.

[toc] | [prev] | [next] | [standalone]

#261455 — Re: Backup systems

On 9/2/23 12:15, Michel Verdier wrote:
> On 2023-09-02, Stefan Monnier wrote:
> 
>> I switched to Bup a few years ago and saw a significant reduction in the
>> size of my backups that is partly due to the deduplication *between*
>> machines (I backup several Debian machines to the same backup
>> repository) as well as because the deduplication occurs even when I move
>> files around (most obvious when I move directories filled with large
>> files like videos or music).
> 
> I setup deduplication between hosts with rsnapshot as you do. But it was
> a small gain in my case as the larger part was users data, logs and the
> like. So always different between hosts. I gain only on system
> files. Mainly /etc as I don't backup binaries and libs.
> I almost never move large directories. But if needed it's easy to move it
> also in rsnapshot directories.


I have a SOHO LAN:

* My primary workstation is Debian Xfce on a 60GB  2.5" SATA SSD with 1G 
boot, 1G swap, and 12G root partitions.  It has one user (myself) with 
minimal home data (e-mail and CVS working directories).  I backup boot 
and root.

* I keep the vast majority of my data on a FreeBSD server with Samba and 
the CVS repository (via SSH) on a ZFS stripe of two mirrors containing 
two 3TB 3.5" SATA HDD's each (e.g. 6TB RAID10).  I backup the Samba data.

* I run rsync(1) and homebrew shell/ Perl scripts on the server to 
backup the various LAN sources to backup destination file system tree on 
the server.  I have enabled ZFS compression on the pool and enabled 
deduplication on the backup tree.


I ran some statistics for the daily driver backups in March.  The 
results were 4.9 GB backup size, 258 backups, 1.2 TB apparent total 
backup storage, and 29.0 GB actual total backup storage.  So, a savings 
of about 42:1:

https://www.mail-archive.com/debian-user@lists.debian.org/msg789807.html


Today, I collected some statistics for the backups of my data on the 
file server:

2023-09-02 14:10:30 toor@f3 ~
# du -hsx /jail/samba/var/local/samba/dpchrist
693G	/jail/samba/var/local/samba/dpchrist

2023-09-02 14:11:09 toor@f3 ~
# ls /jail/samba/var/local/samba/dpchrist/.zfs/snapshot | wc -l
       98

2023-09-02 14:13:50 toor@f3 ~
# du -hs /jail/samba/var/local/samba/dpchrist/.zfs/snapshot
  67T	/jail/samba/var/local/samba/dpchrist/.zfs/snapshot

2023-09-02 14:19:24 toor@f3 ~
# zfs get 
compression,compressratio,dedup,used,usedbydataset,usedbysnapshots 
p3/ds2/samba/dpchrist | sort
NAME                   PROPERTY         VALUE          SOURCE
p3/ds2/samba/dpchrist  compression      lz4            inherited from p3
p3/ds2/samba/dpchrist  compressratio    1.02x          -
p3/ds2/samba/dpchrist  dedup            off            default
p3/ds2/samba/dpchrist  used             777G           -
p3/ds2/samba/dpchrist  usedbydataset    693G           -
p3/ds2/samba/dpchrist  usedbysnapshots  84.2G          -


So, 693 GB backup size, 98 backups, 67 TB apparent total backup storage, 
and 777 GB actual total backup storage.  So, a savings of about 88:1.


What statistics are other readers seeing for similar use-cases and their 
backup solutions?


David

[toc] | [prev] | [next] | [standalone]

#261457 — Re: Backup systems

On 2023-09-02, David Christensen wrote:

> What statistics are other readers seeing for similar use-cases and their
> backup solutions?

I have 83 backups resulting to 130% of data. So a ratio of 63:1. But
because of performance limitation I don't use compression on backup
server. And also rsync deduplication is less consuming than zfs.

[toc] | [prev] | [next] | [standalone]

#261458 — Re: Backup systems

On 9/2/23 15:26, Michel Verdier wrote:
> On 2023-09-02, David Christensen wrote:
> 
>> What statistics are other readers seeing for similar use-cases and
>> their backup solutions?
> 
> I have 83 backups resulting to 130% of data. So a ratio of 63:1.


Nice.


> But because of performance limitation I don't use compression on
> backup server.


What partitioning scheme, volume manager, file system, etc., do you use 
on your backup server?


What is the performance limitation?


If you wanted compression on the backup server, how would you implement it?


> And also rsync deduplication is less consuming than zfs.


Please define your system and metrics.


David

[toc] | [prev] | [next] | [standalone]

#261462 — Re: Backup systems

On 2023-09-02, David Christensen wrote:

> > I have 83 backups resulting to 130% of data. So a ratio of 63:1.
> 
> Nice.

It's just related to the size of periodically modified data, no glory
here. Each usage must be analyzed.

> What partitioning scheme, volume manager, file system, etc., do you use on
> your backup server?

xfs on logical raid10, no VM

> What is the performance limitation?

> If you wanted compression on the backup server, how would you implement it?

I have both small CPU and RAM on the backup server. It degrades
severely if I setup some sort of compression. I choose to crypt
partitions instead. And get larger disks.

>> And also rsync deduplication is less consuming than zfs.
>
> Please define your system and metrics.

I must admit I only did a bench a long time ago on another system. I
don't remember metrics.

[toc] | [prev] | [next] | [standalone]

#261463 — Re: Backup systems

On 2 Sep 2023 14:49 -0700, from dpchrist@holgerdanske.com (David Christensen):
> So, 693 GB backup size, 98 backups, 67 TB apparent total backup storage, and
> 777 GB actual total backup storage.  So, a savings of about 88:1.
> 
> What statistics are other readers seeing for similar use-cases and their
> backup solutions?

8.07 TiB physically stored on one backup drive holding 174 backups;
11.4 TiB total logical (excluding effects of compression) data on the
source; 7.83 TiB hot current logical data on the source (excluding
things like ZFS snapshots and compression).

Which by your way of calculating seems to work out to an about 246:1
savings compared to simply keeping every single copy in full and
uncompressed. (Which would require almost 2 PB of storage.) But this
figure is a bit exaggerated since there are parts of the backups that
I prune after a while while keeping the rest of that backup, so let's
be very generous and call it maybe a 100:1 savings in practice.

Which is still pretty good for something that only does raw copying
with whole-file deduplication.

I have a wide mix of file sizes and content types; everything from
tiny Maildir message files through photos in the tens of megabytes
range to VM disk image files in the tens of gigabytes range, ranging
from highly compressible to essentially incompressible, and ranging
from files that practically never change after I initially store them
to ones that change all the time.

-- 
Michael Kjörling                     🔗 https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

[toc] | [prev] | [next] | [standalone]

#261472 — Re: Backup systems

On 9/3/23 03:02, Michael Kjörling wrote:
> 8.07 TiB physically stored on one backup drive holding 174 backups;
> 11.4 TiB total logical (excluding effects of compression) data on the
> source; 7.83 TiB hot current logical data on the source (excluding
> things like ZFS snapshots and compression).
> 
> Which by your way of calculating seems to work out to an about 246:1
> savings compared to simply keeping every single copy in full and
> uncompressed.


Without seeing a console session, I am unsure what you mean by 
"physically stored", "total logical (excluding effects of compression) 
data", and "hot current logical data ... (excluding things like ZFS 
snapshots and compression)".


What partitioning scheme, volume manager, file system, compression, 
etc., do you use on your backup server?


I had thought you were using rsnapsnot/ rsync --link-dest, but you also 
mention ZFS snapshots.  Please clarify.


David

[toc] | [prev] | [next] | [standalone]

#261477 — Re: Backup systems

On 3 Sep 2023 14:20 -0700, from dpchrist@holgerdanske.com (David Christensen):
>> 8.07 TiB physically stored on one backup drive holding 174 backups;
>> 11.4 TiB total logical (excluding effects of compression) data on the
>> source; 7.83 TiB hot current logical data on the source (excluding
>> things like ZFS snapshots and compression).
> 
> Without seeing a console session, I am unsure what you mean by "physically
> stored", "total logical (excluding effects of compression) data", and "hot
> current logical data ... (excluding things like ZFS snapshots and
> compression)".

"Physically stored" is how much data, after compression and including
file system metadata, is actually written to disk and necessary for
all data to be accessible; it's the relevant metric for whether I need
to add disk space.

"Logical" is the sum of all apparent file sizes as visible to userland
utilities e.g. through stat(2).

Something like `dd if=/dev/zero of=$(mktemp) bs=1M count=1M` would
result in a large logical size but, because of compression, a very
small amount of physically stored data.

"Hot" is perhaps better referred to as the "current" data set; since
snapshots (and earlier backups) can include data which has since been
deleted, and is thus no longer current but still exists on disk.


> What partitioning scheme, volume manager, file system, compression, etc., do
> you use on your backup server?

ZFS within LUKS containers. If I recall correctly, the backup pool is
set to use zstd compression.


> I had thought you were using rsnapsnot/ rsync --link-dest, but you also
> mention ZFS snapshots.  Please clarify.

Mostly ZFS with a rotating snapshot schedule on the source (the root
file system is ext4); copied using rsync --link-dest (through
rsnapshot) to a ZFS file system which doesn't use snapshots on the
backup target. Most of the ZFS file systems are set up to use
compression; there are a few where I know _a priori_ that the data is
in effect completely incompressible so there's no point in using CPU
to even try to compress that data, so those have compression turned
off.

(In ZFS, creating a file system is barely any more involved than
creating a directory, and all file systems come out of the same "pool"
which is a collection of >=1 storage devices set up with some
particular method of redundancy, possibly none. In more traditional
*nix parlace, a *nix file system is conceptually closer to a ZFS
pool.)

Hopefully this is more clear.

-- 
Michael Kjörling                     🔗 https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

[toc] | [prev] | [next] | [standalone]

#261484 — Re: Backup systems

On 9/4/23 00:53, Michael Kjörling wrote:
> On 3 Sep 2023 14:20 -0700, from dpchrist@holgerdanske.com (David Christensen):
>> Without seeing a console session, I am unsure what you mean by "physically
>> stored", "total logical (excluding effects of compression) data", and "hot
>> current logical data ... (excluding things like ZFS snapshots and
>> compression)".
> 
> "Physically stored" is how much data, after compression and including
> file system metadata, is actually written to disk and necessary for
> all data to be accessible; it's the relevant metric for whether I need
> to add disk space.
> 
> "Logical" is the sum of all apparent file sizes as visible to userland
> utilities e.g. through stat(2).
> 
> Something like `dd if=/dev/zero of=$(mktemp) bs=1M count=1M` would
> result in a large logical size but, because of compression, a very
> small amount of physically stored data.
> 
> "Hot" is perhaps better referred to as the "current" data set; since
> snapshots (and earlier backups) can include data which has since been
> deleted, and is thus no longer current but still exists on disk.
> 
> 
>> What partitioning scheme, volume manager, file system, compression, etc., do
>> you use on your backup server?
> 
> ZFS within LUKS containers. If I recall correctly, the backup pool is
> set to use zstd compression.
> 
> 
>> I had thought you were using rsnapsnot/ rsync --link-dest, but you also
>> mention ZFS snapshots.  Please clarify.
> 
> Mostly ZFS with a rotating snapshot schedule on the source (the root
> file system is ext4); copied using rsync --link-dest (through
> rsnapshot) to a ZFS file system which doesn't use snapshots on the
> backup target. Most of the ZFS file systems are set up to use
> compression; there are a few where I know _a priori_ that the data is
> in effect completely incompressible so there's no point in using CPU
> to even try to compress that data, so those have compression turned
> off.
> 
> (In ZFS, creating a file system is barely any more involved than
> creating a directory, and all file systems come out of the same "pool"
> which is a collection of >=1 storage devices set up with some
> particular method of redundancy, possibly none. In more traditional
> *nix parlace, a *nix file system is conceptually closer to a ZFS
> pool.)
> 
> Hopefully this is more clear.


So for backup storage:

* We are both using ZFS with default compression.

* You are using 'rsync --link-dest' (via rsnapshot(1)) for deduplication 
and I am using ZFS for deduplication.

Related:

* I am using zfs-auto-snapsnot(8) for snapsnots.  Are you using 
rsnapsnot(1) for snapshots?


Here are the current backups for my current daily driver:

2023-09-04 13:26:15 toor@f3 ~
# zfs get -o property,value 
compression,compressratio,dedup,logicalreferenced,logicalused,refcompressratio,referenced,used,usedbydataset,usedbysnapshots 
p3/backup/taz.tracy.holgerdanske.com
PROPERTY           VALUE
compression        lz4
compressratio      2.14x
dedup              verify
logicalreferenced  6.59G
logicalused        48.7G
refcompressratio   1.83x
referenced         3.89G
used               23.4G
usedbydataset      3.89G
usedbysnapshots    19.5G

2023-09-04 13:26:36 toor@f3 ~
# ls -1 /var/local/backup/taz.tracy.holgerdanske.com/.zfs/snapshot | wc -l
      186

2023-09-04 13:27:15 toor@f3 ~
# du -hs /var/local/backup/taz.tracy.holgerdanske.com/ 
/var/local/backup/taz.tracy.holgerdanske.com/.zfs
3.9G	/var/local/backup/taz.tracy.holgerdanske.com/
722G	/var/local/backup/taz.tracy.holgerdanske.com/.zfs

2023-09-04 13:28:02 toor@f3 ~
# crontab -l
  9 3 * * * /usr/local/sbin/zfs-auto-snapshot -k d 40
21 3 1 * * /usr/local/sbin/zfs-auto-snapshot -k m 99
27 3 1 1 * /usr/local/sbin/zfs-auto-snapshot -k y 99


Observations:

* du(1) of the backup file system matches ZFS properties 'referenced' 
and 'usedbydataset'.

* I am unable to correlate du(1) of the snapshots to any ZFS properties 
-- du(1) reports much more storage than ZFS 'usedbysnapshots', even when 
scaled by 'compressratio'.


David

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.user

csiph-web