Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #261477

Re: Backup systems

Show all headers | View raw

On 3 Sep 2023 14:20 -0700, from dpchrist@holgerdanske.com (David Christensen):
>> 8.07 TiB physically stored on one backup drive holding 174 backups;
>> 11.4 TiB total logical (excluding effects of compression) data on the
>> source; 7.83 TiB hot current logical data on the source (excluding
>> things like ZFS snapshots and compression).
> 
> Without seeing a console session, I am unsure what you mean by "physically
> stored", "total logical (excluding effects of compression) data", and "hot
> current logical data ... (excluding things like ZFS snapshots and
> compression)".

"Physically stored" is how much data, after compression and including
file system metadata, is actually written to disk and necessary for
all data to be accessible; it's the relevant metric for whether I need
to add disk space.

"Logical" is the sum of all apparent file sizes as visible to userland
utilities e.g. through stat(2).

Something like `dd if=/dev/zero of=$(mktemp) bs=1M count=1M` would
result in a large logical size but, because of compression, a very
small amount of physically stored data.

"Hot" is perhaps better referred to as the "current" data set; since
snapshots (and earlier backups) can include data which has since been
deleted, and is thus no longer current but still exists on disk.


> What partitioning scheme, volume manager, file system, compression, etc., do
> you use on your backup server?

ZFS within LUKS containers. If I recall correctly, the backup pool is
set to use zstd compression.


> I had thought you were using rsnapsnot/ rsync --link-dest, but you also
> mention ZFS snapshots.  Please clarify.

Mostly ZFS with a rotating snapshot schedule on the source (the root
file system is ext4); copied using rsync --link-dest (through
rsnapshot) to a ZFS file system which doesn't use snapshots on the
backup target. Most of the ZFS file systems are set up to use
compression; there are a few where I know _a priori_ that the data is
in effect completely incompressible so there's no point in using CPU
to even try to compress that data, so those have compression turned
off.

(In ZFS, creating a file system is barely any more involved than
creating a directory, and all file systems come out of the same "pool"
which is a collection of >=1 storage devices set up with some
particular method of redundancy, possibly none. In more traditional
*nix parlace, a *nix file system is conceptually closer to a ZFS
pool.)

Hopefully this is more clear.

-- 
Michael Kjörling                     🔗 https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

Back to linux.debian.user | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

I uninstalled OpenMediaVault (because totally overkill for me) and  replaced it with borgbackup and rsync Jason <jason@7748229.xyz> - 2023-09-01 06:50 +0200
  Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsync mlnl <mlnl@mailbox.org> - 2023-09-01 08:20 +0200
  Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsync jeremy ardley <jeremy.ardley@gmail.com> - 2023-09-01 08:40 +0200
  Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsync Manphiz <manphiz@gmail.com> - 2023-09-01 12:40 +0200
  Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq John Hasler <john@sugarbit.com> - 2023-09-01 14:30 +0200
    Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq Default User <hunguponcontent@gmail.com> - 2023-09-01 21:30 +0200
      Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq Michel Verdier <mv524@free.fr> - 2023-09-01 22:50 +0200
        Re: I uninstalled OpenMediaVault (because totally overkill for me)          and replaced it with borgbackup and rsyncq Linux-Fan <Ma_Sys.ma@web.de> - 2023-09-01 23:30 +0200
          Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq Michel Verdier <mv524@free.fr> - 2023-09-02 00:10 +0200
            Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq Michael Kjörling <2695bd53d63c@ewoof.net> - 2023-09-02 11:20 +0200
              Backup systems (was: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq) Stefan Monnier <monnier@iro.umontreal.ca> - 2023-09-02 19:30 +0200
                Re: Backup systems Michel Verdier <mv524@free.fr> - 2023-09-02 21:20 +0200
                Re: Backup systems David Christensen <dpchrist@holgerdanske.com> - 2023-09-02 23:50 +0200
                Re: Backup systems Michel Verdier <mv524@free.fr> - 2023-09-03 00:30 +0200
                Re: Backup systems David Christensen <dpchrist@holgerdanske.com> - 2023-09-03 02:00 +0200
                Re: Backup systems Michel Verdier <mv524@free.fr> - 2023-09-03 11:00 +0200
                Re: Backup systems Michael Kjörling <2695bd53d63c@ewoof.net> - 2023-09-03 12:10 +0200
                Re: Backup systems David Christensen <dpchrist@holgerdanske.com> - 2023-09-03 23:30 +0200
                Re: Backup systems Michael Kjörling <2695bd53d63c@ewoof.net> - 2023-09-04 10:00 +0200
                Re: Backup systems David Christensen <dpchrist@holgerdanske.com> - 2023-09-04 23:00 +0200
                Re: Backup systems Michael Kjörling <2695bd53d63c@ewoof.net> - 2023-09-05 16:40 +0200
                Re: Backup systems David Christensen <dpchrist@holgerdanske.com> - 2023-09-06 02:20 +0200
              Re: I uninstalled OpenMediaVault (because totally overkill for me)          and replaced it with borgbackup and rsyncq Linux-Fan <Ma_Sys.ma@web.de> - 2023-09-03 00:10 +0200
                Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq David <david.g_jones@ntlworld.com> - 2023-09-03 20:20 +0200
                Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq Gareth Evans <donotspam@fastmail.fm> - 2023-09-03 21:40 +0200
        Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq Default User <hunguponcontent@gmail.com> - 2023-09-06 02:10 +0200
          Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq Default User <hunguponcontent@gmail.com> - 2023-09-06 02:40 +0200
            Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq David Christensen <dpchrist@holgerdanske.com> - 2023-09-06 07:40 +0200
      Re: I uninstalled OpenMediaVault (because totally overkill for me)          and replaced it with borgbackup and rsyncq Linux-Fan <Ma_Sys.ma@web.de> - 2023-09-01 23:20 +0200
        Re: I uninstalled OpenMediaVault (because totally overkill for me)  and replaced it with borgbackup and rsyncq Default User <hunguponcontent@gmail.com> - 2023-09-06 03:40 +0200
          Re: I uninstalled OpenMediaVault (because totally overkill for me)          and replaced it with borgbackup and rsyncq Linux-Fan <Ma_Sys.ma@web.de> - 2023-09-09 19:00 +0200

csiph-web