Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.debian.user > #225505 > unrolled thread
| Started by | Patrick Bartek <patrick.bartek@gmail.com> |
|---|---|
| First post | 2020-07-29 22:00 +0200 |
| Last post | 2020-08-04 06:30 +0200 |
| Articles | 20 on this page of 114 — 18 participants |
Back to article view | Back to linux.debian.user
Homebuilt NAS Advice Patrick Bartek <patrick.bartek@gmail.com> - 2020-07-29 22:00 +0200
Re: Homebuilt NAS Advice Dan Ritter <dsr@randomstring.org> - 2020-07-29 22:20 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-07-29 22:30 +0200
Re: Homebuilt NAS Advice Dan Ritter <dsr@randomstring.org> - 2020-07-29 23:40 +0200
Re: Homebuilt NAS Advice Patrick Bartek <patrick.bartek@gmail.com> - 2020-07-30 02:50 +0200
Re: Homebuilt NAS Advice Tom Dial <tddial@comcast.net> - 2020-07-30 22:50 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-02 01:40 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-02 03:10 +0200
Re: Homebuilt NAS Advice Keith Bainbridge <ke1th3216@gmail.com> - 2020-08-02 04:50 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-02 06:10 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-02 21:30 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-02 22:10 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-03 06:10 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-03 17:30 +0200
Re: Homebuilt NAS Advice Dan Ritter <dsr@randomstring.org> - 2020-08-03 17:40 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-03 18:40 +0200
Re: Homebuilt NAS Advice <tomas@tuxteam.de> - 2020-08-03 19:10 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-03 22:00 +0200
Re: Homebuilt NAS Advice <tomas@tuxteam.de> - 2020-08-03 22:20 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-03 23:50 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-04 00:50 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-04 00:40 +0200
Re: Homebuilt NAS Advice Anssi Saari <as@sci.fi> - 2020-07-30 10:00 +0200
Re: Homebuilt NAS Advice rhkramer@gmail.com - 2020-07-30 13:20 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-07-30 01:50 +0200
Re: Homebuilt NAS Advice Patrick Bartek <nemommxiv@gmail.com> - 2020-07-30 23:40 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-08 01:30 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-08 03:30 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-09 14:40 +0200
Re: Homebuilt NAS Advice Peter Ehlert <peter@sdi-baja.com> - 2020-07-30 19:00 +0200
Re: Homebuilt NAS Advice Charles Curley <charlescurley@charlescurley.com> - 2020-07-30 21:50 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-07-31 13:10 +0200
Re: Homebuilt NAS Advice Patrick Bartek <nemommxiv@gmail.com> - 2020-07-30 23:30 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-02 01:50 +0200
Re: Homebuilt NAS Advice Patrick Bartek <nemommxiv@gmail.com> - 2020-08-02 17:50 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-03 17:20 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-04 00:10 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-04 01:20 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-04 10:40 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-04 11:30 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-04 16:00 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-04 16:30 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-05 05:50 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-05 16:20 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-05 17:00 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-05 17:40 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 12:10 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-06 16:10 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 17:10 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-06 18:10 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-07 18:30 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-07 19:40 +0200
Re: Homebuilt NAS Advice rhkramer@gmail.com - 2020-08-08 00:10 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-09 04:30 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-09 11:10 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-09 17:00 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-09 18:30 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-09 19:20 +0200
Re: Homebuilt NAS Advice rhkramer@gmail.com - 2020-08-09 23:50 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-10 00:00 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-10 00:30 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-10 00:50 +0200
Re: Homebuilt NAS Advice rhkramer@gmail.com - 2020-08-10 00:30 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-10 00:30 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-09 18:40 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-09 23:00 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-06 18:20 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-07 05:30 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-07 18:30 +0200
Re: Homebuilt NAS Advice rhkramer@gmail.com - 2020-08-06 17:20 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-06 18:20 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-06 18:20 +0200
Re: Homebuilt NAS Advice Andrei POPESCU <andreimpopescu@gmail.com> - 2020-08-04 17:20 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-05 06:20 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-05 04:30 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-05 09:20 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-05 11:00 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-05 21:00 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 03:50 +0200
Re: Homebuilt NAS Advice Kushal Kumaran <kushal@locationd.net> - 2020-08-06 06:20 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-06 18:20 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-07 04:10 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-07 05:50 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-05 12:30 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-05 12:30 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-05 21:10 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-05 23:30 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-06 00:30 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 01:20 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-08 09:10 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-06 04:30 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 04:50 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-06 06:20 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 07:10 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 07:30 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-06 18:10 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-06 18:30 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-07 03:00 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-07 03:20 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-07 05:00 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-07 05:40 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-07 08:00 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-07 08:50 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-07 09:10 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-07 17:50 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-08 01:40 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-08 04:50 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-08 22:30 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-09 00:20 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-09 01:30 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-09 22:40 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-10 00:30 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-10 01:30 +0200
Re: Homebuilt NAS Advice Patrick Bartek <nemommxiv@gmail.com> - 2020-08-04 06:30 +0200
Page 4 of 6 — ← Prev page 1 2 3 [4] 5 6 Next page →
| From | Stefan Monnier <monnier@iro.umontreal.ca> |
|---|---|
| Date | 2020-08-10 00:30 +0200 |
| Message-ID | <AC1Zf-6vS-1@gated-at.bofh.it> |
| In reply to | #226048 |
> Perhaps, but that has nothing to do with whether the host is down or up
> - any time you're doing a recovery, you're going to run into this
> issue, insofar as it's an issue.
I wasn't thinking of recovery, but rather the case where you're at the
office and need to access a file that you accidentally left on your home
computer which is now turned off.
Stefan
[toc] | [prev] | [next] | [standalone]
| From | David Christensen <dpchrist@holgerdanske.com> |
|---|---|
| Date | 2020-08-10 00:50 +0200 |
| Message-ID | <AC2iB-6Ct-1@gated-at.bofh.it> |
| In reply to | #226051 |
On 2020-08-09 15:22, Stefan Monnier wrote: > I wasn't thinking of recovery, but rather the case where you're at the > office and need to access a file that you accidentally left on your home > computer which is now turned off. I have pondered putting my home directory on the Samba server, which is up 24x7 and accessible both from the LAN and from the Internet via a VPN. Most everything I do should handle this, except for Thunderbird -- e.g. I start Thunderbird on my desktop, lock the screen, go remote with the laptop, connect the VPN, and start Thunderbird. Now I have two Thunderbird instances accessing the same backing store. That sounds like a good way to trash my e-mail. The only solution I can think of is to keep the Thunderbird instances separate and keep my messages on the IMAP server. David
[toc] | [prev] | [next] | [standalone]
| From | rhkramer@gmail.com |
|---|---|
| Date | 2020-08-10 00:30 +0200 |
| Message-ID | <AC1Zf-6vS-7@gated-at.bofh.it> |
| In reply to | #226048 |
On Sunday, August 09, 2020 05:53:56 PM Celejar wrote: > On Sun, 9 Aug 2020 17:42:43 -0400 > rhkramer@gmail.com wrote: > > On Sunday, August 09, 2020 12:22:16 PM Celejar wrote: > > > Are you saying that classic forward-delta > > > based systems don't store backup metadata (or whatever the right term > > > is) on the backup server, but on the host? > > > > I don't think he is saying that (but I could be wrong) (and I think that > > is pretty clear from subsequent comments, but I wanted to perhaps make > > it a little more clear), iiuc, he's just pointing out things like the > > overhead on recovery when you have to take the base file and apply the > > forward deltas to it. > > Perhaps, but that has nothing to do with whether the host is down or up Exactly -- I was trying to address your question about needing metadata (or anything) from the host in order to recover files. > - any time you're doing a recovery, you're going to run into this > issue, insofar as it's an issue.
[toc] | [prev] | [next] | [standalone]
| From | David Christensen <dpchrist@holgerdanske.com> |
|---|---|
| Date | 2020-08-10 00:30 +0200 |
| Message-ID | <AC1Zf-6vS-5@gated-at.bofh.it> |
| In reply to | #226039 |
On 2020-08-09 07:53, Stefan Monnier wrote:
> it's convenient to make your latest backup readily accessible so
> it's very easy to get back yesterday's version of a file in case your
> fingers fumbled (which might be much more frequent than drive failures).
With ZFS and zfs-auto-snapshot, recovery can be done on the live server:
$ cp -p mountpoint/.zfs/snapshot/snapname/path/to/file
mountpoint/path/to/file
This is my personal Samba directory:
2020-08-09 14:38:51 toor@f32020-08-09 14:55:11 toor@f3 ~
# du -A --si -s /jail/samba/var/local/samba/dpchrist
780G /jail/samba/var/local/samba/dpchrist
From May 30, 2019, to the start of the last hour, I have 179 snapshots
to choose from:
2020-08-09 14:37:35 toor@f3 ~
# zfs list -d 1 -t snapshot p3/ds2/samba/dpchrist | head -n 2
NAME USED AVAIL
REFER MOUNTPOINT
p3/ds2/samba/dpchrist@manual-20190530-1804 8.45M -
706G -
2020-08-09 14:37:50 toor@f3 ~
# zfs list -d 1 -t snapshot p3/ds2/samba/dpchrist | tail -n 1
p3/ds2/samba/dpchrist@zfs-auto-snap_h-2020-08-09-14h03 0 -
716G -
2020-08-09 14:38:35 toor@f3 ~
# zfs list -d 1 -t snapshot p3/ds2/samba/dpchrist | wc -l
179
(This is too many snapshots. I need to thin them out and rework my
replication scripts to handle source deletions.)
ZFS snapshots are essentially de-duplicated (185 to 1):
2020-08-09 14:51:36 toor@f3 ~
# du -A --si -s /jail/samba/var/local/samba/dpchrist/.zfs/snapshot
141T /jail/samba/var/local/samba/dpchrist/.zfs/snapshot
David
[toc] | [prev] | [next] | [standalone]
| From | Celejar <celejar@gmail.com> |
|---|---|
| Date | 2020-08-09 18:40 +0200 |
| Message-ID | <ABWwy-3bc-15@gated-at.bofh.it> |
| In reply to | #226033 |
On Sun, 9 Aug 2020 10:07:33 +0100 Jonathan Dowland <jon+debian-user@dow.land> wrote: > On Sat, Aug 08, 2020 at 10:22:51PM -0400, Celejar wrote: > >Fair enough. But less applicable in the case of backups, since restores > >are quite rare, as I've been pointing out. > > I'd agree they're probably rare, and certainly less common than backups. > But in my experience when you do perform a restore it's almost always > the latest version you want. They should not be as rare as they are. We > should all be in the habit of regularly performing a restore to test our > backups are working. Fair enough. But still, let's say you backup daily, and do a test restore once a month. Is the extra cost of daily rebasing the backup worth the once a month cost saving of a simpler, direct restore? I suppose, though, that with modern (or at least non-ancient) hardware the cost is not that great either way. > One advantage of the reverse-delta storage mechanism: With rdiff-backup, > if I want to restore the latest version I can simply copy the backup > repository (ignoring/excluding "rdiff-backup-increments" directory), I > do not need to use rdiff-backup itself. Which could be useful if the > backup tool breaks, or you can't easily install it on a restore target > for whatever reason (It requires Python 2 for example and that becomes > harder to get). Or if a catastrophic bug in the backup software means > the incremental storage scheme is corrupted in some way. Sure - but this wouldn't apply to backup systems that don't simply store even their full backups as ordinary files, but rather according to some internal protocol (I'm not quite sure about dar, the original topic of this sub-thread), and anyway require the native restore software to do a restore. Celejar
[toc] | [prev] | [next] | [standalone]
| From | David Christensen <dpchrist@holgerdanske.com> |
|---|---|
| Date | 2020-08-09 23:00 +0200 |
| Message-ID | <AC0A9-5xR-1@gated-at.bofh.it> |
| In reply to | #226033 |
On 2020-08-09 02:07, Jonathan Dowland wrote: > We should all be in the habit of regularly performing a restore to test > our backups are working. I would add -- "to a spare server". I am loath to overwrite my primary server data, out of subliminal fear conditioned by past bad experiences. High availability is the logical extension of backup-restore-migrate. This requires services that support it: https://www.samba.org/samba/docs/old/Samba3-HOWTO/SambaHA.html https://www.postgresql.org/docs/9.5/high-availability.html David
[toc] | [prev] | [next] | [standalone]
| From | deloptes <deloptes@gmail.com> |
|---|---|
| Date | 2020-08-06 18:20 +0200 |
| Message-ID | <AAQMy-4gl-17@gated-at.bofh.it> |
| In reply to | #225938 |
Leslie Rhorer wrote: > And how useful is that? There are very few duplicate files on my > systems, because I use applications to eliminate duplicates. > Eliminating duplicates in a live data repository is far more important > than doing it on backup media. > May be you misunderstand deduplication or duplication. It is not about files you have on your disk, but what is already in the backup repository, so you avoid duplicating the backup of the same file for example in each full backup. Because most of the files do not change, this is how you save that much space. If you were having files that change more often, then again it depends on how much of this files change and this much goes into the backup. In fact a backup with deduplication is a sort of always incremental, but the adventage is that you can do a full restore from any backup, which is actually cool.
[toc] | [prev] | [next] | [standalone]
| From | Leslie Rhorer <lesrhorer@att.net> |
|---|---|
| Date | 2020-08-07 05:30 +0200 |
| Message-ID | <AB1eV-28f-1@gated-at.bofh.it> |
| In reply to | #225947 |
On 8/6/2020 11:09 AM, deloptes wrote: > Leslie Rhorer wrote: > >> And how useful is that? There are very few duplicate files on my >> systems, because I use applications to eliminate duplicates. >> Eliminating duplicates in a live data repository is far more important >> than doing it on backup media. >> > > May be you misunderstand deduplication or duplication. It is not about files > you have on your disk, but what is already in the backup repository, so you > avoid duplicating the backup of the same file for example in each full > backup. Because most of the files do not change, this is how you save that > much space. That is the default for DAR for a given existing isolate*. Any backup software that is aware of its own history can accomplish this, even TAR. The command `tar -u` will append only new files or files that have changed to the end of the archive. All you seem to be talking about is a differential backup. * - An isolate is essentially a listing of all the files associated with an archive along with the history if its actions. DAR has the ability to isolate a backup and work from it. The isolate is tiny.
[toc] | [prev] | [next] | [standalone]
| From | Jonathan Dowland <jon+debian-user@dow.land> |
|---|---|
| Date | 2020-08-07 18:30 +0200 |
| Message-ID | <ABdpM-166-5@gated-at.bofh.it> |
| In reply to | #225938 |
On Thu, Aug 06, 2020 at 10:04:28AM -0500, Leslie Rhorer wrote: > And how useful is that? There are very few duplicate files on my >systems, because I use applications to eliminate duplicates. >Eliminating duplicates in a live data repository is far more important >than doing it on backup media. The scenario in which it is useful is when you have moved a large amount of data around in the source, and the next incremental backup needs to account for that. Some tools (e.g. rdiff-backup) will generate a large incremental backup, which contains the data for all of the moved files in their new locations. A Borg backup merely notes the new path of the files. You can therefore store more backups in the same amount of space. It's actually one of the main reasons I am moving from rdiff-backup to Borg. -- Please do not CC me, I am subscribed to the list. 👱🏻 Jonathan Dowland ✎ jmtd@debian.org 🔗 https://jmtd.net
[toc] | [prev] | [next] | [standalone]
| From | rhkramer@gmail.com |
|---|---|
| Date | 2020-08-06 17:20 +0200 |
| Message-ID | <AAPQt-3GE-1@gated-at.bofh.it> |
| In reply to | #225937 |
On Thursday, August 06, 2020 10:07:59 AM Celejar wrote: > * Incremental and differential backups are backups of the delta between > the last full backup and the current system state (either individually > [differential] or collectively [incremental]) I am not Leslie Rhorer, I'm just coming out of left field, but I'd state the above a little differently (based on the reading about backups that I've done recently and past experiences). Incremental and differential backups are similar in that neither are a full backup, they are a backup of the differences between two full images (or backups, or mirrors). A differential backup (seemingly in the usage of most manufacturers) is a backup of the differences between two images but possibly not sequential images. (For example, if somehow a backup image is created every day, a differential backup could be the differences between the image on say Monday and Thursday.) In contrast (and again, seemingly in the usage of most manufacturers) is a backup of each image, for example, (in the example above), there would be an incremental backup of the differences between the Monday and Tuesday images, another incremental backup of the differences between the Tuesday and Wednesday images, and so on. > * I have no idea (nor do Google, DuckDuckGo, or Wikipedia) what a > "decremental backup" is. Perhaps he means (I shouldn't put words in his mouth) a reverse backup? Forward backups of differences are those that list the changes to make an older image into a newer image. You can do the reverse, and list the changes required to make a newer image into an older image. Believe it or not, there are sometimes advantages to that approach, which I won't get into very deeply. But one is, if you do the backups in reverse, the corrolary to that is that, the current full image is the latest, and maybe the most likely to be needed. (Of course, it takes a little longer to arrange the backups in reverse.) Another is, that if you want to get rid of older backups, that is fairly easily done by deleting the older differences. In forward backups, if you want to get rid of older backups, you have to create a "new" old / base image. There is more. > > * Borg doesn't fit the "full / differential / incremental" paradigm > neatly, but it certainly has some of the advantages of differential / > incremental backups (plus others that classic differential / > incremental ones do not have, such as deduplication - i.e., if the > backup source contains multiple copies of some data, that dataonly > needs to be stored once in the backup target). > > * I don't know exactly what you mean by "deletion restoral", but Borg > (and, I assume, many other good backup solutions) offers a > flexible variety of methods to restore deleted files, and / or to > restore a snapshot of the backup source as it existed at a given point > in time: https://docs.borgbase.com/restore/ > > If you are willing to explain further, that would be appreciated. > Thanks! > > Celejar
[toc] | [prev] | [next] | [standalone]
| From | Celejar <celejar@gmail.com> |
|---|---|
| Date | 2020-08-06 18:20 +0200 |
| Message-ID | <AAQMy-4gl-15@gated-at.bofh.it> |
| In reply to | #225939 |
On Thu, 6 Aug 2020 11:11:41 -0400 rhkramer@gmail.com wrote: > On Thursday, August 06, 2020 10:07:59 AM Celejar wrote: > > * Incremental and differential backups are backups of the delta between > > the last full backup and the current system state (either individually > > [differential] or collectively [incremental]) > > I am not Leslie Rhorer, I'm just coming out of left field, but I'd state the > above a little differently (based on the reading about backups that I've done > recently and past experiences). > > Incremental and differential backups are similar in that neither are a full > backup, they are a backup of the differences between two full images (or > backups, or mirrors). > > A differential backup (seemingly in the usage of most manufacturers) is a > backup of the differences between two images but possibly not sequential > images. (For example, if somehow a backup image is created every day, a > differential backup could be the differences between the image on say Monday and > Thursday.) > > In contrast (and again, seemingly in the usage of most manufacturers) is a > backup of each image, for example, (in the example above), there would be an > incremental backup of the differences between the Monday and Tuesday images, > another incremental backup of the differences between the Tuesday and Wednesday > images, and so on. I think this is what I said, albeit expressed at greater length and more clearly ;) > > * I have no idea (nor do Google, DuckDuckGo, or Wikipedia) what a > > "decremental backup" is. > > Perhaps he means (I shouldn't put words in his mouth) a reverse backup? > Forward backups of differences are those that list the changes to make an older > image into a newer image. You can do the reverse, and list the changes > required to make a newer image into an older image. > > Believe it or not, there are sometimes advantages to that approach, which I > won't get into very deeply. > > But one is, if you do the backups in reverse, the corrolary to that is that, > the current full image is the latest, and maybe the most likely to be needed. > (Of course, it takes a little longer to arrange the backups in reverse.) Sure. But I'm not sure I see any significant advantage in this - slightly cheaper restores, at the cost of more expensive backups? Given that I do many, many backups, and very few restores, it hardly seems worth it. > Another is, that if you want to get rid of older backups, that is fairly > easily done by deleting the older differences. In forward backups, if you want > to get rid of older backups, you have to create a "new" old / base image. True. But that work, of rebasing backups, you're going to be doing all the time with decremental, instead of just when pruning with normal incremental / differential, so how is this an advantage? Celejar
[toc] | [prev] | [next] | [standalone]
| From | Stefan Monnier <monnier@iro.umontreal.ca> |
|---|---|
| Date | 2020-08-06 18:20 +0200 |
| Message-ID | <AAQMy-4gl-13@gated-at.bofh.it> |
| In reply to | #225930 |
>> I find it hard to believe that it's a rare feature.
>> The system I use (`bup`) does support that as well (every backup is
>> (more or less) a Git commit, so it doesn't distinguish full-backups from
>> incrementals), but it doesn't bother to mention it probably because it's
>> very basic.
>
> Uh-uh. Full, incremenal, and decremental backups are all completely
> different than deletion restoral.
In `bup` you basically only have full backups (but at the cost of an
incremental backup). So you never apply an incremental backup after
extracting a full backup. And hence there's never any need to do
"deletion restoral".
Stefan
[toc] | [prev] | [next] | [standalone]
| From | Andrei POPESCU <andreimpopescu@gmail.com> |
|---|---|
| Date | 2020-08-04 17:20 +0200 |
| Message-ID | <AA6Tn-1AQ-5@gated-at.bofh.it> |
| In reply to | #225798 |
[Multipart message — attachments visible in raw view] — view raw
On Ma, 04 aug 20, 01:17:13, deloptes wrote: > > I don't backup music and video - too big and not changing. The raid is > enough for that. I'm guessing it depends on the music and videos. If you made them yourself they are basically irreplaceable and RAID will replicate any bitrot to the other copy undetected. Kind regards, Andrei -- http://wiki.debian.org/FAQsFromDebianUser
[toc] | [prev] | [next] | [standalone]
| From | Leslie Rhorer <lesrhorer@att.net> |
|---|---|
| Date | 2020-08-05 06:20 +0200 |
| Message-ID | <AAj4d-z1-3@gated-at.bofh.it> |
| In reply to | #225837 |
On 8/4/2020 10:18 AM, Andrei POPESCU wrote: > On Ma, 04 aug 20, 01:17:13, deloptes wrote: >> >> I don't backup music and video - too big and not changing. The raid is >> enough for that. > > I'm guessing it depends on the music and videos. > > If you made them yourself they are basically irreplaceable and RAID will > replicate any bitrot to the other copy undetected. Even if not, they still represent a great deal of time to accumulate and then to replace. Most of mine have been recorded* from Cable TV programs, and replacing them would require either waiting a huge amount of time for all of them to be re-broadcast (if ever), or else purchasing literally thousands of Blu-Ray disks. Again, I cannot possily reiterate that RAID is #NOT# a backup solution. On several separate occasions I have lost an entire RAID array. Each involved a completely different type of failure. In two cases I was able to manually recover the arrays with only minimal loss of data. In one, no data recovery at all was possible (a Windows system). In every other case I had to resort to backup. One of those times, it was the backup system which failed, and then DAR really saved my butt, as it has done on a number of occasions when the corruption was far less extensive. * - All perfectly legally
[toc] | [prev] | [next] | [standalone]
| From | Leslie Rhorer <lesrhorer@att.net> |
|---|---|
| Date | 2020-08-05 04:30 +0200 |
| Message-ID | <AAhlM-7WC-1@gated-at.bofh.it> |
| In reply to | #225798 |
On 8/3/2020 6:17 PM, deloptes wrote: > Leslie Rhorer wrote: > >> My main server's data is backed up every morning by a >> nearly identical backup server using rsync, which in turn is backed up >> in its entirety on a multi-volume offline, off-site hard drive set using >> DAR every few months. Â All the boot systems employ two disk RAID 1 >> mirrors and all the data repositories consist of 5 - 8 spindle RAID 6 >> arrays. > > any thoughts on using deduplication? For example I started using borg some > time ago. It saves a lot of space and makes it possible to have multiple > backups and longer retention. I don't backup music and video - too big and > not changing. The raid is enough for that. DAR has a number of unique capabilities, including restoring deletions, which is a very unique feature. I don't know about you, but my video files represent many hundreds of hours of work, and replacing them en masse would be a huge and expensive undertaking. The backup server maintains a daily update of all changes and new files. the server just isn't all that expensive, comparatively speaking. The DAR backup is even cheaper, since it is done on old drives removed from the arrays during upgrades. In one sense, they are free. DAR allows for not only incremental backups, but also incremental deletions. I find it extremely useful, and allows for as much space saving on the live system as one likes. I suggest you check it out.
[toc] | [prev] | [next] | [standalone]
| From | deloptes <deloptes@gmail.com> |
|---|---|
| Date | 2020-08-05 09:20 +0200 |
| Message-ID | <AAlSp-2j1-1@gated-at.bofh.it> |
| In reply to | #225864 |
Leslie Rhorer wrote: > DAR allows for not only incremental backups, but also incremental > deletions. Â I find it extremely useful, and allows for as much space > saving on the live system as one likes. Â I suggest you check it out. someone should ask them to implement deduplication, because nowdays it is impossible to do meaningful backups without. May I ask what is your active disk size and how big is the disk size of the backup? What kind of compression does it (DAR) support. thanks
[toc] | [prev] | [next] | [standalone]
| From | Leslie Rhorer <lesrhorer@att.net> |
|---|---|
| Date | 2020-08-05 11:00 +0200 |
| Message-ID | <AAnrb-34D-5@gated-at.bofh.it> |
| In reply to | #225872 |
On 8/5/2020 2:16 AM, deloptes wrote: > Leslie Rhorer wrote: > >> DAR allows for not only incremental backups, but also incremental >> deletions. ??I find it extremely useful, and allows for as much space >> saving on the live system as one likes. ??I suggest you check it out. > > someone should ask them to implement deduplication, because nowdays it is > impossible to do meaningful backups without. In this context what, exactly, is de-duplication? I fail to see how any meaningful interpretation of the term is salient to backups. To compression, yes, to symbolic interpretation, surely, and to saving space on a drive and reducing access times, you bet. To backups? I don't really see it, unless you mean hard-link handling, which it does most admirably. Soft links, of course, are fairly straightforward. DAR does handle sparse files exceedingly well. > May I ask what is your active disk size What do you mean by "active" disk size? In each of my main arrays there are 8 spindles of 8 Terabytes each. Six spindles worth are encoded with flat data and 2 spindles worth with parity. RAID 6 does not assign any disks specifically for data or for parity as RAID 3 and RAID 4 do. Instead, with both RAID 5 and RAID 6, parity is distributed across every drive, and the data is also distributed across all the drives, interleaved with the parity. All put together, the available volume size is 46.9 Terabytes (43.6 Teribytes) after formatting. The main server currently has 22 Terabytes of data on it. The backup server is effectively full. > and how big is the disk size of the > backup? The individual drives vary in size. Mostly they are 3 Terabytes, but some are 2T and some are 4T. Future backups are going to be to 5T and 8T drives. It doesn't really matter. DAR does not care. For ease and simplicity, I usually format each drive into a single file system and direct DAR to use slices just under 100G. This results in no more than 100G of wasted space on any drive, and usually far less. > What kind of compression does it (DAR) support. With my main system, it doesn't really matter, because the vast majority of the data (more than 95%) is incompressible. In general, gzip, bzip2, lzo, xz/lzma algorithms are directly supported by DAR, and DAR supports both inclusion and exclusion parameters.
[toc] | [prev] | [next] | [standalone]
| From | deloptes <deloptes@gmail.com> |
|---|---|
| Date | 2020-08-05 21:00 +0200 |
| Message-ID | <AAwNQ-gO-3@gated-at.bofh.it> |
| In reply to | #225876 |
Leslie Rhorer wrote:
> In this context what, exactly, is de-duplication? Â I fail to see how
> any meaningful interpretation of the term is salient to backups. Â To
> compression, yes, to symbolic interpretation, surely, and to saving
> space on a drive and reducing access times, you bet. Â To backups? Â I
> don't really see it, unless you mean hard-link handling, which it does
> most admirably. Â Soft links, of course, are fairly straightforward. Â DAR
> does handle sparse files exceedingly well.
>
Imagine you have classical backup: daily incrementals, full weekly and full
monthly. Imagine you have retention for the full weekly 3 (until end of
month) and for full monthly 12 (until end of year).
You have to maintain 15 full backups and the 6 daily incrementals. How much
space is it, that you need for your backup storage?
This is why the question what is your active size. No imagine I have 2TB of
data, even if I compress this data - lets say with avg. 60% ratio it is
800GB per full backup. 15 copies + means 12TB+. Of course if you have
video/audio like mp3, it is already compressed and ratio for the backup
compression goes down and space needed up.
Now here comes the trick with deduplication. The backup system makes one
full backup (800GB) and then keeps track of the bits that changed (it is
not that simple, but for the example). Only they are being backuped. Some
systems provide ratio of 90%. So to keep your 15+ copies with deduplication
ratio of ~80% you need about 3TB.
>> May I ask what is your active disk size
>
> What do you mean by "active" disk size? Â In each of my main arrays
> there are 8 spindles of 8 Terabytes each. Â Six spindles worth are
> encoded with flat data and 2 spindles worth with parity. Â RAID 6 does
> not assign any disks specifically for data or for parity as RAID 3 and
> RAID 4 do. Â Instead, with both RAID 5 and RAID 6, parity is distributed
> across every drive, and the data is also distributed across all the
> drives, interleaved with the parity. Â All put together, the available
> volume size is 46.9 Terabytes (43.6 Teribytes) after formatting. Â The
> main server currently has 22 Terabytes of data on it. Â The backup server
> is effectively full.
>
So you have a perfect candidate for deduplication :) because I guess you can
keep only few copies of that size on the backup server.
Live example here one of the servers with borg
the backup archive
------------------------------------------------------------------------------
Original size Compressed size Deduplicated
size
All archives: 5.47 TB 3.37 TB 483.58
GB
Unique chunks Total chunks
Chunk index: 2342114 23644792
The last monthly
Archive name: 2020-07-04T22:01:21
Archive fingerprint: xxxxxxxxxxxxxxxxx
Comment:
Hostname: xxxx
Username: xxxx
Time (start): Sat, 2020-07-04 22:01:32
Time (end): Sat, 2020-07-04 23:28:51
Duration: 1 hours 27 minutes 19.92 seconds
Number of files: 3416089
Command line: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Utilization of maximum supported archive size: 1%
------------------------------------------------------------------------------
Original size Compressed size Deduplicated
size
This archive: 807.19 GB 493.13 GB 14.90
GB
All archives: 5.47 TB 3.37 TB 483.58
GB
Unique chunks Total chunks
Chunk index: 2342114 23644792
[toc] | [prev] | [next] | [standalone]
| From | Leslie Rhorer <lesrhorer@att.net> |
|---|---|
| Date | 2020-08-06 03:50 +0200 |
| Message-ID | <AADcB-4b4-1@gated-at.bofh.it> |
| In reply to | #225909 |
On 8/5/2020 1:51 PM, deloptes wrote: > Imagine you have classical backup: daily incrementals, full weekly and full > monthly. This is not required with DAR. One full backup is all that is required. Anything else is a waste of time and space. One can employ either differential or decremental backups, keeping a full history of all activity on the system. Imagine you have retention for the full weekly 3 (until end of > month) and for full monthly 12 (until end of year). > You have to maintain 15 full backups and the 6 daily incrementals. How much > space is it, that you need for your backup storage? > This is why the question what is your active size. No imagine I have 2TB of > data, even if I compress this data - lets say with avg. 60% ratio it is > 800GB per full backup. 15 copies + means 12TB+. Of course if you have > video/audio like mp3, it is already compressed and ratio for the backup > compression goes down and space needed up. > Now here comes the trick with deduplication. The backup system makes one > full backup (800GB) and then keeps track of the bits that changed (it is > not that simple, but for the example). Only they are being backuped. Some > systems provide ratio of 90%. So to keep your 15+ copies with deduplication > ratio of ~80% you need about 3TB. You are talking about differential archiving at the binary level. DAR can employ delta binaries for both incremental and decremental backups, which means only the changes to the binary data, if any, is stored in subsequent backups, not the entire file. There is no need for multiple full backups. A full backup of my main array to hard drive media will take a minimum of 15 days. Transferring across the 10G link to my backup server would take a minimum of 2.8 days. Although full backups of smaller systems will normally take less time, the fact is full backups on a regular basis are just not necessary, and especially not when using DAR. >> volume size is 46.9 Terabytes (43.6 Teribytes) after formatting. ??The >> main server currently has 22 Terabytes of data on it. ??The backup server >> is effectively full. >> > > So you have a perfect candidate for deduplication :) because I guess you can > keep only few copies of that size on the backup server. A few copies of what size? The backup server is an exact mirror of the main server, plus several T of additional files I don't need on the main server. My DAR backups include new files (typically a few hundred GB) plus changes to files (typically a few MB). Borg isn't really offering anything DAR does not, except DAR is not server based. If I wanted to run DAR on my backup server, I could, but I don't want to. I use DAR for offline archival storage. Thew backup array is a mirror, not an archive. In the case of a catastrophic failure of the main server or its array, I simply bring up the backup server as a replacement until such time - days later - as the main server is rebuilt. In the case of an accidentally lost or corrupted file(s), I just do a cp from the backup to the main server, restoring the files to the state they were in as of 04:00 this morning. Typically, this takes a few seconds.
[toc] | [prev] | [next] | [standalone]
| From | Kushal Kumaran <kushal@locationd.net> |
|---|---|
| Date | 2020-08-06 06:20 +0200 |
| Message-ID | <AAFxL-5Lr-5@gated-at.bofh.it> |
| In reply to | #225919 |
Leslie Rhorer <lesrhorer@att.net> writes: > On 8/5/2020 1:51 PM, deloptes wrote: > >> Imagine you have classical backup: daily incrementals, full weekly and full >> monthly. > > This is not required with DAR. One full backup is all that is > required. Anything else is a waste of time and space. One can employ > either differential or decremental backups, keeping a full history of > all activity on the system. > > Imagine you have retention for the full weekly 3 (until end of >> month) and for full monthly 12 (until end of year). >> You have to maintain 15 full backups and the 6 daily incrementals. How much >> space is it, that you need for your backup storage? >> This is why the question what is your active size. No imagine I have 2TB of >> data, even if I compress this data - lets say with avg. 60% ratio it is >> 800GB per full backup. 15 copies + means 12TB+. Of course if you have >> video/audio like mp3, it is already compressed and ratio for the backup >> compression goes down and space needed up. >> Now here comes the trick with deduplication. The backup system makes one >> full backup (800GB) and then keeps track of the bits that changed (it is >> not that simple, but for the example). Only they are being backuped. Some >> systems provide ratio of 90%. So to keep your 15+ copies with deduplication >> ratio of ~80% you need about 3TB. > > You are talking about differential archiving at the binary > level. DAR can employ delta binaries for both incremental and > decremental backups, which means only the changes to the binary data, > if any, is stored in subsequent backups, not the entire file. There > is no need for multiple full backups. A full backup of my main array > to hard drive media will take a minimum of 15 days. Transferring > across the 10G link to my backup server would take a minimum of 2.8 > days. Although full backups of smaller systems will normally take > less time, the fact is full backups on a regular basis are just not > necessary, and especially not when using DAR. > Deduplication does a little more than that. Deduplicated backup systems can identify duplicated content across different files in addition to the differential backups you mention. This can help when you might have identical (or close to identical) content across multiple files. Things like virtual machine disk images deduplicate rather well. Also helpful when you might have multiple users storing identical/similar files. Sophisticated deduplication systems can even deduplicate across multiple machines being backed up. -- regards, kushal
[toc] | [prev] | [next] | [standalone]
Page 4 of 6 — ← Prev page 1 2 3 [4] 5 6 Next page →
Back to top | Article view | linux.debian.user
csiph-web