Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #178680 > unrolled thread

Guide(s?) to backup philosophies

Started byRichard Owlett <rowlett@cloud85.net>
First post2017-03-11 16:20 +0100
Last post2017-03-19 11:40 +0100
Articles 5 on this page of 45 — 18 participants

Back to article view | Back to linux.debian.user

Contents

  Guide(s?) to backup philosophies Richard Owlett <rowlett@cloud85.net> - 2017-03-11 16:20 +0100
    Re: Guide(s?) to backup philosophies David Christensen <dpchrist@holgerdanske.com> - 2017-03-11 22:10 +0100
      Re: Guide(s?) to backup philosophies Dan Purgert <dan@djph.net> - 2017-03-13 14:00 +0100
        Re: Guide(s?) to backup philosophies David Christensen <dpchrist@holgerdanske.com> - 2017-03-14 05:00 +0100
          Re: Guide(s?) to backup philosophies Dan Purgert <dan@djph.net> - 2017-03-17 11:50 +0100
            Re: Guide(s?) to backup philosophies Glenn English <ghe2001@gmail.com> - 2017-03-17 19:50 +0100
            Re: Guide(s?) to backup philosophies David Christensen <dpchrist@holgerdanske.com> - 2017-03-18 06:20 +0100
              Re: Guide(s?) to backup philosophies Dan Purgert <dan@djph.net> - 2017-03-22 11:50 +0100
                Re: Guide(s?) to backup philosophies <tomas@tuxteam.de> - 2017-03-22 11:50 +0100
                  Re: Guide(s?) to backup philosophies Dan Purgert <dan@djph.net> - 2017-03-22 13:20 +0100
                    Re: Guide(s?) to backup philosophies <tomas@tuxteam.de> - 2017-03-22 13:30 +0100
                      Re: Guide(s?) to backup philosophies Dan Purgert <dan@djph.net> - 2017-03-23 10:30 +0100
                        Re: Guide(s?) to backup philosophies <tomas@tuxteam.de> - 2017-03-23 11:10 +0100
                          Re: Guide(s?) to backup philosophies Dan Purgert <dan@djph.net> - 2017-03-23 13:10 +0100
                Re: Guide(s?) to backup philosophies David Christensen <dpchrist@holgerdanske.com> - 2017-03-23 04:00 +0100
                  Re: Guide(s?) to backup philosophies Dan Purgert <dan@djph.net> - 2017-03-23 10:40 +0100
                    Re: Guide(s?) to backup philosophies David Christensen <dpchrist@holgerdanske.com> - 2017-03-23 20:10 +0100
          Re: should I firewall an open port which isn't used? (was ... Re: Guide(s?) to backup philosophies) Dominik George <nik@naturalnet.de> - 2017-03-31 14:10 +0200
            Re: should I firewall an open port which isn't used? (was ... Re:  Guide(s?) to backup philosophies) Dominik George <nik@naturalnet.de> - 2017-03-31 23:20 +0200
            Re: should I firewall an open port which isn't used? (was ... Re:  Guide(s?) to backup philosophies) Nathan Dorfman <ndorf@rtfm.net> - 2017-04-01 11:00 +0200
          should I firewall an open port which isn't used? (was ... Re:  Guide(s?) to backup philosophies) cbannister@slingshot.co.nz - 2017-03-31 14:10 +0200
            Re: should I firewall an open port which isn't used? (was ... Re:  Guide(s?) to backup philosophies) <tomas@tuxteam.de> - 2017-03-31 14:20 +0200
              Re: should I firewall an open port which isn't used? (was ... Re:  Guide(s?) to backup philosophies) Brian <ad44@cityscape.co.uk> - 2017-03-31 15:20 +0200
                Re: should I firewall an open port which isn't used? (was ... Re:  Guide(s?) to backup philosophies) <tomas@tuxteam.de> - 2017-03-31 15:30 +0200
                Re: should I firewall an open port which isn't used? (was ... Re: Guide(s?) to backup philosophies) Dominik George <nik@naturalnet.de> - 2017-03-31 15:30 +0200
                  Re: should I firewall an open port which isn't used? (was ... Re: Guide(s?) to backup philosophies) Dominik George <nik@naturalnet.de> - 2017-03-31 15:50 +0200
        Re: Guide(s?) to backup philosophies Glenn English <ghe2001@gmail.com> - 2017-03-14 19:00 +0100
          Re: Guide(s?) to backup philosophies Dan Ritter <dsr@randomstring.org> - 2017-03-14 19:20 +0100
            Re: Guide(s?) to backup philosophies Miles Fidelman <mfidelman@meetinghouse.net> - 2017-03-14 20:20 +0100
              Re: Guide(s?) to backup philosophies Dan Ritter <dsr@randomstring.org> - 2017-03-14 21:40 +0100
                Re: Guide(s?) to backup philosophies Stefan Monnier <monnier@iro.umontreal.ca> - 2017-03-15 13:30 +0100
                  Re: Guide(s?) to backup philosophies Dan Ritter <dsr@randomstring.org> - 2017-03-15 14:10 +0100
                    Re: Guide(s?) to backup philosophies "Martin McCormick" <martin.m@suddenlink.net> - 2017-03-15 14:30 +0100
                    Re: Guide(s?) to backup philosophies Jonathan Dowland <jmtd@debian.org> - 2017-03-15 16:40 +0100
                  Re: Guide(s?) to backup philosophies Jonathan Dowland <jmtd@debian.org> - 2017-03-15 16:40 +0100
                  Re: Guide(s?) to backup philosophies Stefan Monnier <monnier@iro.umontreal.ca> - 2017-03-15 21:50 +0100
                    Re: Guide(s?) to backup philosophies songbird <songbird@anthive.com> - 2017-03-16 23:50 +0100
          Re: Guide(s?) to backup philosophies Miles Fidelman <mfidelman@meetinghouse.net> - 2017-03-14 20:10 +0100
    Re: Guide(s?) to backup philosophies Joe <joe@jretrading.com> - 2017-03-11 23:30 +0100
      Re: Guide(s?) to backup philosophies Jonathan Dowland <jmtd@debian.org> - 2017-03-15 13:30 +0100
    Re: Guide(s?) to backup philosophies Dan Ritter <dsr@randomstring.org> - 2017-03-13 15:20 +0100
      Re: Guide(s?) to backup philosophies David Christensen <dpchrist@holgerdanske.com> - 2017-03-14 05:10 +0100
      Re: Guide(s?) to backup philosophies Richard Owlett <rowlett@cloud85.net> - 2017-03-14 15:50 +0100
    Re: Guide(s?) to backup philosophies Merlin Büge <toni@bluenox07.de> - 2017-03-14 19:50 +0100
    Re: Guide(s?) to backup philosophies DdB <debianlist@potentially-spam.de-bruyn.de> - 2017-03-19 11:40 +0100

Page 3 of 3 — ← Prev page 1 2 [3]

#178761

FromDan Ritter <dsr@randomstring.org>
Date2017-03-13 15:20 +0100
Message-ID<tkyZk-1HR-21@gated-at.bofh.it>
In reply to#178680
On Sat, Mar 11, 2017 at 09:10:54AM -0600, Richard Owlett wrote:
> I have one partition that might be called a "production" environment, i.e.
> fairly stable and has the most valuable content.
> A second partition hosts my experiments - I've a project to create an
> optimal install. The third is the target of those experimental installs
> whose content doesn't rate explicit backups. The scripts for creating those
> installs being on the second partition.
> 
> I've vague ideas of what backup pattern(s) I might follow.
> I'm looking for reading materials that might trigger "I hadn't thought of
> that" moments.

A quick overview:

Nobody wants backups. Everybody wants restores.

The questions are: 

- what sort of disaster are you trying to recover from

- how often do you expect each to happen

- how much time are you willing to take recovering

- how much are you willing to spend


Let's take a few common scenarios.

First: a house full of personal use machines, plus a server. We
expect files to go missing or be accidentally deleted fairly
often, and we want it to be easy and cheap to recover from that.

The general answer for that is to store files on a networked
filesystem of some sort - NFS, SMB, sshfs, whatever - which
resides on the server and is snapshotted every so often. Tools
for snapshotting include LVM (not recommended), rsnapshot, and
btrfs and zfs. Anything with a user-accessible snapshot method
is good here - sysadmins don't need to be involved in every
oops.

Second: we have the same setup, but we would also like to make
it reasonably easy to restore a whole machine when we have an
accident with the hard disk.

For that, we need image backups over the network to the server.
We won't want to snapshot these, just keep the most recent good
image. Testing these every so often is necessary.

Third scenario: running a service that makes you money. For
this, we want to be up all the time. We can spend a lot more
money on this, because we expect to make money from it.

The solutions here involve high availability: multiple machines,
possibly in multiple locations, handling the same service in a
coordinated fashion. Users need to be automatically directed to
a working instance, and we need a monitoring system to tell us
when a machine is down, because if the HA system is working we
will not get user complaints.

-dsr-

[toc] | [prev] | [next] | [standalone]

#178811

FromDavid Christensen <dpchrist@holgerdanske.com>
Date2017-03-14 05:10 +0100
Message-ID<tkLWx-30t-3@gated-at.bofh.it>
In reply to#178761
On 03/13/2017 07:12 AM, Dan Ritter wrote:
> ... we need image backups over the network to the server.

Clonezilla?


David

[toc] | [prev] | [next] | [standalone]

#178828

FromRichard Owlett <rowlett@cloud85.net>
Date2017-03-14 15:50 +0100
Message-ID<tkVVU-1uM-23@gated-at.bofh.it>
In reply to#178761
On 03/13/2017 09:12 AM, Dan Ritter wrote:
> On Sat, Mar 11, 2017 at 09:10:54AM -0600, Richard Owlett wrote:
>> I have one partition that might be called a "production" environment, i.e.
>> fairly stable and has the most valuable content.
>> A second partition hosts my experiments - I've a project to create an
>> optimal install. The third is the target of those experimental installs
>> whose content doesn't rate explicit backups. The scripts for creating those
>> installs being on the second partition.
>>
>> I've vague ideas of what backup pattern(s) I might follow.
>> I'm looking for reading materials that might trigger "I hadn't thought of
>> that" moments.
>
> A quick overview:
>
> Nobody wants backups. Everybody wants restores.
>
> The questions are:
>
> - what sort of disaster are you trying to recover from

Primarily operator error. This machine started out as a vehicle to 
experiment with Linux.  And latter as a test-bed for how *I* think a 
personal system should be configured. My old Windows box served for 
internet connectivity and playing Solitaire.

I've got a setup configured *my way* so the Windows box has been demoted 
to Solitaire - not having experimented with wine.

>
> - how often do you expect each to happen

More frequently than I'll admit.

>
> - how much time are you willing to take recovering

Time is not much of an issue as I'm retired and this is a hobby 
environment. But recently Pastor has asked be to administer the church's 
new website (hosted elsewhere) so there will be some files which it 
would be annoying to have to recreate.

>
> - how much are you willing to spend

Minimal. I'm intending to use a physically local hard drive as the 
primary backup medium. For church related files, likely will be flash 
drives stored at the church.

 From what I've been reading the local backup of my machine will likely 
be a weekly incremental backup. The backup for church will be file 
copies as it will also serve as a record for Pastor's use.

[toc] | [prev] | [next] | [standalone]

#178840

FromMerlin Büge <toni@bluenox07.de>
Date2017-03-14 19:50 +0100
Message-ID<tkZGa-4eI-5@gated-at.bofh.it>
In reply to#178680
On Sat, 11 Mar 2017 09:10:54 -0600
Richard Owlett <rowlett@cloud85.net> wrote:

> I've been good about telling others that backups are a good idea.
> Guess who hadn't and then crashed his system and spent hours putting 
> things back together ;<
> 
> In the past individual projects ended up on individual flash drives
> as I was frequently using different machines. I now have some
> reliable hardware and a large internal hard drive.
> 
> I have one partition that might be called a "production" environment, 
> i.e. fairly stable and has the most valuable content.
> A second partition hosts my experiments - I've a project to create an 
> optimal install. The third is the target of those experimental
> installs whose content doesn't rate explicit backups. The scripts for
> creating those installs being on the second partition.
> 
> I've vague ideas of what backup pattern(s) I might follow.
> I'm looking for reading materials that might trigger "I hadn't
> thought of that" moments.
> 
> Suggestions?

There is a nice overview about different backup software in the Arch
Wiki:

https://wiki.archlinux.org/index.php/Synchronization_and_backup_programs

Also an interesting read (about various pitfalls when backing up):

http://www.halfgaar.net/backing-up-unix


Regards,

Merlin



> TIA
> 
> 


-- 
Merlin Büge <toni@bluenox07.de>

[toc] | [prev] | [next] | [standalone]

#179009

FromDdB <debianlist@potentially-spam.de-bruyn.de>
Date2017-03-19 11:40 +0100
Message-ID<tmGpH-4hW-5@gated-at.bofh.it>
In reply to#178680

[Multipart message — attachments visible in raw view] — view raw

Am 11.03.2017 um 16:10 schrieb Richard Owlett:
> I've been good about telling others that backups are a good idea.
Hi,
i know i am late to the party. And without knowing any ready made
documentation, let me add a few things out of the top of my head:

 1. Backups saved my mental health on numerous occassions, even while
    still on Windows
 2. Although i came across tools to automate the process, i have never
    accepted having to trust anything to be - or not to be - in control
    of critical data, except for myself
 3. An understanding of how to set up a bootable system (be it from
    backup or new) seems crucial to me, as some understanding of grub +
    initramfs appears to be extremely useful
 4. Linux has every tool necessary on board, a first backup is easy to
    make using (f.i.) systemrescuecd
 5. And logging all the steps done, while doing them, is the first step
    to a scripted solution
 6. Such a log - turning into a script - paired with the corresponding
    restore - is evolving naturally with time and experience...

Here are some lessons, i learned while being involved with backing up:

  * Different kinds of data need to be considered, according to their
    turnaround and usefulness (a.k.a. backup cycle interval). I adhere
    to 3 types today: (os=critical=when needed: on occassion several
    times a day, regular cycle: almost everything else backing up every
    2 weeks currently, and dont care/throw away data, which doesnt stop
    me from backing it up, but more of the fire and forget kind)
  * always log the reason/state of the backup, similar to git, as that
    logfile turns into a valuable resource useable even after years
  * where to backup to? Initially, i was using the machine to backup for
    holding the backup, which is a really bad idea. Today, i am using a
    pluggable external device with several drives configured in a RAID
    setup.
  * In order to save space, a COW (copy-on-write) fs turns out to be
    mega useful. My recommendation being ZFS (at least on the backup
    devices), as that allows keeping several backups/restore points in
    one place using incremental backup.
    Example: OS (uses 4.5 GB) has 20 incremental backups, making up
    around 16.5 GB = 4.5 + 12 for snapshots in total.
  * for some strange reason, i switched to imaging lately (using
    zerofree + compression, loop mount + rsync), and the gain in
    restore-time is mind-blowing, as a simple dd was enough to restore
    one entire snap from a OS. BTW: in the meantime, i switched to using
    ZFS for the real data mountpoints as well
  * The bash scripts used for backup/restore evolved to 9+6K due to my
    habit to make lots of asserts/checks as a safety measure before
    proceeding.

That kind of approach may not be useful to everybody, as it is some
investment into one self's know-how. But you gain flexibility and
creative handling options, like just restoring the exact file(s) of even
just comparing the differences. And after having gone through different
real world scenarios (hardware failures of different kinds) i know how
safe i am!

But for those only interested in a ready made solution: You are going to
run into a major difficulty sooner or later, that hasnt been foreseen.
Hopefully, you'll have 3 Versions ready (original, backup, next-to-last
backup) and find some admin worth trusting and able to handle your
situation manually. :-)

Those are my 2 cents.
DdB (reading only the lists digest, thus not replying directly)

[toc] | [prev] | [standalone]

Page 3 of 3 — ← Prev page 1 2 [3]

Back to top | Article view | linux.debian.user

csiph-web