Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #180704 > unrolled thread

Live Fille System Backup

Back to article view | Back to linux.debian.user

Contents

  Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-05 01:20 +0200
    Re: Live Fille System Backup Anders Andersson <pipatron@gmail.com> - 2017-05-05 03:50 +0200
      Re: Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-05 06:50 +0200
        Re: Live Fille System Backup Dejan Jocic <jodejka@gmail.com> - 2017-05-05 09:00 +0200
          Re: Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-05 22:40 +0200
            Re: Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-05 23:20 +0200
              Re: Live Fille System Backup Henrique de Moraes Holschuh <hmh@debian.org> - 2017-05-06 00:00 +0200
                Re: Live Fille System Backup conover@rahul.net (John Conover) - 2017-05-06 01:30 +0200
                  Re: Live Fille System Backup Henrique de Moraes Holschuh <hmh@debian.org> - 2017-05-06 02:30 +0200
            Re: Live Fille System Backup Henrique de Moraes Holschuh <hmh@debian.org> - 2017-05-06 00:00 +0200
            Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-06 08:00 +0200
              Re: Live Fille System Backup Dan Ritter <dsr@randomstring.org> - 2017-05-06 12:50 +0200
                Re: Live Fille System Backup tomas@tuxteam.de - 2017-05-06 16:00 +0200
                  Re: Live Fille System Backup Henrique de Moraes Holschuh <hmh@debian.org> - 2017-05-06 19:20 +0200
                    Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-06 21:50 +0200
                      Re: Live Fille System Backup Henrique de Moraes Holschuh <hmh@debian.org> - 2017-05-06 23:10 +0200
                  Re: Live Fille System Backup Ben Caradoc-Davies <ben@transient.nz> - 2017-05-07 01:00 +0200
                    Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-07 12:00 +0200
                      Re: Live Fille System Backup rhkramer@gmail.com - 2017-05-07 14:50 +0200
                        Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-07 17:00 +0200
                          Re: Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-09 00:50 +0200
                            Re: Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-09 01:10 +0200
                              Re: Live Fille System Backup tomas@tuxteam.de - 2017-05-09 10:00 +0200
                                Re: Live Fille System Backup Greg Wooledge <wooledg@eeg.ccf.org> - 2017-05-09 14:10 +0200
                                  Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-09 14:30 +0200
                                  Re: Live Fille System Backup Curt <curty@free.fr> - 2017-05-09 15:20 +0200
                                Re: Live Fille System Backup Curt <curty@free.fr> - 2017-05-09 14:10 +0200
                                  Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-09 14:30 +0200
    Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-05 09:20 +0200
    Re: Live Fille System Backup Bob Weber <bobrweber@gmail.com> - 2017-05-06 01:00 +0200
    Re: Live Fille System Backup Stefan Monnier <monnier@iro.umontreal.ca> - 2017-05-12 15:00 +0200
      Re: Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-25 05:40 +0200

Page 2 of 2 — ← Prev page 1 [2]

#180864

[Multipart message — attachments visible in raw view] — view raw

We have to remember that things are actually worse at the application
consistency level.  An application may think that it has committed its
write, but the file system has not written all the bytes to the disk yet.

File system crash consistency is a sibling of the backup topic.  Here is
relevant article I run into a some time ago
https://blog.acolyer.org/2017/03/15/application-crash-consistency-and-performance-with-ccfs/

According to link above Ext4 FS suffers from these inconsistencies.

On Sun, May 7, 2017 at 7:56 AM, <tomas@tuxteam.de> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Sun, May 07, 2017 at 08:42:36AM -0400, rhkramer@gmail.com wrote:
> > On Sunday, May 07, 2017 05:47:20 AM tomas@tuxteam.de wrote:
> >
> > <good stuff elided>
> >
> > > Otherwise, "on line" backup is simply not an option.
>
> [...]
>
> > I guess the way that comes to mind, maybe somewhat implied by what you've
> > written (or at least inferred by me ;-) would be to take down one or
> more of
> > the servers at a time, make static backups of those, then restore those
> to
> > service and repeat until all have been backed up, very likely repeating
> that
> > continuously.
>
> "Taking down" as in "stopping all relevant applications" (shutting down
> the whole thing would be a superset of that, of course ;)
>
> [...]
>
> > Oh, ok, I guess (well, I know RAID isn't really for backup, it's for
> uptime as
> > is often stated on this and other lists), but something like RAID where
> > relevant data is written to more than one disk and / or storage "farm",
> one
> > being the "in service" device (to serve data to the app in "real time")
> and
> > one or more just recording data as backup.
>
> This is a variant on the snapshot pattern. You take a snapshot of the
> file system and back up that. You are guaranteed that the *file system*
> backup is consistent (well, modulo bugs and glitches), but since the
> applications don't have a clue what's going on, application state
> might still be inconsistent, unless they go out of their way to keep
> a consistent state on-disk all of the time (in the above scenario
> of shutdown, we assumed that the application leaves a consistent
> state after being shut down, which is a reasonable assumption, I'd
> say).
>
> That said, and as you can see in the PostgreSQL example, with a little
> help of your application, the snapshot thing works (in the case of
> PostgreSQL things are much more relaxed and you can usually go with
> plain rsync, for example, without resorting to snapshot).
>
> All in all I'm pretty happy with a plain, straight rsync-style backup
> for my workstation. The probability of a busted backup is so low that
> I fear more for my backup medium going sour.
>
> On a high-churn machine with very valuable data things might look
> differently, though...
>
> regards
> - -- tomás
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
>
> iEYEARECAAYFAlkPNaIACgkQBcgs9XrR2kYGWgCfcR14ZU9wA9kKDampxUY7MCHb
> IjUAnj0zjx2U5yFy7TIIO/rP5UO7E6QI
> =bD6R
> -----END PGP SIGNATURE-----
>
>

[toc] | [prev] | [next] | [standalone]

#180865

[Multipart message — attachments visible in raw view] — view raw

And if you think application consistency is nothing to worry about know
that Git on Ext4 has failed to recover.

My current conclusion:

1. If I want application consistency I have to take down any application
that has an internal database or backup those applications prior to FS
backup.
2. rsnapshot <http://rsnapshot.org/>is a good candidate.  It matches the
way I want to use backup and it is abased on rsync that I am familiar with

I'd like to consider using CCFS, but I have no instructions for that.

Linux FS snapshot seems to be of a temporary nature, because it will grow
over time (it is a diff between snapshot time and current time).  LVM is a
learning curve I have managed to escape so far:

The generic way to make snapshots under Linux is at the level of the
> storage volume. Your filesystem must be on an LVM logical volume
> <https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)>, which is
> Linux's own partition system, as opposed to directly on a platform-native
> disk partition.


So, my existing installations probably avoids LVM and snapshot is not an
option.



On Mon, May 8, 2017 at 3:46 PM, Sergei G <sergeig.public@gmail.com> wrote:

> We have to remember that things are actually worse at the application
> consistency level.  An application may think that it has committed its
> write, but the file system has not written all the bytes to the disk yet.
>
> File system crash consistency is a sibling of the backup topic.  Here is
> relevant article I run into a some time ago https://blog.acolyer.org/2017/
> 03/15/application-crash-consistency-and-performance-with-ccfs/
>
> According to link above Ext4 FS suffers from these inconsistencies.
>
> On Sun, May 7, 2017 at 7:56 AM, <tomas@tuxteam.de> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On Sun, May 07, 2017 at 08:42:36AM -0400, rhkramer@gmail.com wrote:
>> > On Sunday, May 07, 2017 05:47:20 AM tomas@tuxteam.de wrote:
>> >
>> > <good stuff elided>
>> >
>> > > Otherwise, "on line" backup is simply not an option.
>>
>> [...]
>>
>> > I guess the way that comes to mind, maybe somewhat implied by what
>> you've
>> > written (or at least inferred by me ;-) would be to take down one or
>> more of
>> > the servers at a time, make static backups of those, then restore those
>> to
>> > service and repeat until all have been backed up, very likely repeating
>> that
>> > continuously.
>>
>> "Taking down" as in "stopping all relevant applications" (shutting down
>> the whole thing would be a superset of that, of course ;)
>>
>> [...]
>>
>> > Oh, ok, I guess (well, I know RAID isn't really for backup, it's for
>> uptime as
>> > is often stated on this and other lists), but something like RAID where
>> > relevant data is written to more than one disk and / or storage "farm",
>> one
>> > being the "in service" device (to serve data to the app in "real time")
>> and
>> > one or more just recording data as backup.
>>
>> This is a variant on the snapshot pattern. You take a snapshot of the
>> file system and back up that. You are guaranteed that the *file system*
>> backup is consistent (well, modulo bugs and glitches), but since the
>> applications don't have a clue what's going on, application state
>> might still be inconsistent, unless they go out of their way to keep
>> a consistent state on-disk all of the time (in the above scenario
>> of shutdown, we assumed that the application leaves a consistent
>> state after being shut down, which is a reasonable assumption, I'd
>> say).
>>
>> That said, and as you can see in the PostgreSQL example, with a little
>> help of your application, the snapshot thing works (in the case of
>> PostgreSQL things are much more relaxed and you can usually go with
>> plain rsync, for example, without resorting to snapshot).
>>
>> All in all I'm pretty happy with a plain, straight rsync-style backup
>> for my workstation. The probability of a busted backup is so low that
>> I fear more for my backup medium going sour.
>>
>> On a high-churn machine with very valuable data things might look
>> differently, though...
>>
>> regards
>> - -- tomás
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.12 (GNU/Linux)
>>
>> iEYEARECAAYFAlkPNaIACgkQBcgs9XrR2kYGWgCfcR14ZU9wA9kKDampxUY7MCHb
>> IjUAnj0zjx2U5yFy7TIIO/rP5UO7E6QI
>> =bD6R
>> -----END PGP SIGNATURE-----
>>
>>
>

[toc] | [prev] | [next] | [standalone]

#180872

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, May 08, 2017 at 04:03:54PM -0700, Sergei G wrote:
> And if you think application consistency is nothing to worry about know
> that Git on Ext4 has failed to recover.
> 
> My current conclusion:
> 
> 1. If I want application consistency I have to take down any application
> that has an internal database or backup those applications prior to FS
> backup.

That depends. There are applications designed with (on-disk) consistency
in mind (I mention PostgreSQL, I start sounding like a broken record).

But whether you have to "take down" the application or just "alert" it
for the time the snapshot takes, without its cooperation you either risk
inconsistent state or bad performance (there are cases where apps chose
number two, because they aren't designed for high throughput anyway).

The gist is:

  (a) file system magic can't "do" it alone
  (b) most of the time, a naive "snapshot" with rsync will do
  (b') you alone can decide, based on your apps & requirements,
      whether the above "most of the time" works for you

With snapshotting file systems (be it native or assisted) you get just
faster snapshots, i.e. you mitigate the problem. If you want to go all
the way, you have to get the app's cooperation, either voluntary (it's
designed for that) or forced (shutdown)

There's no magic.

cheers
- -- tomás
   "all generalizations suck"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlkRdlsACgkQBcgs9XrR2kZKsACePO1j2O2SFF8Stj9rMdGOSqNF
SaMAnRLQhrrTRRObZLJD1H+C3HtQwsRy
=BgPQ
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#180873

On Tue, May 09, 2017 at 12:03:14PM +0000, Curt wrote:
> Because someone mentioned Btrfs and its backup "magic," I'm looking at
> the wiki, which says (amongst other things):
> 
> https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
> 
>  Instant, Atomic COW Snapshots
>   Since the snapshots are atomic, when a snapshot is restored it appears to
>   applications as if a power loss had occurred (and the filesystem has gone back
>   to an earlier state). Thus it is possible to backup databases without stopping
>   them beforehand. 

So, basically the same as LVM snapshots, which are independent of the
type of file system that sits on the LVM logical volume.

But btrfs is an edge with more blood on it, so people are naturally
drawn to it.  It's also more complicated.  So it's *got* to be better!!!

> I don't know what an atomic cow is (maybe some poor bovine living near
> Chernobyl) but it sounds interesting.

C.O.W. in this context is usually "copy on write".

Atomic means that it's like a database transaction: the entire operation
either succeeds, or fails.  There's no half-way state.

> I wonder if Btrfs has any notable drawbacks as a file system.

Some of the blood on btrfs is starting to dry, so the blood-seekers
are all flocking to ZFS now.  It's got a whole new TIER of buzzwords
and complexity!  It's GOT to be the best!!!

[toc] | [prev] | [next] | [standalone]

#180876

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, May 09, 2017 at 08:08:48AM -0400, Greg Wooledge wrote:

[...]

> So, basically the same as LVM snapshots, which are independent of the
> type of file system that sits on the LVM logical volume.

To be fair, btrfs knows about the file system, whereas LVM does COW
on the underlying "dumb" block device. Thus btrfs is capable of
squeezing quite a bit out in terms of better performance/lower disk
usage.

For some workloads it matters, for others not (the use case I posited,
freezing a snapshot while rsyncing, might tend to fall on the "not"
side *if* you are a bit careful and *if* your write load isn't insane).

> But btrfs is an edge with more blood on it, so people are naturally
> drawn to it.  It's also more complicated.  So it's *got* to be better!!!
> 
> > I don't know what an atomic cow is (maybe some poor bovine living near
> > Chernobyl) but it sounds interesting.
> 
> C.O.W. in this context is usually "copy on write".
> 
> Atomic means that it's like a database transaction: the entire operation
> either succeeds, or fails.  There's no half-way state.
> 
> > I wonder if Btrfs has any notable drawbacks as a file system.
> 
> Some of the blood on btrfs is starting to dry, so the blood-seekers
> are all flocking to ZFS now.  It's got a whole new TIER of buzzwords
> and complexity!  It's GOT to be the best!!!

:-)

ISTR Android has btrfs as default these days? (careful, I didn't check,
and my brain has one fake news source embedded somewhere I couldn't
yet stomp out. Might be wrong).

cheers
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlkRtdAACgkQBcgs9XrR2kZ7fgCcD/B1+IavQMGeEJ51Y4L6uoPe
WaMAnR/seOj+evs/hk7Wb3bJ/TtYBUP2
=3Jdh
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#180879

On 2017-05-09, Greg Wooledge <wooledg@eeg.ccf.org> wrote:
> On Tue, May 09, 2017 at 12:03:14PM +0000, Curt wrote:
>> Because someone mentioned Btrfs and its backup "magic," I'm looking at
>> the wiki, which says (amongst other things):
>> 
>> https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
>> 
>>  Instant, Atomic COW Snapshots
>>   Since the snapshots are atomic, when a snapshot is restored it appears to
>>   applications as if a power loss had occurred (and the filesystem has gone back
>>   to an earlier state). Thus it is possible to backup databases without stopping
>>   them beforehand. 
>
> So, basically the same as LVM snapshots, which are independent of the
> type of file system that sits on the LVM logical volume.
>
> But btrfs is an edge with more blood on it, so people are naturally
> drawn to it.  It's also more complicated.  So it's *got* to be better!!!
>

Put me down for uncomplicated. I'll probably just stick with rsyncing my
home directory (which is all I'm interested in backing up) to external
usb-attached volumes and be done with it.

[toc] | [prev] | [next] | [standalone]

#180874

On 2017-05-09, tomas@tuxteam.de <tomas@tuxteam.de> wrote:
>
> On Mon, May 08, 2017 at 04:03:54PM -0700, Sergei G wrote:
>> And if you think application consistency is nothing to worry about know
>> that Git on Ext4 has failed to recover.
>> 
>> My current conclusion:
>> 
>> 1. If I want application consistency I have to take down any application
>> that has an internal database or backup those applications prior to FS
>> backup.
>
> That depends. There are applications designed with (on-disk) consistency
> in mind (I mention PostgreSQL, I start sounding like a broken record).
>
> But whether you have to "take down" the application or just "alert" it
> for the time the snapshot takes, without its cooperation you either risk
> inconsistent state or bad performance (there are cases where apps chose
> number two, because they aren't designed for high throughput anyway).
>
> The gist is:
>
>   (a) file system magic can't "do" it alone
>   (b) most of the time, a naive "snapshot" with rsync will do
>   (b') you alone can decide, based on your apps & requirements,
>       whether the above "most of the time" works for you
>
> With snapshotting file systems (be it native or assisted) you get just
> faster snapshots, i.e. you mitigate the problem. If you want to go all
> the way, you have to get the app's cooperation, either voluntary (it's
> designed for that) or forced (shutdown)

Because someone mentioned Btrfs and its backup "magic," I'm looking at
the wiki, which says (amongst other things):

https://btrfs.wiki.kernel.org/index.php/Incremental_Backup

 Instant, Atomic COW Snapshots
  Since the snapshots are atomic, when a snapshot is restored it appears to
  applications as if a power loss had occurred (and the filesystem has gone back
  to an earlier state). Thus it is possible to backup databases without stopping
  them beforehand. 

I don't know what an atomic cow is (maybe some poor bovine living near
Chernobyl) but it sounds interesting.

I wonder if Btrfs has any notable drawbacks as a file system.

BTW, I learned (by way of your least favorite search engine) that the inventor
of the Monoyer chart (Ferdinand Monoyer) embedded his name (and occupation--DM,
docteur en médecine) in the chart (beginning on the penultimate line, the last
and first letters read vertically upwards).

These French are clever devils.

> There's no magic.
>
> cheers
> - -- tomás
>    "all generalizations suck"
>
>


-- 
"It might be a vision--of a shell, of a wheelbarrow, of a fairy kingdom on the
far side of the hedge; or it might be the glory of speed; no one knew." --Mrs.
Ramsay, speculating on why her little daughter might be dashing about, in "To
the Lighthouse," by Virginia Woolf.

[toc] | [prev] | [next] | [standalone]

#180875

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, May 09, 2017 at 12:03:14PM +0000, Curt wrote:

[...]

> Because someone mentioned Btrfs and its backup "magic," I'm looking at
> the wiki, which says (amongst other things):
> 
> https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
> 
>  Instant, Atomic COW Snapshots
>   Since the snapshots are atomic, when a snapshot is restored it appears to
>   applications as if a power loss had occurred (and the filesystem has gone back
>   to an earlier state). Thus it is possible to backup databases without stopping
>   them beforehand. 
> 
> I don't know what an atomic cow is (maybe some poor bovine living near
> Chernobyl) but it sounds interesting.

COW stands for "copy-on-write". The idea is that you don't overwrite whatever
you are trying to change, but make a copy first, munge that and when you are
done, you swap old and new.

If that "whatever" happens to be a virtual memory page, there's support
from the operating system kernel, supported by the virtual memory hardware
(the trick is typically that the page is, from the outset, marked as
"read only", and when something tries to write it, a fault is triggered;
this is all typically hardware supported; then the fault handler is
called, which duplicates the page and routes the writes to the fresh
copy).

The "atomic" part is that things are set up carefully in a way that
at the end, regardless of when you shot the thing in the head, you
see either the whole change or nothing. Standard fare for file systems
and databases these days.

(Far from trivial, though. One of the standard tricks databases, ZFS
and btrfs use, afaik, is they have their backbone as a B-tree. Whenever
something has to be updated, it typically runs from one node (perhaps
a leaf) all the way back to the root. This whole path is constructed
by copying all the touched nodes (when the thing dies here nothing
has changed!) and at the end, replacing the root node in one (hopefully
atomic) operation. After that, the change is permanent.

COW has to be one of the nice acronyms, though.

> I wonder if Btrfs has any notable drawbacks as a file system.

It's pretty young, as file systems go. And ext4 is so rock-solid and
trusted that btrfs doesn't get the broad exposure it'd need for
trust to be built. Sysadmin is a conservative and fainthearted
bunch :-)

> BTW, I learned (by way of your least favorite search engine) that the inventor
> of the Monoyer chart (Ferdinand Monoyer) embedded his name (and occupation--DM,
> docteur en médecine) in the chart (beginning on the penultimate line, the last
> and first letters read vertically upwards).
> 
> These French are clever devils.

They are. And they are friendly and know what a good meal is.
Looking forward to my next vacation there :-D

regards
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlkRtHMACgkQBcgs9XrR2ka7GgCdE3ZRJdV07rc1hMKTZG6eUJoP
4oQAoIHoS9LdWsTTZtDJuwOFsKvZJGas
=Z9QB
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#180710

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, May 04, 2017 at 04:17:46PM -0700, Sergei G wrote:
> I am running Raspberry PI and I would like to dump full file system without
> shutting down the system.  One machine runs nginx and another runs
> PostgreSQL.  I have had a good success with FreeBSD and dump software,
> because it is part of the OS and core team maintains it.

Rsync. Hands down.

That said...

there are some thigs you might want to take into account.

 - A solution like rsync alone won't ever solve the "skew" problem
   (i.e. the file system moves while the backup is being taken).
   Depending on how sensitive your "world" is to that, this might
   be a problem or not. "Typically" (famous last words, I know)
   it ain't for me, especially if I take care of taking a "second
   rsync", which will be very fast. If that isn't enough, you might
   like to "snapshot" with LVM or overlayfs and back-up your
   "frozen" (and thus consistent) snapshot, while life throbs on
   on your "real" file system branch.

 - PostgreSQL. Conventional wisdom says you shouldn't back up a
   running database. Nevertheless, you can do that with PostgreSQL,
   if you take care of orchestrating pg_start_backup, pg_stop_backup
   and the right copying of WALs [1]. This sounds complicated, is
   not: your (homegrown or acquired) backup scripts just have to
   provide a "before" and an "after" hook (if they don't, claim your
   money back :) where to enlist the scripts to steer your database(s).

   Having an old-fashioned pg_dump around which isn't too old won't
   hurt, especially if you want to be able to restore to a significantly
   different architecture/ PostgreSQL version.

Things rsync won't be very good at:

  - huge backups (several terabytes or more; millions of files)
    My current backup is around 48 GB and 650K files (not much),
    and the rsync approach (organic, homegrown :) doesn't break
    a sweat, with fairly modest hardware. YMMV. I've seen a photo
    archive (~20TB) where the buildup of the file list would have
    taken rsync short of two days -- that won't work, obviously
    (to be fair, rsync has gotten better in this department as
    of late, and the shop in question has a faible for "seasoned"
    software versions).

 - whatever I forgot: please chime in!

regards

[1] https://www.postgresql.org/docs/current/static/continuous-archiving.html#BACKUP-LOWLEVEL-BASE-BACKUP
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlkMJqgACgkQBcgs9XrR2kZaWQCfY7/XyU5LJJR5Oz3h4W6afU0u
kLUAn0H/ewNQMYkEv/LbTjwwtxOnzDxo
=H0lN
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#180749

On 5/4/17 7:17 PM, Sergei G wrote:
> I am running Raspberry PI and I would like to dump full file system without
> shutting down the system.  One machine runs nginx and another runs
> PostgreSQL.  I have had a good success with FreeBSD and dump software, because
> it is part of the OS and core team maintains it.  
>
> However, dump utility is no longer maintained by the original developer and is
> effectively getting too old.  I am no longer sure what is a dump-like solution
> that's easy to work with at a file system level.
>
> I would like a backup tool that does not bring a million dependencies with MBs
> of files.  Something that works on server without X Windows and can send
> backup to an externally attached USB drive.  Nothing fancy.   No network
> infrastructure.  Incremental backups would be greatly appreciated.  Ability to
> pipe to a compression program is a plus, just like I did with dump.
>
> I'd like to be able to apply a similar solution on actual Debian and Ubuntu
> VMs.  When I go to Ubuntu I might have to deal with newer ext file systems. 
> Not sure what is supported there.
>
> If there is no good live backup solution, I am willing to take the system down
> and back it up using another system.  It is not ideal, but it would work.
>
> Any advice would be greatly appreciated.
>
> P.S. All mentioned systems are Debian based and I feel that it is appropriate
> to ask on Debian user list.
>
>
> Thank you
I use rsnapshot to live backup my root partition (home is mounted on another
partition) to a directory under home.  rsnapshot uses rsync with the appropriate
options (I do add some extras) to do a backup.  I do this before I do an "apt
upgrade"  so that if things don't work well after the upgrade I can restore what
was there before.  I have used this "backup" several times to take my system
back to before the upgrade and have not had any problems.  I am not backing up
databases so those remarks here about that sort of backup apply.  To restore I
boot a copy of  SystemRescueCd, mount the source and destination partitions and
run a script that runs rsync with similar options to restore the root back to a
working state.  The rsnapshot backup is an exact copy (uncompressed) of the root
so you can go get a single file easily if you need to.

I also have several VM's of different Debian installs (unstable and testing)
that I upgrade every few days to follow any problems that arise but since the
VM's can't duplicate my main system's xorg setup they don't catch everything.  I
use rsnapshot on these VM's also and have restored the rsnapshot backup several
times.  I have tried btrfs on these vm's but I always ended up running out of
free space during an upgrade so I would have to delete some older snapshots to
free up some space and run the upgrade again.  These vm's are running on 20 to
30 GB virtual hard drives so there is not a lot of free space for the btrfs
snapshots.

Grub can also handle booting an iso of SystemRescueCd so I don't even have to
find a cd with SystemRescueCd on it.  Just select the grub menu option for
SystemRescueCd.

For a daily backup I use backuppc to backup all my local systems and a remote vm
at digital ocean.  These backups are all on live systems and would have the
problems with live databases as described by others.  You just need an ssh login
using rsa key pairs so you don't have to supply a password to backuppc. 
Postgresql has ways to backup databases and is made very easy by using a program
like "pgadmin III" (in Debian as pgadmin3/testing,unstable,now 1.22.2-1 amd64). 
You might have to stop database updates while you backup though.  Individual
tables or the whole database can be backuped.

...bob

[toc] | [prev] | [next] | [standalone]

#180952

> I would like a backup tool that does not bring a million dependencies with
> MBs of files.  Something that works on server without X Windows and can
> send backup to an externally attached USB drive.  Nothing fancy.   No
> network infrastructure.  Incremental backups would be greatly appreciated.
> Ability to pipe to a compression program is a plus, just like I did with
> dump.

I used to use rsync for that kind of job, but nowadays I tend to prefer
`bup` because it handles incrementals really well (especially in terms
of compression).  It's in Debian stable or you can find it at
https://github.com/bup/bup.


        Stefan


PS: Of course, you'll still want to take a snapshot (e.g. with LVM) and
    back that up rather than the live filesystem.

[toc] | [prev] | [next] | [standalone]

#181362

[Multipart message — attachments visible in raw view] — view raw

thank you for mentioning bup.  That's a great option I am considering.

On Fri, May 12, 2017 at 5:52 AM, Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> > I would like a backup tool that does not bring a million dependencies
> with
> > MBs of files.  Something that works on server without X Windows and can
> > send backup to an externally attached USB drive.  Nothing fancy.   No
> > network infrastructure.  Incremental backups would be greatly
> appreciated.
> > Ability to pipe to a compression program is a plus, just like I did with
> > dump.
>
> I used to use rsync for that kind of job, but nowadays I tend to prefer
> `bup` because it handles incrementals really well (especially in terms
> of compression).  It's in Debian stable or you can find it at
> https://github.com/bup/bup.
>
>
>         Stefan
>
>
> PS: Of course, you'll still want to take a snapshot (e.g. with LVM) and
>     back that up rather than the live filesystem.
>
>

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.debian.user

csiph-web