Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #180808

Re: Live Fille System Backup

Show all headers | View raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, May 07, 2017 at 10:53:51AM +1200, Ben Caradoc-Davies wrote:

[also a reply to Henrique, elsewhere in this thread]

> If a file is updated while it is being copied, it may contain only
> half a change set and be in an internally inconsistent state,
> perhaps making it unusable as a backup. Writes are typically not
> atomic. The same problem applies to collections of files that
> reference each other.
> 
> Kind regards,

Ben, Henrique -- no questions. This is what I subsumed under
"skew": application state may be dispersed across different
places in a file, across different files or even partly not
in files at all (e.g. in RAM: imagine a BTree with just parts
of its pointer structure not yet committed to disk).

Of course you can't ever win unless you collaborate with the
application in those cases (even with magic file systems like
ZFS or btrfs).

Then there is this subtle "file data" and "file metadata"
thing, which is an issue even with carefully designed applications
and file systems. It's even difficult to reach a consensus on
what is "right", remember the ext3/ext4 data loss episode[1]?

This is where shapshotting magic, be it built-in (zfs, btrfs)
or bolted-on (overlayfs, lvm) might help a bit: freeze a snapshot,
back up that (in the first case, the file systems provide a native
way to do that, in the second case, rsync is a pretty viable
way of doing things). I said "might help a bit" because the
ultimate consistency criterion is the application! A consistent
file system view might just be this truncated-to-zero file,
only the application "knows" at that point (e.g. by keeping
its data in an already unlinked file which is still open,
or somewhere in RAM, or...).

So your choices are

 - for the applications you really care about, look into
   what they are doing. Grown up apps will support you in
   that (I gave the PostgreSQL example above). Typically
   you can wrap the backup process in guards like ("keep
   your on-disk state consistent"[1]..."now you can relax").
   Note that to avoid races this structure is more or less
   necessary. The only real difference to the "magic
   snapshot" thing is that the latter happens very quickly.

 - for all the others... just relax.

Otherwise, "on line" backup is simply not an option.

cheers

[1] That doesn't mean necessarily frozen. PostgreSQL, for example,
   continues writing to the WAL, it just eats through its storage
   at a higher pace.

- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlkO7SgACgkQBcgs9XrR2kYf9ACeM2njgrSttOUPRk4D6fJqJtjQ
qmkAn38VbkKiOlADe+33teN8uzcbLa2C
=uKNk
-----END PGP SIGNATURE-----

Back to linux.debian.user | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-05 01:20 +0200
  Re: Live Fille System Backup Anders Andersson <pipatron@gmail.com> - 2017-05-05 03:50 +0200
    Re: Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-05 06:50 +0200
      Re: Live Fille System Backup Dejan Jocic <jodejka@gmail.com> - 2017-05-05 09:00 +0200
        Re: Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-05 22:40 +0200
          Re: Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-05 23:20 +0200
            Re: Live Fille System Backup Henrique de Moraes Holschuh <hmh@debian.org> - 2017-05-06 00:00 +0200
              Re: Live Fille System Backup conover@rahul.net (John Conover) - 2017-05-06 01:30 +0200
                Re: Live Fille System Backup Henrique de Moraes Holschuh <hmh@debian.org> - 2017-05-06 02:30 +0200
          Re: Live Fille System Backup Henrique de Moraes Holschuh <hmh@debian.org> - 2017-05-06 00:00 +0200
          Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-06 08:00 +0200
            Re: Live Fille System Backup Dan Ritter <dsr@randomstring.org> - 2017-05-06 12:50 +0200
              Re: Live Fille System Backup tomas@tuxteam.de - 2017-05-06 16:00 +0200
                Re: Live Fille System Backup Henrique de Moraes Holschuh <hmh@debian.org> - 2017-05-06 19:20 +0200
                Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-06 21:50 +0200
                Re: Live Fille System Backup Henrique de Moraes Holschuh <hmh@debian.org> - 2017-05-06 23:10 +0200
                Re: Live Fille System Backup Ben Caradoc-Davies <ben@transient.nz> - 2017-05-07 01:00 +0200
                Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-07 12:00 +0200
                Re: Live Fille System Backup rhkramer@gmail.com - 2017-05-07 14:50 +0200
                Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-07 17:00 +0200
                Re: Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-09 00:50 +0200
                Re: Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-09 01:10 +0200
                Re: Live Fille System Backup tomas@tuxteam.de - 2017-05-09 10:00 +0200
                Re: Live Fille System Backup Greg Wooledge <wooledg@eeg.ccf.org> - 2017-05-09 14:10 +0200
                Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-09 14:30 +0200
                Re: Live Fille System Backup Curt <curty@free.fr> - 2017-05-09 15:20 +0200
                Re: Live Fille System Backup Curt <curty@free.fr> - 2017-05-09 14:10 +0200
                Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-09 14:30 +0200
  Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-05 09:20 +0200
  Re: Live Fille System Backup Bob Weber <bobrweber@gmail.com> - 2017-05-06 01:00 +0200
  Re: Live Fille System Backup Stefan Monnier <monnier@iro.umontreal.ca> - 2017-05-12 15:00 +0200
    Re: Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-25 05:40 +0200

csiph-web