Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #180704 > unrolled thread

Live Fille System Backup

Back to article view | Back to linux.debian.user

Contents

  Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-05 01:20 +0200
    Re: Live Fille System Backup Anders Andersson <pipatron@gmail.com> - 2017-05-05 03:50 +0200
      Re: Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-05 06:50 +0200
        Re: Live Fille System Backup Dejan Jocic <jodejka@gmail.com> - 2017-05-05 09:00 +0200
          Re: Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-05 22:40 +0200
            Re: Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-05 23:20 +0200
              Re: Live Fille System Backup Henrique de Moraes Holschuh <hmh@debian.org> - 2017-05-06 00:00 +0200
                Re: Live Fille System Backup conover@rahul.net (John Conover) - 2017-05-06 01:30 +0200
                  Re: Live Fille System Backup Henrique de Moraes Holschuh <hmh@debian.org> - 2017-05-06 02:30 +0200
            Re: Live Fille System Backup Henrique de Moraes Holschuh <hmh@debian.org> - 2017-05-06 00:00 +0200
            Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-06 08:00 +0200
              Re: Live Fille System Backup Dan Ritter <dsr@randomstring.org> - 2017-05-06 12:50 +0200
                Re: Live Fille System Backup tomas@tuxteam.de - 2017-05-06 16:00 +0200
                  Re: Live Fille System Backup Henrique de Moraes Holschuh <hmh@debian.org> - 2017-05-06 19:20 +0200
                    Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-06 21:50 +0200
                      Re: Live Fille System Backup Henrique de Moraes Holschuh <hmh@debian.org> - 2017-05-06 23:10 +0200
                  Re: Live Fille System Backup Ben Caradoc-Davies <ben@transient.nz> - 2017-05-07 01:00 +0200
                    Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-07 12:00 +0200
                      Re: Live Fille System Backup rhkramer@gmail.com - 2017-05-07 14:50 +0200
                        Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-07 17:00 +0200
                          Re: Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-09 00:50 +0200
                            Re: Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-09 01:10 +0200
                              Re: Live Fille System Backup tomas@tuxteam.de - 2017-05-09 10:00 +0200
                                Re: Live Fille System Backup Greg Wooledge <wooledg@eeg.ccf.org> - 2017-05-09 14:10 +0200
                                  Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-09 14:30 +0200
                                  Re: Live Fille System Backup Curt <curty@free.fr> - 2017-05-09 15:20 +0200
                                Re: Live Fille System Backup Curt <curty@free.fr> - 2017-05-09 14:10 +0200
                                  Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-09 14:30 +0200
    Re: Live Fille System Backup <tomas@tuxteam.de> - 2017-05-05 09:20 +0200
    Re: Live Fille System Backup Bob Weber <bobrweber@gmail.com> - 2017-05-06 01:00 +0200
    Re: Live Fille System Backup Stefan Monnier <monnier@iro.umontreal.ca> - 2017-05-12 15:00 +0200
      Re: Live Fille System Backup Sergei G <sergeig.public@gmail.com> - 2017-05-25 05:40 +0200

Page 1 of 2  [1] 2  Next page →

#180704 — Live Fille System Backup

[Multipart message — attachments visible in raw view] — view raw

I am running Raspberry PI and I would like to dump full file system without
shutting down the system.  One machine runs nginx and another runs
PostgreSQL.  I have had a good success with FreeBSD and dump software,
because it is part of the OS and core team maintains it.

However, dump utility is no longer maintained by the original developer and
is effectively getting too old.  I am no longer sure what is a dump-like
solution that's easy to work with at a file system level.

I would like a backup tool that does not bring a million dependencies with
MBs of files.  Something that works on server without X Windows and can
send backup to an externally attached USB drive.  Nothing fancy.   No
network infrastructure.  Incremental backups would be greatly appreciated.
Ability to pipe to a compression program is a plus, just like I did with
dump.

I'd like to be able to apply a similar solution on actual Debian and Ubuntu
VMs.  When I go to Ubuntu I might have to deal with newer ext file
systems.  Not sure what is supported there.

If there is no good live backup solution, I am willing to take the system
down and back it up using another system.  It is not ideal, but it would
work.

Any advice would be greatly appreciated.

P.S. All mentioned systems are Debian based and I feel that it is
appropriate to ask on Debian user list.


Thank you

[toc] | [next] | [standalone]

#180706

On Fri, May 5, 2017 at 1:17 AM, Sergei G <sergeig.public@gmail.com> wrote:
> I would like a backup tool that does not bring a million dependencies with
> MBs of files.  Something that works on server without X Windows and can send
> backup to an externally attached USB drive.  Nothing fancy.   No network
> infrastructure.  Incremental backups would be greatly appreciated.  Ability
> to pipe to a compression program is a plus, just like I did with dump.
> [...]
> Any advice would be greatly appreciated.

You won't like my solution, and it doesn't work with your current
setup because it requires a specific file system: btrfs[1]. I'm
posting it here for two reasons:
 1) You might consider using btrfs on new installs
 2) Someone else may search and find the thread

After switching to btrfs I can now take instant snapshots of selected
filesystems, transfer these to remote servers for backup, and most
important: btrfs can track the *exact difference* between two
snapshots taken over time, and only transfer the changes. All of this
is very quick, because the filesystem already knows exactly what
changed: Permission bits, file sizes, deleted files, changed data,
whatever, all is already kept in a log. It also means that nothing
will be missed, for example ACL bits etc.

The delta is just a simple stream of data that can be compressed if
necessary. Typically it is transmitted to a backup server where it is
"replayed" so that you have a full clone of the original system.[2]

In debian I use the little tool, btrbk[3], to automate all of this.
You can simply do it manually if you want.

[1] https://btrfs.wiki.kernel.org/index.php/Main_Page
[2] https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
[3] https://github.com/digint/btrbk

[toc] | [prev] | [next] | [standalone]

#180708

That's good to know.


Thank you


On 5/4/17 6:46 PM, Anders Andersson wrote:
> On Fri, May 5, 2017 at 1:17 AM, Sergei G <sergeig.public@gmail.com> wrote:
>> I would like a backup tool that does not bring a million dependencies with
>> MBs of files.  Something that works on server without X Windows and can send
>> backup to an externally attached USB drive.  Nothing fancy.   No network
>> infrastructure.  Incremental backups would be greatly appreciated.  Ability
>> to pipe to a compression program is a plus, just like I did with dump.
>> [...]
>> Any advice would be greatly appreciated.
> You won't like my solution, and it doesn't work with your current
> setup because it requires a specific file system: btrfs[1]. I'm
> posting it here for two reasons:
>   1) You might consider using btrfs on new installs
>   2) Someone else may search and find the thread
>
> After switching to btrfs I can now take instant snapshots of selected
> filesystems, transfer these to remote servers for backup, and most
> important: btrfs can track the *exact difference* between two
> snapshots taken over time, and only transfer the changes. All of this
> is very quick, because the filesystem already knows exactly what
> changed: Permission bits, file sizes, deleted files, changed data,
> whatever, all is already kept in a log. It also means that nothing
> will be missed, for example ACL bits etc.
>
> The delta is just a simple stream of data that can be compressed if
> necessary. Typically it is transmitted to a backup server where it is
> "replayed" so that you have a full clone of the original system.[2]
>
> In debian I use the little tool, btrbk[3], to automate all of this.
> You can simply do it manually if you want.
>
> [1] https://btrfs.wiki.kernel.org/index.php/Main_Page
> [2] https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
> [3] https://github.com/digint/btrbk
>

[toc] | [prev] | [next] | [standalone]

#180709

On 04-05-17, Sergei G wrote:
> That's good to know.
> 
> 
> Thank you
> 
> 
> On 5/4/17 6:46 PM, Anders Andersson wrote:
> > On Fri, May 5, 2017 at 1:17 AM, Sergei G <sergeig.public@gmail.com> wrote:
> > > I would like a backup tool that does not bring a million dependencies with
> > > MBs of files.  Something that works on server without X Windows and can send
> > > backup to an externally attached USB drive.  Nothing fancy.   No network
> > > infrastructure.  Incremental backups would be greatly appreciated.  Ability
> > > to pipe to a compression program is a plus, just like I did with dump.
> > > [...]
> > > Any advice would be greatly appreciated.
> > You won't like my solution, and it doesn't work with your current
> > setup because it requires a specific file system: btrfs[1]. I'm
> > posting it here for two reasons:
> >   1) You might consider using btrfs on new installs
> >   2) Someone else may search and find the thread
> > 
> > After switching to btrfs I can now take instant snapshots of selected
> > filesystems, transfer these to remote servers for backup, and most
> > important: btrfs can track the *exact difference* between two
> > snapshots taken over time, and only transfer the changes. All of this
> > is very quick, because the filesystem already knows exactly what
> > changed: Permission bits, file sizes, deleted files, changed data,
> > whatever, all is already kept in a log. It also means that nothing
> > will be missed, for example ACL bits etc.
> > 
> > The delta is just a simple stream of data that can be compressed if
> > necessary. Typically it is transmitted to a backup server where it is
> > "replayed" so that you have a full clone of the original system.[2]
> > 
> > In debian I use the little tool, btrbk[3], to automate all of this.
> > You can simply do it manually if you want.
> > 
> > [1] https://btrfs.wiki.kernel.org/index.php/Main_Page
> > [2] https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
> > [3] https://github.com/digint/btrbk
> > 
> 

Perhaps rsync can do the job for you?

[toc] | [prev] | [next] | [standalone]

#180744

[Multipart message — attachments visible in raw view] — view raw

it would be naive at best to think that busy files can be handled at
application level.   No, rsync cannot handle the problem of entire file
system dump.  Neither it is safe to handle all special file cases.  I am
thinking about sparse files (if Linux has those).

On Thu, May 4, 2017 at 11:52 PM, Dejan Jocic <jodejka@gmail.com> wrote:

> On 04-05-17, Sergei G wrote:
> > That's good to know.
> >
> >
> > Thank you
> >
> >
> > On 5/4/17 6:46 PM, Anders Andersson wrote:
> > > On Fri, May 5, 2017 at 1:17 AM, Sergei G <sergeig.public@gmail.com>
> wrote:
> > > > I would like a backup tool that does not bring a million
> dependencies with
> > > > MBs of files.  Something that works on server without X Windows and
> can send
> > > > backup to an externally attached USB drive.  Nothing fancy.   No
> network
> > > > infrastructure.  Incremental backups would be greatly appreciated.
> Ability
> > > > to pipe to a compression program is a plus, just like I did with
> dump.
> > > > [...]
> > > > Any advice would be greatly appreciated.
> > > You won't like my solution, and it doesn't work with your current
> > > setup because it requires a specific file system: btrfs[1]. I'm
> > > posting it here for two reasons:
> > >   1) You might consider using btrfs on new installs
> > >   2) Someone else may search and find the thread
> > >
> > > After switching to btrfs I can now take instant snapshots of selected
> > > filesystems, transfer these to remote servers for backup, and most
> > > important: btrfs can track the *exact difference* between two
> > > snapshots taken over time, and only transfer the changes. All of this
> > > is very quick, because the filesystem already knows exactly what
> > > changed: Permission bits, file sizes, deleted files, changed data,
> > > whatever, all is already kept in a log. It also means that nothing
> > > will be missed, for example ACL bits etc.
> > >
> > > The delta is just a simple stream of data that can be compressed if
> > > necessary. Typically it is transmitted to a backup server where it is
> > > "replayed" so that you have a full clone of the original system.[2]
> > >
> > > In debian I use the little tool, btrbk[3], to automate all of this.
> > > You can simply do it manually if you want.
> > >
> > > [1] https://btrfs.wiki.kernel.org/index.php/Main_Page
> > > [2] https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
> > > [3] https://github.com/digint/btrbk
> > >
> >
>
> Perhaps rsync can do the job for you?
>
>

[toc] | [prev] | [next] | [standalone]

#180745

[Multipart message — attachments visible in raw view] — view raw

I just realized that busy files are pretty much impossible to deal with no
matter what.  If DB is in the middle of the transaction, then a "snapshot"
based backup will work (we simply loose that transaction data as it is
equivalent to crash).  Other busy files might be corrupted if those files
don't use transaction model.

I am not convinced that rsync can handle all device type files, all types
of links and other attributes I don't even know about.  I am not convinced
that anyone actually restored file system using rsync.  Does it really work?

I personally use rsync to synchronize data for application deployment and
it works great.  But I usually have no special files to worry about.

On Fri, May 5, 2017 at 1:35 PM, Sergei G <sergeig.public@gmail.com> wrote:

> it would be naive at best to think that busy files can be handled at
> application level.   No, rsync cannot handle the problem of entire file
> system dump.  Neither it is safe to handle all special file cases.  I am
> thinking about sparse files (if Linux has those).
>
> On Thu, May 4, 2017 at 11:52 PM, Dejan Jocic <jodejka@gmail.com> wrote:
>
>> On 04-05-17, Sergei G wrote:
>> > That's good to know.
>> >
>> >
>> > Thank you
>> >
>> >
>> > On 5/4/17 6:46 PM, Anders Andersson wrote:
>> > > On Fri, May 5, 2017 at 1:17 AM, Sergei G <sergeig.public@gmail.com>
>> wrote:
>> > > > I would like a backup tool that does not bring a million
>> dependencies with
>> > > > MBs of files.  Something that works on server without X Windows and
>> can send
>> > > > backup to an externally attached USB drive.  Nothing fancy.   No
>> network
>> > > > infrastructure.  Incremental backups would be greatly appreciated.
>> Ability
>> > > > to pipe to a compression program is a plus, just like I did with
>> dump.
>> > > > [...]
>> > > > Any advice would be greatly appreciated.
>> > > You won't like my solution, and it doesn't work with your current
>> > > setup because it requires a specific file system: btrfs[1]. I'm
>> > > posting it here for two reasons:
>> > >   1) You might consider using btrfs on new installs
>> > >   2) Someone else may search and find the thread
>> > >
>> > > After switching to btrfs I can now take instant snapshots of selected
>> > > filesystems, transfer these to remote servers for backup, and most
>> > > important: btrfs can track the *exact difference* between two
>> > > snapshots taken over time, and only transfer the changes. All of this
>> > > is very quick, because the filesystem already knows exactly what
>> > > changed: Permission bits, file sizes, deleted files, changed data,
>> > > whatever, all is already kept in a log. It also means that nothing
>> > > will be missed, for example ACL bits etc.
>> > >
>> > > The delta is just a simple stream of data that can be compressed if
>> > > necessary. Typically it is transmitted to a backup server where it is
>> > > "replayed" so that you have a full clone of the original system.[2]
>> > >
>> > > In debian I use the little tool, btrbk[3], to automate all of this.
>> > > You can simply do it manually if you want.
>> > >
>> > > [1] https://btrfs.wiki.kernel.org/index.php/Main_Page
>> > > [2] https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
>> > > [3] https://github.com/digint/btrbk
>> > >
>> >
>>
>> Perhaps rsync can do the job for you?
>>
>>
>

[toc] | [prev] | [next] | [standalone]

#180748

On Fri, 05 May 2017, Sergei G wrote:
> I am not convinced that rsync can handle all device type files, all types
> of links and other attributes I don't even know about.  I am not convinced
> that anyone actually restored file system using rsync.  Does it really work?

Yes, it does.  Been there, done that.  But it takes a lot of command
line options, just a plain "rsync source/ dest/" usually won't be
enough.

The caveats about quiescing anything that might modify data while rsync
is running do apply.

-- 
  Henrique Holschuh

[toc] | [prev] | [next] | [standalone]

#180753

Henrique de Moraes Holschuh writes:
> On Fri, 05 May 2017, Sergei G wrote:
> > I am not convinced that rsync can handle all device type files, all types
> > of links and other attributes I don't even know about.  I am not convinced
> > that anyone actually restored file system using rsync.  Does it really work?
> 
> Yes, it does.  Been there, done that.  But it takes a lot of command
> line options, just a plain "rsync source/ dest/" usually won't be
> enough.
> 
> The caveats about quiescing anything that might modify data while rsync
> is running do apply.
>

Also be advised if "dest/" is a FAT formatted SD card, there is a file
size limit of 4GB, (and the -aAXv argument to rsync(1) may not be
viable to restore the permission/ACL of files/directories.)

If "dest/" is archival, (i.e., must be recoverable for a long period
of time,) saving the MD5/SHA* hash values of files in "source/" is
advisable for verification that a file is not corrupt sometime in the
future.

    John

-- 

John Conover, conover@rahul.net, http://www.johncon.com/

[toc] | [prev] | [next] | [standalone]

#180758

On Fri, 05 May 2017, John Conover wrote:
> Also be advised if "dest/" is a FAT formatted SD card, there is a file
> size limit of 4GB, (and the -aAXv argument to rsync(1) may not be
> viable to restore the permission/ACL of files/directories.)

Using rsync on non-POSIX filesystems is a Bad Idea, it is going to
mangle a lot of metadata.

> If "dest/" is archival, (i.e., must be recoverable for a long period
> of time,) saving the MD5/SHA* hash values of files in "source/" is
> advisable for verification that a file is not corrupt sometime in the
> future.

In that case, using tar+lzip (*not* xz) and adding par2 ECC blocks might
be in order.  Refer to the lzip/plzip and par2 packages.

-- 
  Henrique Holschuh

[toc] | [prev] | [next] | [standalone]

#180747

On Fri, 05 May 2017, Sergei G wrote:
> it would be naive at best to think that busy files can be handled at
> application level.   No, rsync cannot handle the problem of entire file
> system dump.  Neither it is safe to handle all special file cases.  I am
> thinking about sparse files (if Linux has those).

Yes, Linux has sparse files.  And yes, rsync can deal with sparse files,
check its documentation.  It can also deal with many "special files",
extended attributes and ACLs properly, if directed to do so.

Also, shortcut "file/inode changed" detection can break on Debian
*system* directories.  *You have been warned*.  Either always do a full
backup of those, or use something that actually _always_ checksums the
contents (such as rsync -c), when doing dedup or incremental backups.

As for live data such as running database backends, or active logs, you
have to snapshot/quiece such applications for a rsync backup just like
you'd have to do for filesystem or device snapshotting, or any other
non-application-level backup.

-- 
  Henrique Holschuh

[toc] | [prev] | [next] | [standalone]

#180767

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, May 05, 2017 at 01:35:35PM -0700, Sergei G wrote:
> it would be naive at best to think that busy files [...]

What is a "busy" file?

regards
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlkNZJkACgkQBcgs9XrR2kYJZACfdF9cY0x4iocuTu6amaWGF1Vk
EsoAnAoJm8i7EwwpxJOo5RluDrvQ/2mu
=x7Yi
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#180773

On Sat, May 06, 2017 at 07:52:25AM +0200, tomas@tuxteam.de wrote:
> On Fri, May 05, 2017 at 01:35:35PM -0700, Sergei G wrote:
> > it would be naive at best to think that busy files [...]
> 
> What is a "busy" file?

Open for writing.

-dsr-

[toc] | [prev] | [next] | [standalone]

#180774

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sat, May 06, 2017 at 06:42:46AM -0400, Dan Ritter wrote:
> On Sat, May 06, 2017 at 07:52:25AM +0200, tomas@tuxteam.de wrote:
> > On Fri, May 05, 2017 at 01:35:35PM -0700, Sergei G wrote:
> > > it would be naive at best to think that busy files [...]
> > 
> > What is a "busy" file?
> 
> Open for writing.

Hm. That just means that the file will change after being copied.
In itself not a problem (Note that there are other, stronger concepts
of busy. An open file for write can be opened for read/write by
another process, can be deleted, yadda, yadda. It can be definitely
copied. Less civilised operating systems lock files when opened for
write, but not the Unixoids).

cheers
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlkN1f4ACgkQBcgs9XrR2kZG5ACcCLASlzpysLzYvxA/VLSZfM8u
w0cAmwW4hLprAR9DDeO0dUkA78JpA/OO
=sRyK
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#180778

On Sat, 06 May 2017, tomas@tuxteam.de wrote:
> Hm. That just means that the file will change after being copied.

There can also exist busy file *sets*, which need to be atomically
snapshotted (this is, of course, application specific).

"busy", as far as backups go, means:

1. a file changes [either in memory/buffer cache, or in the backend
   storage] *while* being copied (so it is internally inconsistent) --
   backup will have sections of the old file, and sections of the new
   file.

2. one or more components of a set of several files change *during*
   backup (files may be internally consistent, but the set of files
   itself isn't).

In some cases, (1) and (2) can manifest as files missing from the
backup, or being zero-sized.

-- 
  Henrique Holschuh

[toc] | [prev] | [next] | [standalone]

#180783

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sat, May 06, 2017 at 02:18:47PM -0300, Henrique de Moraes Holschuh wrote:
> On Sat, 06 May 2017, tomas@tuxteam.de wrote:
> > Hm. That just means that the file will change after being copied.
> 
> There can also exist busy file *sets*, which need to be atomically
> snapshotted (this is, of course, application specific).

Yes, this is what I meant by "skew": the point in time files are
copied varies. Even worse, some applications might have essential
parts of their state in RAM, not committed to file at all.

That's why I mentioned snapshotting (LVM, or overlayfs, or a file
system which can do the trick, as zfs or btrfs). If you go the whole
way of zfs or btrfs, those file systems can do better than rsync
anyway.

For a database worth its salt there are other possibilities (PostgreSQL
can save a consistent state on-disk) -- after all they're in the
transaction business.

> "busy", as far as backups go, means:
> 
> 1. a file changes [either in memory/buffer cache, or in the backend
>    storage] *while* being copied (so it is internally inconsistent) --
>    backup will have sections of the old file, and sections of the new
>    file.

Rsync will notice and warn on that. Of course, at the next attempt
that can happen again :)

> 2. one or more components of a set of several files change *during*
>    backup (files may be internally consistent, but the set of files
>    itself isn't).

Only the application can know that. But this is a realm where snapshots
won't help either, without the application's help.

> In some cases, (1) and (2) can manifest as files missing from the
> backup, or being zero-sized.

If I understood you correctly (2) won't produce empty or missing files,
just inconsistent application state.

cheers
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlkOJ/sACgkQBcgs9XrR2kZ9DQCfX6V5a9UdO2pixunhFMyuM0ug
GFoAn1R/RLYL2KeRIkfBsL7BqrJJb1/V
=nRRb
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#180788

On Sat, 06 May 2017, tomas@tuxteam.de wrote:
> > 2. one or more components of a set of several files change *during*
> >    backup (files may be internally consistent, but the set of files
> >    itself isn't).
> 
> Only the application can know that. But this is a realm where snapshots
> won't help either, without the application's help.
> 
> > In some cases, (1) and (2) can manifest as files missing from the
> > backup, or being zero-sized.
> 
> If I understood you correctly (2) won't produce empty or missing files,
> just inconsistent application state.

If the application does delete-then-create, it is a race, you can hit it
after the unlink but before create (file will be missing from the
backup), after the create before any writing (file in backup will be
zero-sized), or during writes (file will be incomplete).  This is valid
for (1) and (2).

If the application does write-in-place (typical for mmap()), you get an
internally inconsistent file, possibly incorrectly sized.  If it does
truncate-and-write, "possibly incorrectly sized" also includes the
possibility of an empty file.

Applications doing atomic replacement change this: you will get either
the old or the new file or directory (application places the entire file
set on its own directory).  No race, there.


And, finally, as you pointed out, rsync will usually[1] be able to
notice when an inode changes behind its feet (i.e. while it is copying
its contents).

[1] I am not sure it can detect changes that are not reflected in the
inode for whatever reason (e.g. because the writer reset all relevant
inode metadata after writing).

-- 
  Henrique Holschuh

[toc] | [prev] | [next] | [standalone]

#180795

On 07/05/17 01:56, tomas@tuxteam.de wrote:
> On Sat, May 06, 2017 at 06:42:46AM -0400, Dan Ritter wrote:
>> On Sat, May 06, 2017 at 07:52:25AM +0200, tomas@tuxteam.de wrote:
>>> On Fri, May 05, 2017 at 01:35:35PM -0700, Sergei G wrote:
>>>> it would be naive at best to think that busy files [...]
>>> What is a "busy" file?
>> Open for writing.
> Hm. That just means that the file will change after being copied.
> In itself not a problem (Note that there are other, stronger concepts
> of busy. An open file for write can be opened for read/write by
> another process, can be deleted, yadda, yadda. It can be definitely
> copied. Less civilised operating systems lock files when opened for
> write, but not the Unixoids).

If a file is updated while it is being copied, it may contain only half 
a change set and be in an internally inconsistent state, perhaps making 
it unusable as a backup. Writes are typically not atomic. The same 
problem applies to collections of files that reference each other.

Kind regards,

-- 
Ben Caradoc-Davies <ben@transient.nz>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

[toc] | [prev] | [next] | [standalone]

#180808

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, May 07, 2017 at 10:53:51AM +1200, Ben Caradoc-Davies wrote:

[also a reply to Henrique, elsewhere in this thread]

> If a file is updated while it is being copied, it may contain only
> half a change set and be in an internally inconsistent state,
> perhaps making it unusable as a backup. Writes are typically not
> atomic. The same problem applies to collections of files that
> reference each other.
> 
> Kind regards,

Ben, Henrique -- no questions. This is what I subsumed under
"skew": application state may be dispersed across different
places in a file, across different files or even partly not
in files at all (e.g. in RAM: imagine a BTree with just parts
of its pointer structure not yet committed to disk).

Of course you can't ever win unless you collaborate with the
application in those cases (even with magic file systems like
ZFS or btrfs).

Then there is this subtle "file data" and "file metadata"
thing, which is an issue even with carefully designed applications
and file systems. It's even difficult to reach a consensus on
what is "right", remember the ext3/ext4 data loss episode[1]?

This is where shapshotting magic, be it built-in (zfs, btrfs)
or bolted-on (overlayfs, lvm) might help a bit: freeze a snapshot,
back up that (in the first case, the file systems provide a native
way to do that, in the second case, rsync is a pretty viable
way of doing things). I said "might help a bit" because the
ultimate consistency criterion is the application! A consistent
file system view might just be this truncated-to-zero file,
only the application "knows" at that point (e.g. by keeping
its data in an already unlinked file which is still open,
or somewhere in RAM, or...).

So your choices are

 - for the applications you really care about, look into
   what they are doing. Grown up apps will support you in
   that (I gave the PostgreSQL example above). Typically
   you can wrap the backup process in guards like ("keep
   your on-disk state consistent"[1]..."now you can relax").
   Note that to avoid races this structure is more or less
   necessary. The only real difference to the "magic
   snapshot" thing is that the latter happens very quickly.

 - for all the others... just relax.

Otherwise, "on line" backup is simply not an option.

cheers

[1] That doesn't mean necessarily frozen. PostgreSQL, for example,
   continues writing to the WAL, it just eats through its storage
   at a higher pace.

- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlkO7SgACgkQBcgs9XrR2kYf9ACeM2njgrSttOUPRk4D6fJqJtjQ
qmkAn38VbkKiOlADe+33teN8uzcbLa2C
=uKNk
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#180810

On Sunday, May 07, 2017 05:47:20 AM tomas@tuxteam.de wrote:

<good stuff elided>

> Otherwise, "on line" backup is simply not an option.

Thanks!  I've been paying at least peripheral attention to this thread (as I 
do to many), and got to thinking / wondering about how to deal with backup in 
an "enterprise" type of setting (by that, I'm trying to imply an "application" 
that is distributed and run over multiple computers / servers to serve a large 
number of (possibly transient) users).

I guess the way that comes to mind, maybe somewhat implied by what you've 
written (or at least inferred by me ;-) would be to take down one or more of 
the servers at a time, make static backups of those, then restore those to 
service and repeat until all have been backed up, very likely repeating that 
continuously.

I'm sure there are some subtleties that I'm not even thinking about (not that 
I'd expect to be able to do so).

I guess I sort of wanted to write that down and see if anyone had any insights 
into possibly better ways.

Oh, ok, I guess (well, I know RAID isn't really for backup, it's for uptime as 
is often stated on this and other lists), but something like RAID where 
relevant data is written to more than one disk and / or storage "farm", one 
being the "in service" device (to serve data to the app in "real time") and 
one or more just recording data as backup.

Anyway, I'd be interested in any comments.

I was going to mark this OT, but then I re-read the Subject, and it still 
seems OT (on topic) ;-)

[toc] | [prev] | [next] | [standalone]

#180815

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, May 07, 2017 at 08:42:36AM -0400, rhkramer@gmail.com wrote:
> On Sunday, May 07, 2017 05:47:20 AM tomas@tuxteam.de wrote:
> 
> <good stuff elided>
> 
> > Otherwise, "on line" backup is simply not an option.

[...]

> I guess the way that comes to mind, maybe somewhat implied by what you've 
> written (or at least inferred by me ;-) would be to take down one or more of 
> the servers at a time, make static backups of those, then restore those to 
> service and repeat until all have been backed up, very likely repeating that 
> continuously.

"Taking down" as in "stopping all relevant applications" (shutting down
the whole thing would be a superset of that, of course ;)

[...]

> Oh, ok, I guess (well, I know RAID isn't really for backup, it's for uptime as 
> is often stated on this and other lists), but something like RAID where 
> relevant data is written to more than one disk and / or storage "farm", one 
> being the "in service" device (to serve data to the app in "real time") and 
> one or more just recording data as backup.

This is a variant on the snapshot pattern. You take a snapshot of the
file system and back up that. You are guaranteed that the *file system*
backup is consistent (well, modulo bugs and glitches), but since the
applications don't have a clue what's going on, application state
might still be inconsistent, unless they go out of their way to keep
a consistent state on-disk all of the time (in the above scenario
of shutdown, we assumed that the application leaves a consistent
state after being shut down, which is a reasonable assumption, I'd
say).

That said, and as you can see in the PostgreSQL example, with a little
help of your application, the snapshot thing works (in the case of
PostgreSQL things are much more relaxed and you can usually go with
plain rsync, for example, without resorting to snapshot).

All in all I'm pretty happy with a plain, straight rsync-style backup
for my workstation. The probability of a busted backup is so low that
I fear more for my backup medium going sour.

On a high-churn machine with very valuable data things might look
differently, though...

regards
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlkPNaIACgkQBcgs9XrR2kYGWgCfcR14ZU9wA9kKDampxUY7MCHb
IjUAnj0zjx2U5yFy7TIIO/rP5UO7E6QI
=bD6R
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.user

csiph-web