Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #188743 > unrolled thread

Sync two disks and hot swap

Back to article view | Back to linux.debian.user

Contents

  Sync two disks and hot swap Dominik George <nik@naturalnet.de> - 2017-11-08 12:00 +0100
    Re: Sync two disks and hot swap Michael Stone <mstone@debian.org> - 2017-11-08 18:10 +0100
      Re: Sync two disks and hot swap Dominik George <nik@naturalnet.de> - 2017-11-08 20:00 +0100
        Re: Sync two disks and hot swap Michael Stone <mstone@debian.org> - 2017-11-08 20:20 +0100
          Re: Sync two disks and hot swap Dominik George <nik@naturalnet.de> - 2017-11-09 10:20 +0100
            Re: Sync two disks and hot swap Michael Stone <mstone@debian.org> - 2017-11-09 13:40 +0100
              Re: Sync two disks and hot swap David Christensen <dpchrist@holgerdanske.com> - 2017-11-09 20:00 +0100
                Re: Sync two disks and hot swap Michael Stone <mstone@debian.org> - 2017-11-09 20:40 +0100
                  Re: Sync two disks and hot swap Curt <curty@free.fr> - 2017-11-10 11:00 +0100
                    Re: Sync two disks and hot swap Curt <curty@free.fr> - 2017-11-10 11:10 +0100
    Re: Sync two disks and hot swap David Christensen <dpchrist@holgerdanske.com> - 2017-11-09 00:10 +0100
      Re: Sync two disks and hot swap Bob Weber <bobrweber@gmail.com> - 2017-11-09 02:50 +0100
        Re: Sync two disks and hot swap David Christensen <dpchrist@holgerdanske.com> - 2017-11-09 08:10 +0100
          Re: Sync two disks and hot swap Bob Weber <bobrweber@gmail.com> - 2017-11-09 23:50 +0100
            Re: Sync two disks and hot swap David Christensen <dpchrist@holgerdanske.com> - 2017-11-10 00:10 +0100

#188743 — Sync two disks and hot swap

[Multipart message — attachments visible in raw view] — view raw

Hi,

I have the following scenario:

 * A server with two hard drives in removable cases
 * A backup process writes data to both disks, making up a live backup server
 * A third disk is to be kept off-site
 * On a ergular basis, I want to hot-swap one of the disks, as in, remove
   one of the two synced disks and replace it with the stale off-site copy,
   and put the now recent copy off-site

I figure that a simple software RAID 1 would do the trick, but it is not
really made for it and would need some complex manual intervention in
order to not break the state on the removed disk.

Any ideas on how to achieve this, or arguments that RAID 1 would indeed
be a good solution?

Cheers,
Nik

[toc] | [next] | [standalone]

#188758

On Wed, Nov 08, 2017 at 11:49:33AM +0100, Dominik George wrote:
> * A server with two hard drives in removable cases
> * A backup process writes data to both disks, making up a live backup server
> * A third disk is to be kept off-site
> * On a ergular basis, I want to hot-swap one of the disks, as in, remove
>   one of the two synced disks and replace it with the stale off-site copy,
>   and put the now recent copy off-site
>
>I figure that a simple software RAID 1 would do the trick, but it is not
>really made for it and would need some complex manual intervention in
>order to not break the state on the removed disk.
>
>Any ideas on how to achieve this, or arguments that RAID 1 would indeed
>be a good solution?

I'd personally tend to avoid using an md mirror like this, though it can 
be made to work. There are some cases in which you might accidentally 
lose a mirror (e.g., plug the offsite backup in to restore something, 
have it sync with a newer copy unintentionally) and it requires a bit 
more effort to do partial restores (like trying to retrieve one deleted 
file) because of conflicts between the UUIDs on the filesystems as well 
as having to deal with the mirror itself.

Instead, if you just want a disk that has a readable copy of the files, 
you may find that rsync is more straightforward and can be a lot faster 
after the first time if the volume of changes is a small percentage of 
the total.

Mike Stone

[toc] | [prev] | [next] | [standalone]

#188764

Hi,

> Instead, if you just want a disk that has a readable copy of the files, you
> may find that rsync is more straightforward and can be a lot faster after
> the first time if the volume of changes is a small percentage of the total.

Yes, of course. But that would not lead to an identical copy of the
disk, only the files in its filesystem.

I will choose that way if nothing else comes up in this thread.

-nik

[toc] | [prev] | [next] | [standalone]

#188765

On Wed, Nov 08, 2017 at 07:55:24PM +0100, Dominik George wrote:
>Hi,
>
>> Instead, if you just want a disk that has a readable copy of the files, you
>> may find that rsync is more straightforward and can be a lot faster after
>> the first time if the volume of changes is a small percentage of the total.
>
>Yes, of course. But that would not lead to an identical copy of the
>disk, only the files in its filesystem.

what is the goal in having an identical copy of the disk?

Mike Stone

[toc] | [prev] | [next] | [standalone]

#188805

> what is the goal in having an identical copy of the disk?

It's not even so much that.

It's that the person who will be changing the disks will be hardly
capable of just that, and will not get anything close to root access to
the machine.

And I am afraid that all that complexity around automounting the
different filesystems, autounmounting, and automatically ensuring the
filesystem is clean and unmounted at the time the disk is to be swapped
could be quite unreliable.

-nik

[toc] | [prev] | [next] | [standalone]

#188810

On Thu, Nov 09, 2017 at 10:13:17AM +0100, Dominik George wrote:
>> what is the goal in having an identical copy of the disk?
>
>It's not even so much that.
>
>It's that the person who will be changing the disks will be hardly
>capable of just that, and will not get anything close to root access to
>the machine.
>
>And I am afraid that all that complexity around automounting the
>different filesystems, autounmounting, and automatically ensuring the
>filesystem is clean and unmounted at the time the disk is to be swapped
>could be quite unreliable.

Honestly, it's an easier problem than ensuring that a raid element is 
cleanly unmounted. Historically there have been external drives with 
buttons that could be mapped to events (but I have no idea what's on the 
market right now that might have an addressable button). Otherwise maybe 
allow the user to run "eject /whatever" with sudo, or wrap that in a 
single-function console login or somesuch. Another possibility is to 
wrap the rsync functionality in a mount/unmount cycle, so that the disk 
is always unmounted and ready to go except when actively transferring 
data.

Mike Stone

[toc] | [prev] | [next] | [standalone]

#188821

On 11/09/17 04:35, Michael Stone wrote:
> On Thu, Nov 09, 2017 at 10:13:17AM +0100, Dominik George wrote:
>>> what is the goal in having an identical copy of the disk?
>>
>> It's not even so much that.
>>
>> It's that the person who will be changing the disks will be hardly
>> capable of just that, and will not get anything close to root access to
>> the machine.
>>
>> And I am afraid that all that complexity around automounting the
>> different filesystems, autounmounting, and automatically ensuring the
>> filesystem is clean and unmounted at the time the disk is to be swapped
>> could be quite unreliable.
> 
> Honestly, it's an easier problem than ensuring that a raid element is 
> cleanly unmounted. Historically there have been external drives with 
> buttons that could be mapped to events (but I have no idea what's on the 
> market right now that might have an addressable button). Otherwise maybe 
> allow the user to run "eject /whatever" with sudo, or wrap that in a 
> single-function console login or somesuch. Another possibility is to 
> wrap the rsync functionality in a mount/unmount cycle, so that the disk 
> is always unmounted and ready to go except when actively transferring data.

There are many choices for software:

https://en.wikipedia.org/wiki/List_of_backup_software

https://en.wikipedia.org/wiki/Comparison_of_backup_software


STFW for others that aren't listed by WikiPedia.  For example, backup 
software with ZFS back end:

http://www.freenas.org/

http://www.znapzend.org/


Beware of falling into the "roll your own" trap.


David

[toc] | [prev] | [next] | [standalone]

#188827

On Thu, Nov 09, 2017 at 10:54:57AM -0800, David Christensen wrote:
>There are many choices for software:

I didn't get the impression it's a software question as much as a "how 
does unskilled/unprivileged person know when it's ok to pull out the 
drive" question, because the actual backing up is already done.

And remember the first rule of backup software: all backup software is 
terrible. :) The reason there are so many packages available is that so 
many people couldn't stand using anything that already existed.

Mike Stone

[toc] | [prev] | [next] | [standalone]

#188836

On 2017-11-09, Michael Stone <mstone@debian.org> wrote:
> On Thu, Nov 09, 2017 at 10:54:57AM -0800, David Christensen wrote:
>>There are many choices for software:
>
> I didn't get the impression it's a software question as much as a "how 
> does unskilled/unprivileged person know when it's ok to pull out the 
> drive" question, because the actual backing up is already done.

Hack the person.

> And remember the first rule of backup software: all backup software is 
> terrible. :) The reason there are so many packages available is that so 
> many people couldn't stand using anything that already existed.

There are good Chinese restaurants.

> Mike Stone
>
>


-- 
"A simpering Bambi narcissist and a thieving, fanatical Albanian dwarf."
Christopher Hitchens, commenting shortly after the nearly concurrent deaths 
of Lady Diana and Mother Theresa.

[toc] | [prev] | [next] | [standalone]

#188837

On 2017-11-10, Curt <curty@free.fr> wrote:
>>
>> I didn't get the impression it's a software question as much as a "how 
>> does unskilled/unprivileged person know when it's ok to pull out the 
>> drive" question, because the actual backing up is already done.
>
> Hack the person.
>
>> And remember the first rule of backup software: all backup software is 
>> terrible. :) The reason there are so many packages available is that so 
>> many people couldn't stand using anything that already existed.
>
> There are good Chinese restaurants.

Sorry. I meant to say there can be good dishes in Chinese restaurants.
Excuse me for the mental faux pas.

>> Mike Stone
>>
>>
>
>


-- 
"A simpering Bambi narcissist and a thieving, fanatical Albanian dwarf."
Christopher Hitchens, commenting shortly after the nearly concurrent deaths 
of Lady Diana and Mother Theresa.

[toc] | [prev] | [next] | [standalone]

#188789

On 11/08/17 02:49, Dominik George wrote:
> Hi,
> 
> I have the following scenario:
> 
>   * A server with two hard drives in removable cases
>   * A backup process writes data to both disks, making up a live backup server
>   * A third disk is to be kept off-site
>   * On a ergular basis, I want to hot-swap one of the disks, as in, remove
>     one of the two synced disks and replace it with the stale off-site copy,
>     and put the now recent copy off-site
> 
> I figure that a simple software RAID 1 would do the trick, but it is not
> really made for it and would need some complex manual intervention in
> order to not break the state on the removed disk.
> 
> Any ideas on how to achieve this, or arguments that RAID 1 would indeed
> be a good solution?

Are the two drives in RAID (1?) or do they each have their own file system?


I have read articles about building a RAID 1 with three drives, 
migrating in data, pulling one drive and placing it off-site, operating 
in degraded mode on two drives, and then periodically re-installing the 
third drive, resilvering, pulling one drive and placing it off-site, and 
returning to degraded operations on two drives.  But STFW just now, I 
see a lot of posts with titles indicating this is a bad idea.


I have three drives in mobile dock drawers, each with LUKS and ext4. 
One is on-line in my backup server, one is near-site, and one is 
off-site.  Periodically, I put the near-site drive into the backup 
server, rsync the on-line drive to the near-site drive, remove the 
near-site drive, and then swap the near-site and off-site drives. 
Admittedly the wear is uneven, but it's KISS and it works.


But what I really want is some form of snapshot technology (rsync/hard 
link, LVM, btrfs, ZFS) with all the goodies -- realtime compression, 
realtime de-duplication, and encryption.  I need a more powerful backup 
server (many core processor with AES-NI, 16+ GB RAM, SSD caches, etc.).


David

[toc] | [prev] | [next] | [standalone]

#188794

[Multipart message — attachments visible in raw view] — view raw

On 11/8/17 5:59 PM, David Christensen wrote:
> On 11/08/17 02:49, Dominik George wrote:
>> Hi,
>>
>> I have the following scenario:
>>
>>   * A server with two hard drives in removable cases
>>   * A backup process writes data to both disks, making up a live backup server
>>   * A third disk is to be kept off-site
>>   * On a ergular basis, I want to hot-swap one of the disks, as in, remove
>>     one of the two synced disks and replace it with the stale off-site copy,
>>     and put the now recent copy off-site
>>
>> I figure that a simple software RAID 1 would do the trick, but it is not
>> really made for it and would need some complex manual intervention in
>> order to not break the state on the removed disk.
>>
>> Any ideas on how to achieve this, or arguments that RAID 1 would indeed
>> be a good solution?
>
> Are the two drives in RAID (1?) or do they each have their own file system?
>
>
> I have read articles about building a RAID 1 with three drives, migrating in
> data, pulling one drive and placing it off-site, operating in degraded mode on
> two drives, and then periodically re-installing the third drive, resilvering,
> pulling one drive and placing it off-site, and returning to degraded
> operations on two drives.  But STFW just now, I see a lot of posts with titles
> indicating this is a bad idea.
>
>
> I have three drives in mobile dock drawers, each with LUKS and ext4. One is
> on-line in my backup server, one is near-site, and one is off-site. 
> Periodically, I put the near-site drive into the backup server, rsync the
> on-line drive to the near-site drive, remove the near-site drive, and then
> swap the near-site and off-site drives. Admittedly the wear is uneven, but
> it's KISS and it works.
>
>
> But what I really want is some form of snapshot technology (rsync/hard link,
> LVM, btrfs, ZFS) with all the goodies -- realtime compression, realtime
> de-duplication, and encryption.  I need a more powerful backup server (many
> core processor with AES-NI, 16+ GB RAM, SSD caches, etc.).
>
>
> David
>
>
I have used raid 1 to make a drive I can take off site for backup.  You just
grow the raid 1 array by one disk and add the disk you want to take out (even on
a usb/sata connection ... but slow).   Of course the disk or partition(s) need
to be the same size as the array.  Let it sync and then boot to a live cd and
you can fail and remove that drive.   Or just power down and remove the drive.  
That way the embedded file system will be unmounted correctly.  I have then
taken that one drive and connected it to another system and been able to run the
raid 1 in degraded mode and mount the embedded file system(s) to get to the
files.  To make the original raid happy just grow the array again setting the
number of drives back to what it was originally (you can grow to a smaller
number).  The syncing can be slow since every byte on the drive needs to be
"synced" instead of just the space the files take up.

I tried to use btrfs in several VMs running debian but I kept having to delete
snapshots to make sure I had enough free space.

-- 


*...Bob*

[toc] | [prev] | [next] | [standalone]

#188801

On 11/08/17 17:44, Bob Weber wrote:
> On 11/8/17 5:59 PM, David Christensen wrote:
>> I have read articles about building a RAID 1 with three drives, migrating in
>> data, pulling one drive and placing it off-site, operating in degraded mode on
>> two drives, and then periodically re-installing the third drive, resilvering,
>> pulling one drive and placing it off-site, and returning to degraded
>> operations on two drives.  But STFW just now, I see a lot of posts with titles
>> indicating this is a bad idea.
...
>> But what I really want is some form of snapshot technology (rsync/hard link,
>> LVM, btrfs, ZFS) with all the goodies -- realtime compression, realtime
>> de-duplication, and encryption.  I need a more powerful backup server (many
>> core processor with AES-NI, 16+ GB RAM, SSD caches, etc.).

> I have used raid 1 to make a drive I can take off site for backup.  You just
> grow the raid 1 array by one disk and add the disk you want to take out (even on
> a usb/sata connection ... but slow).   Of course the disk or partition(s) need
> to be the same size as the array.  Let it sync and then boot to a live cd and
> you can fail and remove that drive.   Or just power down and remove the drive.
> That way the embedded file system will be unmounted correctly.  I have then
> taken that one drive and connected it to another system and been able to run the
> raid 1 in degraded mode and mount the embedded file system(s) to get to the
> files.  To make the original raid happy just grow the array again setting the
> number of drives back to what it was originally (you can grow to a smaller
> number).  The syncing can be slow since every byte on the drive needs to be
> "synced" instead of just the space the files take up.

Okay.  What RAID technology were you using -- LVM, mdadm, btrfs, ZFS, other?


> I tried to use btrfs in several VMs running debian but I kept having to delete
> snapshots to make sure I had enough free space.

Compression and deduplication may have helped.


Backups are an ideal use-case for compression and deduplication.


The other half of snapshots is replication; ideally using a deduplicated 
and compressed stream.


Now that ZOL is in the Debian 'contrib' repository, I need to play with 
it some more:

     https://packages.debian.org/stretch/zfs-dkms


David

[toc] | [prev] | [next] | [standalone]

#188832

[Multipart message — attachments visible in raw view] — view raw

On 11/9/17 2:01 AM, David Christensen wrote:
> On 11/08/17 17:44, Bob Weber wrote:
>> On 11/8/17 5:59 PM, David Christensen wrote:
>>> I have read articles about building a RAID 1 with three drives, migrating in
>>> data, pulling one drive and placing it off-site, operating in degraded mode on
>>> two drives, and then periodically re-installing the third drive, resilvering,
>>> pulling one drive and placing it off-site, and returning to degraded
>>> operations on two drives.  But STFW just now, I see a lot of posts with titles
>>> indicating this is a bad idea.
> ...
>>> But what I really want is some form of snapshot technology (rsync/hard link,
>>> LVM, btrfs, ZFS) with all the goodies -- realtime compression, realtime
>>> de-duplication, and encryption.  I need a more powerful backup server (many
>>> core processor with AES-NI, 16+ GB RAM, SSD caches, etc.).
>
>> I have used raid 1 to make a drive I can take off site for backup.  You just
>> grow the raid 1 array by one disk and add the disk you want to take out (even on
>> a usb/sata connection ... but slow).   Of course the disk or partition(s) need
>> to be the same size as the array.  Let it sync and then boot to a live cd and
>> you can fail and remove that drive.   Or just power down and remove the drive.
>> That way the embedded file system will be unmounted correctly.  I have then
>> taken that one drive and connected it to another system and been able to run the
>> raid 1 in degraded mode and mount the embedded file system(s) to get to the
>> files.  To make the original raid happy just grow the array again setting the
>> number of drives back to what it was originally (you can grow to a smaller
>> number).  The syncing can be slow since every byte on the drive needs to be
>> "synced" instead of just the space the files take up.
>
> Okay.  What RAID technology were you using -- LVM, mdadm, btrfs, ZFS, other?
>
I use software raid with mdadm.  Its pretty forgiving with powering down and
removing a drive (after sync) and growing the array back down to the original
size.  I mainly do this on my backup server running backuppc.  The files are
compressed and hard linked between backups if they have not changed.  This makes
any other type of offsite backup pretty hard ... rsync just ran out of memory. 
So adding a drive to the raid 1 and syncing is easy in comparison.  Having grub
make the drive bootable  (since the os is also on a raid 1 partition) makes the
drive very easy to just install in new hardware and get going again (assuming
the original backup system was destroyed).

-- 


*...Bob*

[toc] | [prev] | [next] | [standalone]

#188834

On 11/09/17 14:46, Bob Weber wrote:
> On 11/9/17 2:01 AM, David Christensen wrote:
>> Okay.  What RAID technology were you using -- LVM, mdadm, btrfs, ZFS, other?
>>
> I use software raid with mdadm.  Its pretty forgiving with powering down and
> removing a drive (after sync) and growing the array back down to the original
> size.  I mainly do this on my backup server running backuppc.  The files are
> compressed and hard linked between backups if they have not changed.  This makes
> any other type of offsite backup pretty hard ... rsync just ran out of memory.
> So adding a drive to the raid 1 and syncing is easy in comparison.  Having grub
> make the drive bootable  (since the os is also on a raid 1 partition) makes the
> drive very easy to just install in new hardware and get going again (assuming
> the original backup system was destroyed).

Yes, rsync's --link-dest=DIR option gets tedious very quickly when you 
have trees of directories.  (Driving rsync using scripts can solve this; 
perhaps this was a motivator for BackupPC?)


If your array file system is ext2, ext3, or ext4, how about unmounting 
the array, unmounting the destination partition, and using dump(8) and 
restore(8)?


David

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web