Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #225961

Re: Homebuilt NAS Advice

From Leslie Rhorer <lesrhorer@att.net>
Newsgroups linux.debian.user
Subject Re: Homebuilt NAS Advice
Date 2020-08-07 04:10 +0200
Message-ID <AAZZv-1pc-5@gated-at.bofh.it> (permalink)
References (5 earlier) <AAlSp-2j1-1@gated-at.bofh.it> <AAnrb-34D-5@gated-at.bofh.it> <AAwNQ-gO-3@gated-at.bofh.it> <AADcB-4b4-1@gated-at.bofh.it> <AAQMx-4gl-7@gated-at.bofh.it>
Organization linux.* mail to news gateway

Show all headers | View raw

On 8/6/2020 11:15 AM, deloptes wrote:
> Leslie Rhorer wrote:
> 
>> A few copies of what size? ??The backup server is an exact mirror of the
>> main server, plus several T of additional files I don't need on the main
>> server.
> 
> The question is how much back in time you can go. If you have just a
> mirror - it is one copy, so you can go one step back only.

	Which is why I have offline archival backups in addition to the mirror. 
  Since this is primarily a media server, more than 21T of the files 
will never change.  More than 1/2 of the files are at least 5 years old:

root@RAID-Server:/# find /RAID/ | wc -l
382489
root@RAID-Server:/# find /RAID/ -mtime +365 | wc -l
352044
root@RAID-Server:/# find /RAID/ -mtime +730 | wc -l
286695
root@RAID-Server:/# find /RAID/ -mtime +1825 | wc -l
244813
root@RAID-Server:/# find /RAID/ -mtime +3650 | wc -l
22954

	Almost all the files older than 1 year are new, not modified.

	Note also that while there are 382489 files on the array, about 3000 
are considerably larger than 1G, representing more than 70% of the space 
on the drive.  Not one of these files are duplicates at any level.  The 
majority of the remaining files (275729) are less than 10K each.  Even 
if every one were a duplicate, that only represents less than 1.5G of 
uncompressed storage.  The vast majority of those never change, either. 
In reality, the savings would be zilch.

> I don't argue about your approach, it is your design, your servers, but in
> your case I would use definitely deduplication.

	And save what?  About 0.002%.  There are maybe 100 duplicate files 
larger than 50K on my server, excluding hard links and temporary files. 
The hard links are handled expeditiously by DAR.  Temporary files don't 
get backed up.

> If you have a problem with data transfer - use fiber, by that volume of data
> 10G is a must.

	I do.  The servers have 10G optical links between them.  A full backup 
to the RAID 6 array takes several days.  A full backup to single drives 
takes 2 weeks, because single drives are limited to about 800Mbps, while 
the array can gulp down nerly 4Gbps.  Nightly backups (via rsync) take a 
few minutes, at most.

Back to linux.debian.user | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Homebuilt NAS Advice Patrick Bartek <patrick.bartek@gmail.com> - 2020-07-29 22:00 +0200
  Re: Homebuilt NAS Advice Dan Ritter <dsr@randomstring.org> - 2020-07-29 22:20 +0200
    Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-07-29 22:30 +0200
      Re: Homebuilt NAS Advice Dan Ritter <dsr@randomstring.org> - 2020-07-29 23:40 +0200
    Re: Homebuilt NAS Advice Patrick Bartek <patrick.bartek@gmail.com> - 2020-07-30 02:50 +0200
      Re: Homebuilt NAS Advice Tom Dial <tddial@comcast.net> - 2020-07-30 22:50 +0200
        Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-02 01:40 +0200
          Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-02 03:10 +0200
            Re: Homebuilt NAS Advice Keith Bainbridge <ke1th3216@gmail.com> - 2020-08-02 04:50 +0200
              Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-02 06:10 +0200
                Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-02 21:30 +0200
            Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-02 22:10 +0200
          Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-03 06:10 +0200
            Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-03 17:30 +0200
              Re: Homebuilt NAS Advice Dan Ritter <dsr@randomstring.org> - 2020-08-03 17:40 +0200
              Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-03 18:40 +0200
                Re: Homebuilt NAS Advice <tomas@tuxteam.de> - 2020-08-03 19:10 +0200
                Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-03 22:00 +0200
                Re: Homebuilt NAS Advice <tomas@tuxteam.de> - 2020-08-03 22:20 +0200
                Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-03 23:50 +0200
                Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-04 00:50 +0200
            Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-04 00:40 +0200
    Re: Homebuilt NAS Advice Anssi Saari <as@sci.fi> - 2020-07-30 10:00 +0200
      Re: Homebuilt NAS Advice rhkramer@gmail.com - 2020-07-30 13:20 +0200
  Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-07-30 01:50 +0200
    Re: Homebuilt NAS Advice Patrick Bartek <nemommxiv@gmail.com> - 2020-07-30 23:40 +0200
    Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-08 01:30 +0200
      Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-08 03:30 +0200
        Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-09 14:40 +0200
  Re: Homebuilt NAS Advice Peter Ehlert <peter@sdi-baja.com> - 2020-07-30 19:00 +0200
    Re: Homebuilt NAS Advice Charles Curley <charlescurley@charlescurley.com> - 2020-07-30 21:50 +0200
      Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-07-31 13:10 +0200
    Re: Homebuilt NAS Advice Patrick Bartek <nemommxiv@gmail.com> - 2020-07-30 23:30 +0200
      Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-02 01:50 +0200
        Re: Homebuilt NAS Advice Patrick Bartek <nemommxiv@gmail.com> - 2020-08-02 17:50 +0200
  Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-03 17:20 +0200
    Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-04 00:10 +0200
      Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-04 01:20 +0200
        Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-04 10:40 +0200
          Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-04 11:30 +0200
            Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-04 16:00 +0200
              Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-04 16:30 +0200
              Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-05 05:50 +0200
                Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-05 16:20 +0200
                Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-05 17:00 +0200
                Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-05 17:40 +0200
                Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 12:10 +0200
                Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-06 16:10 +0200
                Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 17:10 +0200
                Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-06 18:10 +0200
                Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-07 18:30 +0200
                Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-07 19:40 +0200
                Re: Homebuilt NAS Advice rhkramer@gmail.com - 2020-08-08 00:10 +0200
                Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-09 04:30 +0200
                Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-09 11:10 +0200
                Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-09 17:00 +0200
                Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-09 18:30 +0200
                Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-09 19:20 +0200
                Re: Homebuilt NAS Advice rhkramer@gmail.com - 2020-08-09 23:50 +0200
                Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-10 00:00 +0200
                Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-10 00:30 +0200
                Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-10 00:50 +0200
                Re: Homebuilt NAS Advice rhkramer@gmail.com - 2020-08-10 00:30 +0200
                Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-10 00:30 +0200
                Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-09 18:40 +0200
                Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-09 23:00 +0200
                Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-06 18:20 +0200
                Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-07 05:30 +0200
                Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-07 18:30 +0200
                Re: Homebuilt NAS Advice rhkramer@gmail.com - 2020-08-06 17:20 +0200
                Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-06 18:20 +0200
                Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-06 18:20 +0200
        Re: Homebuilt NAS Advice Andrei POPESCU <andreimpopescu@gmail.com> - 2020-08-04 17:20 +0200
          Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-05 06:20 +0200
        Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-05 04:30 +0200
          Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-05 09:20 +0200
            Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-05 11:00 +0200
              Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-05 21:00 +0200
                Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 03:50 +0200
                Re: Homebuilt NAS Advice Kushal Kumaran <kushal@locationd.net> - 2020-08-06 06:20 +0200
                Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-06 18:20 +0200
                Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-07 04:10 +0200
                Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-07 05:50 +0200
            Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-05 12:30 +0200
      Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-05 12:30 +0200
        Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-05 21:10 +0200
          Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-05 23:30 +0200
            Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-06 00:30 +0200
              Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 01:20 +0200
                Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-08 09:10 +0200
              Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-06 04:30 +0200
                Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 04:50 +0200
                Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-06 06:20 +0200
                Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 07:10 +0200
                Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 07:30 +0200
                Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-06 18:10 +0200
                Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-06 18:30 +0200
                Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-07 03:00 +0200
                Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-07 03:20 +0200
                Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-07 05:00 +0200
                Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-07 05:40 +0200
                Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-07 08:00 +0200
                Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-07 08:50 +0200
                Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-07 09:10 +0200
                Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-07 17:50 +0200
                Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-08 01:40 +0200
                Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-08 04:50 +0200
                Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-08 22:30 +0200
                Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-09 00:20 +0200
                Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-09 01:30 +0200
                Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-09 22:40 +0200
                Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-10 00:30 +0200
                Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-10 01:30 +0200
    Re: Homebuilt NAS Advice Patrick Bartek <nemommxiv@gmail.com> - 2020-08-04 06:30 +0200

csiph-web