Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.debian.user > #225505 > unrolled thread
| Started by | Patrick Bartek <patrick.bartek@gmail.com> |
|---|---|
| First post | 2020-07-29 22:00 +0200 |
| Last post | 2020-08-04 06:30 +0200 |
| Articles | 20 on this page of 114 — 18 participants |
Back to article view | Back to linux.debian.user
Homebuilt NAS Advice Patrick Bartek <patrick.bartek@gmail.com> - 2020-07-29 22:00 +0200
Re: Homebuilt NAS Advice Dan Ritter <dsr@randomstring.org> - 2020-07-29 22:20 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-07-29 22:30 +0200
Re: Homebuilt NAS Advice Dan Ritter <dsr@randomstring.org> - 2020-07-29 23:40 +0200
Re: Homebuilt NAS Advice Patrick Bartek <patrick.bartek@gmail.com> - 2020-07-30 02:50 +0200
Re: Homebuilt NAS Advice Tom Dial <tddial@comcast.net> - 2020-07-30 22:50 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-02 01:40 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-02 03:10 +0200
Re: Homebuilt NAS Advice Keith Bainbridge <ke1th3216@gmail.com> - 2020-08-02 04:50 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-02 06:10 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-02 21:30 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-02 22:10 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-03 06:10 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-03 17:30 +0200
Re: Homebuilt NAS Advice Dan Ritter <dsr@randomstring.org> - 2020-08-03 17:40 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-03 18:40 +0200
Re: Homebuilt NAS Advice <tomas@tuxteam.de> - 2020-08-03 19:10 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-03 22:00 +0200
Re: Homebuilt NAS Advice <tomas@tuxteam.de> - 2020-08-03 22:20 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-03 23:50 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-04 00:50 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-04 00:40 +0200
Re: Homebuilt NAS Advice Anssi Saari <as@sci.fi> - 2020-07-30 10:00 +0200
Re: Homebuilt NAS Advice rhkramer@gmail.com - 2020-07-30 13:20 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-07-30 01:50 +0200
Re: Homebuilt NAS Advice Patrick Bartek <nemommxiv@gmail.com> - 2020-07-30 23:40 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-08 01:30 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-08 03:30 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-09 14:40 +0200
Re: Homebuilt NAS Advice Peter Ehlert <peter@sdi-baja.com> - 2020-07-30 19:00 +0200
Re: Homebuilt NAS Advice Charles Curley <charlescurley@charlescurley.com> - 2020-07-30 21:50 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-07-31 13:10 +0200
Re: Homebuilt NAS Advice Patrick Bartek <nemommxiv@gmail.com> - 2020-07-30 23:30 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-02 01:50 +0200
Re: Homebuilt NAS Advice Patrick Bartek <nemommxiv@gmail.com> - 2020-08-02 17:50 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-03 17:20 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-04 00:10 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-04 01:20 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-04 10:40 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-04 11:30 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-04 16:00 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-04 16:30 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-05 05:50 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-05 16:20 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-05 17:00 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-05 17:40 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 12:10 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-06 16:10 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 17:10 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-06 18:10 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-07 18:30 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-07 19:40 +0200
Re: Homebuilt NAS Advice rhkramer@gmail.com - 2020-08-08 00:10 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-09 04:30 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-09 11:10 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-09 17:00 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-09 18:30 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-09 19:20 +0200
Re: Homebuilt NAS Advice rhkramer@gmail.com - 2020-08-09 23:50 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-10 00:00 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-10 00:30 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-10 00:50 +0200
Re: Homebuilt NAS Advice rhkramer@gmail.com - 2020-08-10 00:30 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-10 00:30 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-09 18:40 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-09 23:00 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-06 18:20 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-07 05:30 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-07 18:30 +0200
Re: Homebuilt NAS Advice rhkramer@gmail.com - 2020-08-06 17:20 +0200
Re: Homebuilt NAS Advice Celejar <celejar@gmail.com> - 2020-08-06 18:20 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-06 18:20 +0200
Re: Homebuilt NAS Advice Andrei POPESCU <andreimpopescu@gmail.com> - 2020-08-04 17:20 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-05 06:20 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-05 04:30 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-05 09:20 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-05 11:00 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-05 21:00 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 03:50 +0200
Re: Homebuilt NAS Advice Kushal Kumaran <kushal@locationd.net> - 2020-08-06 06:20 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-06 18:20 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-07 04:10 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-07 05:50 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-05 12:30 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-05 12:30 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-05 21:10 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-05 23:30 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-06 00:30 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 01:20 +0200
Re: Homebuilt NAS Advice Jonathan Dowland <jon+debian-user@dow.land> - 2020-08-08 09:10 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-06 04:30 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 04:50 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-06 06:20 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 07:10 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-06 07:30 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-06 18:10 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-06 18:30 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-07 03:00 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-07 03:20 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-07 05:00 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-07 05:40 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-07 08:00 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-07 08:50 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-07 09:10 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-07 17:50 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-08 01:40 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-08 04:50 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-08 22:30 +0200
Re: Homebuilt NAS Advice Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-09 00:20 +0200
Re: Homebuilt NAS Advice deloptes <deloptes@gmail.com> - 2020-08-09 01:30 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-09 22:40 +0200
Re: Homebuilt NAS Advice David Christensen <dpchrist@holgerdanske.com> - 2020-08-10 00:30 +0200
Re: Homebuilt NAS Advice Leslie Rhorer <lesrhorer@att.net> - 2020-08-10 01:30 +0200
Re: Homebuilt NAS Advice Patrick Bartek <nemommxiv@gmail.com> - 2020-08-04 06:30 +0200
Page 5 of 6 — ← Prev page 1 2 3 4 [5] 6 Next page →
| From | deloptes <deloptes@gmail.com> |
|---|---|
| Date | 2020-08-06 18:20 +0200 |
| Message-ID | <AAQMx-4gl-7@gated-at.bofh.it> |
| In reply to | #225919 |
Leslie Rhorer wrote: > A few copies of what size? The backup server is an exact mirror of the > main server, plus several T of additional files I don't need on the main > server. The question is how much back in time you can go. If you have just a mirror - it is one copy, so you can go one step back only. I don't argue about your approach, it is your design, your servers, but in your case I would use definitely deduplication. If you have a problem with data transfer - use fiber, by that volume of data 10G is a must.
[toc] | [prev] | [next] | [standalone]
| From | Leslie Rhorer <lesrhorer@att.net> |
|---|---|
| Date | 2020-08-07 04:10 +0200 |
| Message-ID | <AAZZv-1pc-5@gated-at.bofh.it> |
| In reply to | #225944 |
On 8/6/2020 11:15 AM, deloptes wrote: > Leslie Rhorer wrote: > >> A few copies of what size? ??The backup server is an exact mirror of the >> main server, plus several T of additional files I don't need on the main >> server. > > The question is how much back in time you can go. If you have just a > mirror - it is one copy, so you can go one step back only. Which is why I have offline archival backups in addition to the mirror. Since this is primarily a media server, more than 21T of the files will never change. More than 1/2 of the files are at least 5 years old: root@RAID-Server:/# find /RAID/ | wc -l 382489 root@RAID-Server:/# find /RAID/ -mtime +365 | wc -l 352044 root@RAID-Server:/# find /RAID/ -mtime +730 | wc -l 286695 root@RAID-Server:/# find /RAID/ -mtime +1825 | wc -l 244813 root@RAID-Server:/# find /RAID/ -mtime +3650 | wc -l 22954 Almost all the files older than 1 year are new, not modified. Note also that while there are 382489 files on the array, about 3000 are considerably larger than 1G, representing more than 70% of the space on the drive. Not one of these files are duplicates at any level. The majority of the remaining files (275729) are less than 10K each. Even if every one were a duplicate, that only represents less than 1.5G of uncompressed storage. The vast majority of those never change, either. In reality, the savings would be zilch. > I don't argue about your approach, it is your design, your servers, but in > your case I would use definitely deduplication. And save what? About 0.002%. There are maybe 100 duplicate files larger than 50K on my server, excluding hard links and temporary files. The hard links are handled expeditiously by DAR. Temporary files don't get backed up. > If you have a problem with data transfer - use fiber, by that volume of data > 10G is a must. I do. The servers have 10G optical links between them. A full backup to the RAID 6 array takes several days. A full backup to single drives takes 2 weeks, because single drives are limited to about 800Mbps, while the array can gulp down nerly 4Gbps. Nightly backups (via rsync) take a few minutes, at most.
[toc] | [prev] | [next] | [standalone]
| From | Stefan Monnier <monnier@iro.umontreal.ca> |
|---|---|
| Date | 2020-08-07 05:50 +0200 |
| Message-ID | <AB1yi-2et-3@gated-at.bofh.it> |
| In reply to | #225961 |
>> I don't argue about your approach, it is your design, your servers, but in
>> your case I would use definitely deduplication.
> And save what? About 0.002%.
I don't know if it's important for your use case (probably not), but in
backups, deduplication is (mostly) not about finding common data between
the different files you backup, but to find common data between
different backups, thus allowing the backup tool to get rid of the
distinction between a full backup and an incremental backup: all backups
are "like full" but cost "like incremental".
So deduplication doesn't really save disk space, instead it saves
you from having to worry about tradeoffs between making full backups and
making incremental backups. AFAICT you already have made those
decisions long ago, so saving you the worry is probably not of much
value to you.
Stefan
[toc] | [prev] | [next] | [standalone]
| From | Jonathan Dowland <jon+debian-user@dow.land> |
|---|---|
| Date | 2020-08-05 12:30 +0200 |
| Message-ID | <AAoQh-42Y-1@gated-at.bofh.it> |
| In reply to | #225872 |
On Wed, Aug 05, 2020 at 09:16:15AM +0200, deloptes wrote:
>Leslie Rhorer wrote:
>
>> DAR allows for not only incremental backups, but also incremental
>> deletions. I find it extremely useful, and allows for as much space
>> saving on the live system as one likes. I suggest you check it out.
>
>someone should ask them to implement deduplication, because nowdays it is
>impossible to do meaningful backups without.
I've survived without just fine until now, although it is a feature I
want. I'm migrating some things from rdiff-backup to borg at the moment,
and one of the most impressive results I've got so far is
filesystem size: 228G
compressed size: 219G
deduplicated size: 59G
I'm sure the rest of my filesystems will not de-duplicate quite so much.
If I was to try and convert my rdiff-backup increments over to borg, I'd
probably see some savings (moving large sets of files around results in
enormous increments in rdiff-backup) but I don't think I can be bothered
to do that.
--
Please do not CC me, I am subscribed to the list.
👱🏻 Jonathan Dowland
✎ jmtd@debian.org
🔗 https://jmtd.net
[toc] | [prev] | [next] | [standalone]
| From | Jonathan Dowland <jon+debian-user@dow.land> |
|---|---|
| Date | 2020-08-05 12:30 +0200 |
| Message-ID | <AAoQh-42Y-11@gated-at.bofh.it> |
| In reply to | #225794 |
On Mon, Aug 03, 2020 at 04:59:22PM -0500, Leslie Rhorer wrote: >I concur. One area in wich I disagree is your decision not to employ >RAID. As you very properly comment in the linked page, RAID is not a >backup. They are *VERY* different things. Therefore both,in my >estimation, are absolutely required. Reading your message I'm fairly sure you agree that I understand the difference between RAID and backup, but I'm not sure whether you mean that for *your* needs, RAIDS is essential (fine, no objection there), or that given your understanding of *my* needs you disagree that *I* don't need RAID. In brief my rationale is: without RAID, when a drive failure occurs I lose access to my NAS until I've replaced the drive and restored the files. I judge that the inconvenience of system downtime is outweighed by the increased cost (upfront and running), complexity, and raised failure frequency of deploying RAID. Everyone's mileage varies, of course. -- Please do not CC me, I am subscribed to the list. 👱🏻 Jonathan Dowland ✎ jmtd@debian.org 🔗 https://jmtd.net
[toc] | [prev] | [next] | [standalone]
| From | deloptes <deloptes@gmail.com> |
|---|---|
| Date | 2020-08-05 21:10 +0200 |
| Message-ID | <AAwXv-zk-1@gated-at.bofh.it> |
| In reply to | #225879 |
Jonathan Dowland wrote: > In brief my rationale is: without RAID, when a drive failure occurs I > lose access to my NAS until I've replaced the drive and restored the > files. I judge that the inconvenience of system downtime is outweighed > by the increased cost (upfront and running), complexity, and raised > failure frequency of deploying RAID. Everyone's mileage varies, of > course. Sorry to answer a question that was not addressed to me, but I was just reading this topic as I had experience in the past few years with this topic in a project I worked on. IMO you should do RAID and probably consider doing backups. RAID is the most essential part, because you can replace a failed drive without too much effort. Most suitable are those cases with at least 4 drive bays as you can use RAID5 or similar. It is all about probability of failure. The probability of a disk to fail is avg. but it happens. The probability that 2 disks fail at the same time ... not really except you experience some kind of electric shock or you have a really bad luck. For this it is recommended to use backup. Some people go even further burning backups on disks (like blu-rays) because they fear EMP. My experience is that I replaced many failed disks in the raids in the past 15+y, but never had to recover from backup because of disk failure. I have used the back to restore data either because I accidentally deleted something or because I created a test copy of one of the systems. It all depends on how much risk you want to take. The truth is somewhere in the middle. For me personal information or business related files are important and go into backup and multimedia does not, but everything is on a RAID.
[toc] | [prev] | [next] | [standalone]
| From | Jonathan Dowland <jon+debian-user@dow.land> |
|---|---|
| Date | 2020-08-05 23:30 +0200 |
| Message-ID | <AAz8Z-1Oz-1@gated-at.bofh.it> |
| In reply to | #225910 |
On Wed, Aug 05, 2020 at 09:09:07PM +0200, deloptes wrote: >Sorry to answer a question that was not addressed to me, but I was just >reading this topic as I had experience in the past few years with this >topic in a project I worked on. No problem — it's a public list :) >IMO you should do RAID and probably consider doing backups. RAID is the most >essential part For the majority of use-cases, I really disagree that RAID is ever more essential than backup. I can cook up some scenarios where this isn't true, but they are not common ones. >It is all about probability of failure. The probability of a disk to fail is >avg. but it happens. The probability that 2 disks fail at the same time ... >not really except you experience some kind of electric shock or you have a >really bad luck. For this it is recommended to use backup. Some people go >even further burning backups on disks (like blu-rays) because they fear >EMP. Backups protect you from more than disk failure. You could lose all your data with perfectly fine disks thanks to software failure, user error, or malicious attack. >I have used the back to restore data either because I accidentally deleted >something That's definitely one thing I use backups for, as well as to protect against disk failure. -- Please do not CC me, I am subscribed to the list. 👱🏻 Jonathan Dowland ✎ jmtd@debian.org 🔗 https://jmtd.net
[toc] | [prev] | [next] | [standalone]
| From | deloptes <deloptes@gmail.com> |
|---|---|
| Date | 2020-08-06 00:30 +0200 |
| Message-ID | <AAA54-2nB-1@gated-at.bofh.it> |
| In reply to | #225914 |
Jonathan Dowland wrote: > For the majority of use-cases, I really disagree that RAID is ever more > essential than backup. I can cook up some scenarios where this isn't > true, but they are not common ones. And I disagree with you, because if you do not have RAID, you loose the data instantly and in best case your backup is a few hours old. Plus you have the effort to restore probably the last monthly, then the last weekly and then few incrementals - compared to replacing just one disk and letting it resync. In general I agree with you that both are essential, but if someone asks me I definitely recommend RAID and in addition the backup. This means if you want to be able to restore let's say yesterdays version of a file. True backup is very costly if you want to do it the proper way. For example you shouldn't have the backup in the same room - cause in case of fire ... you know what I mean. I wouldn't spend that much money on it. As the question was about NAS ... it looks like you need two - one for the data and one for the backup So I appreciate and share your opinion, but here it is not an enterprise consulting company I guess ... although with 40+TB ... :D 22 of them in use ... I doubt that avg. user has high demand on backup such large volumes. AS shown in the example with deduplication you can use even an external disk as backup device to keep large amount of backups. All I want to say is setting priorities and calculating the risk makes more sense to me.
[toc] | [prev] | [next] | [standalone]
| From | Leslie Rhorer <lesrhorer@att.net> |
|---|---|
| Date | 2020-08-06 01:20 +0200 |
| Message-ID | <AAARr-2Tq-1@gated-at.bofh.it> |
| In reply to | #225915 |
On 8/5/2020 5:25 PM, deloptes wrote: > Jonathan Dowland wrote: > >> For the majority of use-cases, I really disagree that RAID is ever more >> essential than backup. I can cook up some scenarios where this isn't >> true, but they are not common ones. > > And I disagree with you, So do I. Both are essential, and without both, sooner or later there is going to be a very significant problem. > because if you do not have RAID, you loose the data > instantly and in best case your backup is a few hours old. Not only that, the system is simply unavailable until the file are restored. This can take hours or even days. > Plus you have > the effort to restore probably the last monthly, then the last weekly and > then few incrementals - compared to replacing just one disk and letting it > resync. If that. On my large arrays, I keep a hot spare disk online, so the resynch starts the moment a disk fails. > In general I agree with you that both are essential, but if someone asks me > I definitely recommend RAID and in addition the backup. > This means if you want to be able to restore let's say yesterdays version of > a file. Exactly. RAID does not protect against Oops. Backups do, at least toa reasonable extent. > True backup is very costly if you want to do it the proper way. Well, there are solutions that are both economical and effective. They tend to require more work on the part of the admin, though. I mean, it would be really, really nice to buy a Quantum tape drive for $10K and change, I definitely cannot afford it. I have a ton of older hard drives pulled from the arrays during upgrades sitting around doing nothing, however. Their use as backup media costs nothing.
[toc] | [prev] | [next] | [standalone]
| From | Jonathan Dowland <jon+debian-user@dow.land> |
|---|---|
| Date | 2020-08-08 09:10 +0200 |
| Message-ID | <ABr9n-18R-13@gated-at.bofh.it> |
| In reply to | #225916 |
On Wed, Aug 05, 2020 at 06:18:16PM -0500, Leslie Rhorer wrote: > Not only that, the system is simply unavailable until the file are >restored. This can take hours or even days. Much more likely weeks in my case, but that's fine. This NAS has been running for 5 years with one interruption to service and that was a mainboard failure. So I would expect the downtime from a dead disk to occur once every 5 years or less, which also correlates with my experiences with all my other personal systems (except my first attempt of a NAS, which had more frequent disk failures which I think were caused by thermal problems. I did use RAID-1 in that system.) In other words, I understand the risks and impacts for RAID or not RAID in my personal NAS, and the inconvenience of RAID far outweighs the inconvenience of downtime when one (or both) of the 2 disks fail, since downtime is not a big deal for this system for me and the frequency I expect to face it is very low. I would make, and have made, different judgement calls for business systems in my professional life, of course. And my VPS is doubtless RAID-backed by the ISP that supplies it; downtime for that would be much more of an inconvenience for me than my NAS. -- Please do not CC me, I am subscribed to the list. 👱🏻 Jonathan Dowland ✎ jmtd@debian.org 🔗 https://jmtd.net
[toc] | [prev] | [next] | [standalone]
| From | Stefan Monnier <monnier@iro.umontreal.ca> |
|---|---|
| Date | 2020-08-06 04:30 +0200 |
| Message-ID | <AADPk-4GO-5@gated-at.bofh.it> |
| In reply to | #225915 |
> And I disagree with you, because if you do not have RAID, you loose the data
> instantly and in best case your backup is a few hours old. Plus you have
> the effort to restore probably the last monthly, then the last weekly and
> then few incrementals - compared to replacing just one disk and letting it
> resync.
Again, this heavily depends on the specifics.
In my case, RAID would have been helpful once over the last 20 years
(and FWIW, I'm still using that disk that failed (after having created
a partition that covers about 4% of the disk and which encloses all the
sectors that can't be read/written any more ;-)
So in my experience it's rarely useful. That doesn't mean it's not
worth it, tho. For that you need to look even more at the details of
the actual upsides and downsides:
- I work on 3 machines "all the time" (one desktop at home, one at the
office, plus a laptop), so if one of those machines dies I can just
keep working with one of the other ones (and I have an extra laptop
that I keep as hot spare). W.r.t my servers, I similarly have 2 (one
at the office and one at work, both of them keeping a copy of my
backups), so if one dies, well, I just use the other until I can fix
that one (and FWIW one of them tends to crash fairly often lately, for
software reasons).
- Regarding backups: I wouldn't restore a full install from a backup,
instead I'd clone one of my other machines. But w.r.t
full/incremental, I use `bup` which doesn't distinguish between those.
- More importantly, most of the things I work on are "backed up" several
times a day on a remote server by virtue of being pushed to
a Git repository or being stored in an IMAP server (which *does* use
RAID, but that machine is not under my control). So in the unlikely
(but definitely possible) even that my machine dies, I won't lose very
much work anyway.
- RAID would require extra hardware in my machines, for some of them
that would be a non-trivial constraint (e.g. my BananaPi servers and
my laptops).
RAID makes a lot of sense in many cases, but I don't think it makes
sense in mine.
Stefan
[toc] | [prev] | [next] | [standalone]
| From | Leslie Rhorer <lesrhorer@att.net> |
|---|---|
| Date | 2020-08-06 04:50 +0200 |
| Message-ID | <AAE8F-4NO-1@gated-at.bofh.it> |
| In reply to | #225920 |
> - RAID would require extra hardware in my machines, for some of them > that would be a non-trivial constraint (e.g. my BananaPi servers and > my laptops). How do you figure? Adding an external USB drive enclosure to a laptop or a Banana Pi is pretty trivial. RAID does not require any hardware other than the drives. A four drive USB enclosure can be had for under $100.
[toc] | [prev] | [next] | [standalone]
| From | Stefan Monnier <monnier@iro.umontreal.ca> |
|---|---|
| Date | 2020-08-06 06:20 +0200 |
| Message-ID | <AAFxL-5Lr-3@gated-at.bofh.it> |
| In reply to | #225921 |
>> - RAID would require extra hardware in my machines, for some of them
>> that would be a non-trivial constraint (e.g. my BananaPi servers and
>> my laptops).
> How do you figure? Adding an external USB drive enclosure to
> a laptop or a Banana Pi is pretty trivial.
Are you seriously suggesting a RAID setup where one half is the
SATA-connected drive inside the laptop and the other is an external
USB-connected drive?
Sounds like a recipe for a world of pain for all the times that the
external drive gets disconnected (not to mention the performance impact
of having one half going over the slow USB2 connection, defeating the
benefits of the internal SSD).
For the BananaPi, the suggestion is marginally less problematic but
still: a non-trivial constraint with significant immediate downsides.
RAID is basically an insurance. Taking an insurance makes a lot of
sense when it's important to spread the cost of the impact of an
"event", but in my case the cost of such an event would be fairly low
anyway (not to mention that RAID doesn't prevent me from losing work
when the OS or my editor crashes, or when I make a mistake).
Stefan
[toc] | [prev] | [next] | [standalone]
| From | Leslie Rhorer <lesrhorer@att.net> |
|---|---|
| Date | 2020-08-06 07:10 +0200 |
| Message-ID | <AAGka-6kL-7@gated-at.bofh.it> |
| In reply to | #225924 |
On 8/5/2020 11:13 PM, Stefan Monnier wrote: >>> - RAID would require extra hardware in my machines, for some of them >>> that would be a non-trivial constraint (e.g. my BananaPi servers and >>> my laptops). >> How do you figure? Adding an external USB drive enclosure to >> a laptop or a Banana Pi is pretty trivial. > > Are you seriously suggesting a RAID setup where one half is the > SATA-connected drive inside the laptop and the other is an external > USB-connected drive? It's not what I would do, probably, but it is possible. In fact, a RAID 1 mirror with write-mostly enabled on the external drive is not a terrible idea. For better performance, more space, and higher throughput, I would probably create a RAID 4 or RAID 6 array from the external enclosure and use it as the data repository. > Sounds like a recipe for a world of pain for all the times that the > external drive gets disconnected (not to mention the performance impact > of having one half going over the slow USB2 connection, defeating the > benefits of the internal SSD). Not on a RAID 1 mirror with write-mostly enabled on the external drive. > For the BananaPi, the suggestion is marginally less problematic but > still: a non-trivial constraint with significant immediate downsides. Such as? > RAID is basically an insurance. Not entirely. A RAID 5 or RAID 6 array is far, far faster than a single hard drive. It is also much larger than a single hard drive, sometimes at less expense than a single large hard drive. It is also portable from one system to another. Unplug the array from the laptop, plug it in to the Banana Pi, and presto! The array is now attached to the Pi. It's really not any different logically than an external drive, except it is faster, larger, and more robust. > Taking an insurance makes a lot of > sense when it's important to spread the cost of the impact of an > "event" Or eliminate it entirely. > anyway (not to mention that RAID doesn't prevent me from losing work > when the OS or my editor crashes Uh, yeah, it can. It definitely can be used to eliminate data loss when the OS crashes, and with a little magic, even the loss of OS related informatin (like /home) can be eliminated. The editor is a little different matter, although the result of it's failure can also be limited. > or when I make a mistake). That is another matter. Indeed, it is probably the most likely reason a need for a backup solution exists.
[toc] | [prev] | [next] | [standalone]
| From | Leslie Rhorer <lesrhorer@att.net> |
|---|---|
| Date | 2020-08-06 07:30 +0200 |
| Message-ID | <AAGDv-6rl-3@gated-at.bofh.it> |
| In reply to | #225926 |
On 8/6/2020 12:06 AM, Leslie Rhorer wrote: > For better performance, more space, and higher throughput, I would > probably create a RAID 4 or RAID 6 array from the external enclosure and > use it as the data repository. Sorry, that was a typo. That should be RAID 5 or RAID 6.
[toc] | [prev] | [next] | [standalone]
| From | Stefan Monnier <monnier@iro.umontreal.ca> |
|---|---|
| Date | 2020-08-06 18:10 +0200 |
| Message-ID | <AAQCS-4cZ-3@gated-at.bofh.it> |
| In reply to | #225926 |
> For better performance, more space, and higher throughput, I would
> probably create a RAID 4 or RAID 6 array from the external enclosure
> and use it as the data repository.
And you suggest I put a 4-drive enclosure in my backpack next to my laptop?
Seriously?
>> For the BananaPi, the suggestion is marginally less problematic but
>> still: a non-trivial constraint with significant immediate downsides.
> Such as?
Extra space taken, extra power used 24/7 (which in turn requires an
extra plug because the poor BananaPi can't provide all that power),
extra costs (the single drive I use was itself more than 3/4 the price
of the whole system, so adding a second drive (complete with external
power supply and enclosure) would double the price of the system),
Extra failures (more hardware => more failures), ...
>> RAID is basically an insurance.
> Not entirely. A RAID 5 or RAID 6 array is far, far faster than
> a single hard drive.
Right, RAID-over-USB is of course going to blow my SSD-over-SATA out of
the water by a wide margin (not!).
> It is also much larger than a single hard drive, sometimes at
> less expense than a single large hard drive.
That's great if you happen to be in that spot, but that's not my case.
> It is also portable from one system to another. Unplug the array
> from the laptop, plug it in to the Banana Pi, and presto! The array
> is now attached to the Pi.
Wonderful. But then it's not a RAID shared with the internal drive
any more. So it won't protect my root partition. And if I keep my home
partition in it, the "presto" comes with the footnote "after you logout
and log back in" (fun!), but if I don't keep my home partition on it,
then my home partition is again not protects by RAID at which point I'm
starting to wonder what I would put on that RAID.
> It's really not any different logically than an external drive,
> except it is faster, larger, and more robust.
It's no different, indeed, except a bit more expensive and bigger.
But more importantly: there's a reason why I'm not using an external
drive at all in the first place.
>> Taking an insurance makes a lot of sense when it's important to
>> spread the cost of the impact of an "event"
> Or eliminate it entirely.
If a disk dies in the RAID, you still have to replace it and the
performance is affected for a while. The cost is never
completely eliminated.
>> anyway (not to mention that RAID doesn't prevent me from losing work
>> when the OS or my editor crashes
> Uh, yeah, it can. It definitely can be used to eliminate data loss
> when the OS crashes,
Of course it can't: when the OS crashes, the unsaved work in your editor
is lost. And as for the work that was saved to disk, I wonder in what
way RAID can improve the situation compared to a single disk (in both
cases, I expect that the data will not be lost).
> That is another matter. Indeed, it is probably the most likely
> reason a need for a backup solution exists.
For my use-case, RAID would cost a fair bit of money and inconvenience,
and the benefit would be rare and minor.
I suspect your "world" is one where you have U1 blades and things like
that and/or where the disk's content is not 100% your own (i.e. other
people depend on it), so for your use case RAID is probably
a no-brainer. For me, the choice is similarly blindingly obvious but
diametrically opposed because my use case is radically different.
Stefan
[toc] | [prev] | [next] | [standalone]
| From | deloptes <deloptes@gmail.com> |
|---|---|
| Date | 2020-08-06 18:30 +0200 |
| Message-ID | <AAQWe-4jK-23@gated-at.bofh.it> |
| In reply to | #225942 |
Stefan Monnier wrote: > And you suggest I put a 4-drive enclosure in my backpack next to my > laptop? Seriously? :D +1 Stefan, I have the same situation - on the Laptop only backup works :D Basically you can forget RAID in USB2 context. I have not tried USB3, but I heard that even the spead is compareable the issue is with the controllers. This is BS idea - imagine you have a meeting and you make a presentation packing out 2-3 USB disks just to have a RAID :D This was funny!
[toc] | [prev] | [next] | [standalone]
| From | Leslie Rhorer <lesrhorer@att.net> |
|---|---|
| Date | 2020-08-07 03:00 +0200 |
| Message-ID | <AAYTL-yu-1@gated-at.bofh.it> |
| In reply to | #225948 |
On 8/6/2020 11:25 AM, deloptes wrote: > Stefan Monnier wrote: > >> And you suggest I put a 4-drive enclosure in my backpack next to my >> laptop? Seriously? Why not? > Stefan, I have the same situation - on the Laptop only backup works :D > > Basically you can forget RAID in USB2 context. I have not tried USB3, but I > heard that even the spead is compareable the issue is with the controllers. > > This is BS idea - imagine you have a meeting and you make a presentation > packing out 2-3 USB disks just to have a RAID :D First of all, it is one small chassis, not 2 -3 separate disks. Secondly, so what? You act as if a small box sitting next to a laptop is some sort of elephant sitting in your lap. It is not difficult at all to imagine. What's difficult to imagine is using a laptop for a business presentation. I have done thousands of business presentations. Not one on a laptop. > This was funny! >
[toc] | [prev] | [next] | [standalone]
| From | deloptes <deloptes@gmail.com> |
|---|---|
| Date | 2020-08-07 03:20 +0200 |
| Message-ID | <AAZd7-Uf-3@gated-at.bofh.it> |
| In reply to | #225956 |
Leslie Rhorer wrote: > First of all, it is one small chassis, not 2 -3 separate disks. > Secondly, so what? You act as if a small box sitting next to a laptop > is some sort of elephant sitting in your lap. It is not difficult at > all to imagine. What's difficult to imagine is using a laptop for a > business presentation. I have done thousands of business presentations. > Not one on a laptop. I don't know from which universe you came here, honestly :D, sorry, I don't mean to hurt you ... it is just (lets say) some untraditional thinking. You must know that this is a public list that stays in the memory of the Earths internet for long time. The questions here should have suitable answers. This is the only motivation I write to this thread. Not that someone reads this and thinks those debian users are all freaks and if you see someone walking around with a pack of external drives, must have read this thread. regards
[toc] | [prev] | [next] | [standalone]
| From | Leslie Rhorer <lesrhorer@att.net> |
|---|---|
| Date | 2020-08-07 05:00 +0200 |
| Message-ID | <AB0LT-1Jd-1@gated-at.bofh.it> |
| In reply to | #225959 |
On 8/6/2020 8:12 PM, deloptes wrote: > I don't know from which universe you came here, honestly :D, sorry, I don't > mean to hurt you ... You would find that quite impossible. I would never be upset over a simple disagreement. Or a complex one, for that matter. Any forum without disagreement is useless. That very last thing I ever want around me is a bunch of "yes" men. > it is just (lets say) some untraditional thinking. Your point here totally escapes me. I started working with computers in 1974. Computers, let alone personal computers, were anything but traditional at that point. "Tradition" is nothing but a synonym for stagnation, ineffectiveness, and inefficiency. Anyone who thinks traditionally isn't really thinking. > You must know that this is a public list that stays in the memory of the > Earths internet for long time. It exists in cold storage, not so much in memory. Unless the post is very memorable, only a figurative handful of people will remember it more than a week, if that. Very few people go back and read archived messages in the list. Come on. Be honest. How many messages in this or any other forum can you specifically remember from a year ago? When was the last time you read a post more than a year old? My servers have a permanent and non-volatile memory (thank heavens), but I don't, and neither does the average Debian user. > The questions here should have suitable answers. This is the only motivation > I write to this thread. Not that someone reads this and thinks those debian Personally, I don't walk around with anything - not even a laptop, but an external enclosure such as the following can easily fit in a backpack, a large laptop case, or a camera bag. https://www.amazon.com/gp/product/B0711L68MS/ref=ask_ql_qh_dp_hza It's not even 5' in diameter and barely 6" long. The non-RAID version is $51. The version with built-in RAID is $90. > users are all freaks and if you see someone walking around with a pack > of external drives, must have read this thread. You keep saying "pack of drives", or "bunch of disks". An external enclosure is no larger in volume than many laptops, and much, much smaller than the load of books I carried around in college. The one above is smaller than a box of tissues and not much bigger than the cell phone I carried around for nearly ten years. If I were one to use a laptop - which I most certainly do not - I wouldn't carry the array around, however. It would sit either at home or at work (one and the same, for me) and I would access it through the internet.
[toc] | [prev] | [next] | [standalone]
Page 5 of 6 — ← Prev page 1 2 3 4 [5] 6 Next page →
Back to top | Article view | linux.debian.user
csiph-web