Re: linux raid vs hw raid

Date	2011-04-06 10:03 +0200
From	David Brown <david@westcontrol.removethisbit.com>
Newsgroups	comp.os.linux.misc
Subject	Re: linux raid vs hw raid
References	<fc0t68x5ci.ln2@goaway.wombat.san-francisco.ca.us> <dmft68-2q8.ln1@squidward.dionic.net>
Message-ID	<Xe-dnXd4LLb1gwHQnZ2dnUVZ7sWdnZ2d@lyse.net> (permalink)

Show all headers | View raw

On 06/04/2011 09:01, Tim Watts wrote:
> Keith Keller wrote:
>
>> Hi all,
>>
>> I am attempting to build a snapshot server for a ~15TB fileserver with
>> old fileserver hardware I have on hand.  My initial plan was to use the
>> hardware card in the old fileserver in a RAID50 (the card is old enough
>> that it doesn't support RAID6 natively) using new 2TB enterprise hard
>> drives.  But, as you probably know, these drives are reasonably
>> expensive.  So, since this machine will not be used by end-users very
>> much, I was contemplating using linux software raid instead, exporting
>> desktop-class drives as JBODs and using mdadm to RAID them.
>>
>> The obvious advantage to this is cost: I can save almost 40% of my
>> original estimate by using desktop drives instead, thus fulfilling the
>> original meaning of the I of the RAID acronym.  There are other
>> advantages, as well, including being able to build a RAID6, which I
>> slightly prefer over a RAID50, and having more flexibility later on if I
>> want to move to bigger disks.  (Yes, I have seen the documentation
>> warning against too-large RAID arrays resulting in a failure during a
>> rebuild.)  A tertiary advantage would be that I would learn how to work
>> with linux software RAID, a skill I haven't yet acquired.
>>
>> The disadvantages I can think of are: higher probability of disk
>> failures, resulting in more work for me in swapping out and RMAing
>> failed drives; potential degradation in performance, due both to RAID in
>> software and slower disks; a learning curve for linux RAID; and a
>> configuration less likely to be supported by the hardware RAID vendor.
>
> Hi,
>
> Highly dependant on your server and RAID card of course, but you may find MD
> software raid is quicker.
>
> Even and older server has far more CPU horsepower available compared to a
> mediocre RAID card (and by mediocre, I mean anything costing less than 100's
> pounds.
>

I'd go further than that and say that software raid will be faster 
unless your hardware raid card costs many 1000's of pounds.  Unless you 
are using the sort of raid card that comes with its own backup battery 
for caching, then mdadm raid is going to be faster with a modern 
processor.  Even with such a card, mdadm raid is probably going to be 
faster for raid 5 or raid 6, simply because the host has access to more 
memory for caching stripes.

A key bottleneck to consider is IO throughput, rather than CPU power. 
This is especially true for RAID1 setups - doing the RAID1 on a hardware 
card halves the IO on the host.

However, if the server is old enough, there was a time when commonly 
used hardware raid cards were faster than doing it in software on the 
host.  In particular, if the host is single core, or Intel's old and 
crappy shared bus SMP, then a hardware raid card will be faster.

Not that this matters too much to the OP, of course!

>> My counters to most of the disadvantages would be that performance only
>> has to be decent, not great, on this box; the learning curve shouldn't
>> be too bad; and this configuration shouldn't require support from the
>> hardware RAID vendor anyway.  The disk failures would be the only issue
>> I couldn't counter, except by trying to determine if my labor costs
>> would end up being more than the savings in moving to cheaper disks.
>
> The learning curve is fairly easy with mdadm - furthermore, linux MD is now
> more functionally complete than all but the better end *modern* hardware
> RAID systems. Specifically, some things linux will do that a lot of
> older/cheaper HW RAID won't:
>
> 1) Attempt to rewrite a disck block that has failed to read<- triggers a
> bad block remap on most drives.
>
> 2) If you run the monitor daemon, linux will alert you if stuff goes bad, eg
> failed disk (OK, a crappy HW raid knos this, but can it alert you by email
> or just sit there with a falshing red LED?)
>
> 3) Perform a full sweep and parity verify on demand?
>
> There are more, but those are what I consider most useful.
>

One hint about learning mdadm - with mdadm, you can build your arrays 
from partitions, not just whole disks.  So you can give your disks a 4 
GB partition at the start and use that when testing and learning - it's 
a lot easier to learn when your rebuild times are a couple of minutes, 
rather than most of the day!

One thing to practice is identifying drives - when a drive fails, you 
want to be very sure of which one you should be replacing :-)

>> My questions:
>>
>> 1) Has anyone done this before, and if so, what were the results?  Was
>> performance acceptable in this configuration?  Are there any gotchas to
>> an otherwise workable configuration?
>
> Yep - been running SW raid 5 at home on 1.5TB total for 3 years. I have used
> a lot of mid range RAID controllers too (Chaparrel, Infotrend, ARECA,
> Eurologic)
>

I haven't tried any hardware raid cards seriously, but I've used mdadm 
raid often on servers and desktops.

Personally, I like RAID10 with "far" layout - it gives you greater 
safety than RAID5 or RAID6, and most of the speed of RAID0.  It works 
well with 2 or 3 disks (something that no hardware raid card can do).

>> 2) From what I've read so far, using desktop-class disks with linux
>> software RAID should not be a major problem, unlike using them on a true
>> hardware RAID card.  Is this reasonably accurate?  If not, are there
>> links that describe the difficulties?
>
> Yep - desktop are fine. Enterprise class or "RAID Edition" may be better
> quality and/or quicker. Quicker is usually related to RPM and at least is
> checkable in the specifications. "Well built" is more abstract. I prefer to
> use a mixture of makes in the same server, eg Hitachi, Seagate, Fujitsu, WD)
> - that way, you lessen the risk of the "Maxtor Deathstar" whole buch failing
> at once syndrome.
>

I am not convinced that enterprise class disks really offer much more 
than desktop disks if you have a reasonable environment (not too hot or 
cold, reliable power, etc.).  There will be a difference in the expected 
lifetimes of the drives - but since disk failures are actually fairly 
rare, it won't show in the statistics unless you have hundreds of drives 
or drive them under very heavy load.

>> 3) Suppose that my RAID6 starts out using 12 2TB disks, with three free
>> drive bays (one would be a hot spare).  Later on, I want to seamlessly
>> replace the 2TB disks with 3TB or larger disks.  Can mdadm grow an array
>> like this if, say, I replace one drive, rebuild, and repeat until I've
>> replaced all 12 disks with larger ones?  Or will the new 3TB disks only
>> be used up to 2TB, the size of the original disks?
>
> RAID5/6 need to be spread over identically sized partitions. So you can't
> add a 3TB drive to a 2TB disk based array. You can partition and make a new
> RAID across the 1TB partition. This is where ZFS gets clever, but that's not
> really an option for linux (BTRFS will probably get there one day).
>

You can increase the size of the RAID5/6 devices (whole disks, or 
partitions) if you re-size them all.  So if you replace one 2 TB drive 
with a 3 TB drive and let it rebuild, you can't use more than the first 
2 TB.  But if you continue the process and replace all of the drives, 
you can then "grow" the array to use the new space.

Another option for growth is to use mdadm over partitions, rather than 
whole disks.  Then when you add bigger disks, you have spare space that 
you can make into new partitions, and make another mdadm raid using 
them.  If you are using LVM to organise your real partitions (which I 
highly recommend), then you can add your new raid as a new physical 
partition and extend your working space.

One other thing to think about if you are planning to replace disks, is 
that you are reducing your redundancy while it is happening.  For 
example, if you have a RAID5 array and you pull one drive to replace it 
with a bigger drive, then you have no redundancy during that operation. 
  With RAID6 you have one drive redundancy rather than two.  And like 
all rebuilds, the rebuild for the drive replacement is particularly 
stressful for the rest of the disks in the array - and you are going to 
do the whole operation 12 times in a row.

But the beauty of md raid is its flexibility.  Rather than use twelve 
disks in a RAID6, build twelve RAID1 pairs from a real drive and a 
missing drive.  Then build your RAID6 on top of these "pairs".  The 
result is the same in terms of speed, capacity and redundancy.  But when 
you want to replace a drive with a bigger disk, you do it by adding the 
new drive to one of the pairs and letting the pair "rebuild".  Then you 
remove the old disk from the pair.  You keep the same redundancy over 
the whole array throughout the operation, and the rebuild is done as a 
mirror copy from one disk - the other drives are unaffected.  You can 
happily do the replacement with multiple disks in parallel - as many as 
you have spare drive bays.

(Future plans for md include "hot replace" functionality that will 
effectively automate this, but that's for the future.)

>> Thanks for any advice or pointers you can provide!
>
> One thing, whichever system you go for: set it up and do some speed and
> breakage tests to make sure it all works correctly - pull a disk out live,
> be sure you know how to put the disk back and bring the array back to fault
> tolerant and stuff like that.
>
> It's good fun, enjoy :)
>
> Cheers
>
> Tim
>
>> --keith
>>
>

Back to comp.os.linux.misc | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-05 19:39 -0700
  Re: linux raid vs hw raid Tim Watts <tw@dionic.net> - 2011-04-06 08:01 +0100
    Re: linux raid vs hw raid David Brown <david@westcontrol.removethisbit.com> - 2011-04-06 10:03 +0200
      Re: linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-06 14:00 -0700
        Re: linux raid vs hw raid David Brown <david.brown@removethis.hesbynett.no> - 2011-04-06 23:42 +0200
        Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-08 10:45 +1000
          Re: linux raid vs hw raid David Brown <david@westcontrol.removethisbit.com> - 2011-04-08 11:12 +0200
            Re: linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-08 08:22 -0700
              Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-09 09:51 +1000
                Re: linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-08 17:10 -0700
                Re: linux raid vs hw raid David Brown <david.brown@removethis.hesbynett.no> - 2011-04-09 13:14 +0200
            Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-09 09:47 +1000
              Re: linux raid vs hw raid David Brown <david.brown@removethis.hesbynett.no> - 2011-04-09 13:55 +0200
          Re: linux raid vs hw raid Tris Orendorff <triso@remove-me.cogeco.ca> - 2011-04-12 18:04 +0000
            Re: linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-12 11:34 -0700
              Re: linux raid vs hw raid The Natural Philosopher <tnp@invalid.invalid> - 2011-04-12 21:13 +0100
                Re: linux raid vs hw raid David Brown <david@westcontrol.removethisbit.com> - 2011-04-13 09:45 +0200
                Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-14 13:42 +1000
                Re: linux raid vs hw raid David Brown <david@westcontrol.removethisbit.com> - 2011-04-14 09:15 +0200
                Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-15 08:03 +1000
                Re: linux raid vs hw raid Tim Watts <tw@dionic.net> - 2011-04-15 07:22 +0100
                Re: linux raid vs hw raid David Brown <david@westcontrol.removethisbit.com> - 2011-04-15 09:28 +0200
                Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-19 11:20 +1000
              Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-14 13:38 +1000
                Re: linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-13 21:49 -0700
            Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-14 13:34 +1000
              Re: linux raid vs hw raid Tris Orendorff <triso@remove-me.cogeco.ca> - 2011-04-15 21:59 +0000
                Re: linux raid vs hw raid "Peter J. Holzer" <hjp-usenet2@hjp.at> - 2011-04-16 00:56 +0200
                Re: linux raid vs hw raid The Natural Philosopher <tnp@invalid.invalid> - 2011-04-16 01:32 +0100
  Re: linux raid vs hw raid Tauno Voipio <tauno.voipio@notused.fi.invalid> - 2011-04-08 21:38 +0300
    Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-09 09:53 +1000
  Re: linux raid vs hw raid KR <kristian.rasmussen@broadpark.no.spam.com> - 2011-04-09 11:56 +0200
    Re: linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-09 10:32 -0700
      Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-10 11:12 +1000
        Re: linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-09 18:59 -0700
          Re: linux raid vs hw raid KR <kristian.rasmussen@broadpark.no.spam.com> - 2011-04-10 04:32 +0200
            Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-10 12:46 +1000
              Re: linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-09 20:39 -0700
    Re: linux raid vs hw raid Robert Riches <spamtrap42@jacob21819.net> - 2011-04-10 03:47 +0000
      Re: linux raid vs hw raid Balwinder S Dheeman <bsd.SANSPAM@anu.homelinux.net> - 2011-04-10 11:11 +0530
        Re: linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-09 23:29 -0700
          Re: linux raid vs hw raid Balwinder S Dheeman <bsd.SANSPAM@anu.homelinux.net> - 2011-04-10 14:05 +0530
        Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-10 20:16 +1000
          Re: linux raid vs hw raid Tim Watts <tw@dionic.net> - 2011-04-10 11:28 +0100
          Re: linux raid vs hw raid Balwinder S Dheeman <bsd.SANSPAM@anu.homelinux.net> - 2011-04-10 19:43 +0530
        Re: linux raid vs hw raid Robert Riches <spamtrap42@jacob21819.net> - 2011-04-12 03:44 +0000
          Re: linux raid vs hw raid Balwinder S Dheeman <bsd.SANSPAM@anu.homelinux.net> - 2011-04-12 13:56 +0530
      Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-10 20:09 +1000
        Re: linux raid vs hw raid Robert Riches <spamtrap42@jacob21819.net> - 2011-04-12 03:37 +0000

csiph-web