Re: linux raid vs hw raid

Date	2011-04-09 11:56 +0200
From	KR <kristian.rasmussen@broadpark.no.spam.com>
Newsgroups	comp.os.linux.misc
Subject	Re: linux raid vs hw raid
References	<fc0t68x5ci.ln2@goaway.wombat.san-francisco.ca.us>
Message-ID	<4da02d29@news.broadpark.no> (permalink)

Show all headers | View raw

On 06.04.2011 04:39, Keith Keller wrote:
> 
> The obvious advantage to this is cost: I can save almost 40% of my
> original estimate by using desktop drives instead, thus fulfilling the
> original meaning of the I of the RAID acronym.  There are other
> advantages, as well, including being able to build a RAID6, which I
> slightly prefer over a RAID50, and having more flexibility later on if I
> want to move to bigger disks.  (Yes, I have seen the documentation
> warning against too-large RAID arrays resulting in a failure during a
> rebuild.)  A tertiary advantage would be that I would learn how to work
> with linux software RAID, a skill I haven't yet acquired.

"Enterprise hard drives" can mean at least two different things: either
the drives are high-performance SATA or SAS drives (10k or 15k RPM), or
they are regular high-end (7.2k RPM) SATA drives with non-sequential
serial numbers.

In the latter case, the drives are really regular desktop drives,
sometimes fitted with slightly more cache, but the drives are from
different batches. This saves you from experiencing multiple drive
failures over a short time period due to some manufacturing defect.

> The disadvantages I can think of are: higher probability of disk
> failures, resulting in more work for me in swapping out and RMAing
> failed drives; potential degradation in performance, due both to RAID in
> software and slower disks; a learning curve for linux RAID; and a
> configuration less likely to be supported by the hardware RAID vendor.

Another issue is that while most hardware RAID cards can cope with
failing drive electronics that hang the bus, simple on-board/PCIe SATA
controllers are usually not so forgiving and may cause a system freeze
or a kernel OOPS, the latter possibly causing severe file system
corruption (I've experienced it a few times).

Also, most hardware RAID controllers will perform an automatic block
reallocation when it encounters a bad block during read or verify. This
will be logged, so you'll usually get ample warning before a drive failure.

> My questions:
> 
> 1) Has anyone done this before, and if so, what were the results?  Was
> performance acceptable in this configuration?  Are there any gotchas to
> an otherwise workable configuration?

I've used software RAID on occations, mostly with device-mapper but I've
used md a few times as well. No problems to speak of (or at all, really).

Neither the MD nor device-mapper reads the S.M.A.R.T. data from the
drives in the array. mdadm can send a notification by mail when a drive
fails, but I usually monitor the S.M.A.R.T. data from each drive with
smartd as well.

> 2) From what I've read so far, using desktop-class disks with linux
> software RAID should not be a major problem, unlike using them on a true
> hardware RAID card.  Is this reasonably accurate?  If not, are there
> links that describe the difficulties?

There are no issues that I'm aware of with using desktop class drives
with a hardware RAID card either, other than the obvious reduction in
performance if you're going from 15k or 10k RPM drives to 7.2k RPM or lower.

> 3) Suppose that my RAID6 starts out using 12 2TB disks, with three free
> drive bays (one would be a hot spare).  Later on, I want to seamlessly
> replace the 2TB disks with 3TB or larger disks.  Can mdadm grow an array
> like this if, say, I replace one drive, rebuild, and repeat until I've
> replaced all 12 disks with larger ones?  Or will the new 3TB disks only
> be used up to 2TB, the size of the original disks?

Your data will be striped across the 2Tb drives, so replacing one of
them with a 3Tb drive will leave 1Tb unused (or leave you with a
degraded RAID set if the 2Tb drives are emulating 512 byte sectors,
something the 3Tb drive won't be doing).

Since most of the other posts in this thread claim that software RAID
will give better performance, I thought I'd chime in with the exact
opposite viewpoint: Unless you're using a so-called 'fakeRAID'
controller (one without a CPU), a hardware RAID controller is likely to
perform slightly better for read operations due to caching, and perhaps
considerably better when writing.

Yes, the CPUs on RAID controllers are usually quite anemic. The reason
for this is that the complexity of the mathematical operations involved
with generating RAID stripes range from completely trivial (RAID 0, 1,
5, 10, 50) to rather simple (RAID 6, 60). Even with a slow CPU
generating stripe data, the limiting factor is more likely to be the
drives or the host OS.

However, there is one other major difference between hardware and
software RAID which may severely affect write speeds:

- A hardware RAID controller receives the data over the PCIe bus,
generates the stripes, and pushes the data to the drives.

- A software RAID driver receives the data from the OS, generates the
stripes, and pushes the data to the controller(s) over the PCIe bus.

As you can see, a software RAID setup means that all stripes have to be
transferred across the bus. For RAID 1 or 10 this means a 100% overhead
for any write operation; for RAID 5, 6, 50 or 60 the overhead percentage
varies with the number of drives in the set (more drives are better).

This may or may not be significant for a given setup. If you're building
a RAID 6 set with 12 reasonably fast 7.2k RPM SATA-II drives and a good
PCIe based motherboard, the 20% overhead for the parity blocks may not
be noticable.

Whatever you choose, make sure you verify your array on a regular basis.
You want to discover that bad block when the array is operational, not
during a rebuild operation. Also, if you're going for desktop drives, I
would really recommend a hot or cold spare.

Back to comp.os.linux.misc | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-05 19:39 -0700
  Re: linux raid vs hw raid Tim Watts <tw@dionic.net> - 2011-04-06 08:01 +0100
    Re: linux raid vs hw raid David Brown <david@westcontrol.removethisbit.com> - 2011-04-06 10:03 +0200
      Re: linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-06 14:00 -0700
        Re: linux raid vs hw raid David Brown <david.brown@removethis.hesbynett.no> - 2011-04-06 23:42 +0200
        Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-08 10:45 +1000
          Re: linux raid vs hw raid David Brown <david@westcontrol.removethisbit.com> - 2011-04-08 11:12 +0200
            Re: linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-08 08:22 -0700
              Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-09 09:51 +1000
                Re: linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-08 17:10 -0700
                Re: linux raid vs hw raid David Brown <david.brown@removethis.hesbynett.no> - 2011-04-09 13:14 +0200
            Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-09 09:47 +1000
              Re: linux raid vs hw raid David Brown <david.brown@removethis.hesbynett.no> - 2011-04-09 13:55 +0200
          Re: linux raid vs hw raid Tris Orendorff <triso@remove-me.cogeco.ca> - 2011-04-12 18:04 +0000
            Re: linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-12 11:34 -0700
              Re: linux raid vs hw raid The Natural Philosopher <tnp@invalid.invalid> - 2011-04-12 21:13 +0100
                Re: linux raid vs hw raid David Brown <david@westcontrol.removethisbit.com> - 2011-04-13 09:45 +0200
                Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-14 13:42 +1000
                Re: linux raid vs hw raid David Brown <david@westcontrol.removethisbit.com> - 2011-04-14 09:15 +0200
                Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-15 08:03 +1000
                Re: linux raid vs hw raid Tim Watts <tw@dionic.net> - 2011-04-15 07:22 +0100
                Re: linux raid vs hw raid David Brown <david@westcontrol.removethisbit.com> - 2011-04-15 09:28 +0200
                Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-19 11:20 +1000
              Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-14 13:38 +1000
                Re: linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-13 21:49 -0700
            Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-14 13:34 +1000
              Re: linux raid vs hw raid Tris Orendorff <triso@remove-me.cogeco.ca> - 2011-04-15 21:59 +0000
                Re: linux raid vs hw raid "Peter J. Holzer" <hjp-usenet2@hjp.at> - 2011-04-16 00:56 +0200
                Re: linux raid vs hw raid The Natural Philosopher <tnp@invalid.invalid> - 2011-04-16 01:32 +0100
  Re: linux raid vs hw raid Tauno Voipio <tauno.voipio@notused.fi.invalid> - 2011-04-08 21:38 +0300
    Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-09 09:53 +1000
  Re: linux raid vs hw raid KR <kristian.rasmussen@broadpark.no.spam.com> - 2011-04-09 11:56 +0200
    Re: linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-09 10:32 -0700
      Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-10 11:12 +1000
        Re: linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-09 18:59 -0700
          Re: linux raid vs hw raid KR <kristian.rasmussen@broadpark.no.spam.com> - 2011-04-10 04:32 +0200
            Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-10 12:46 +1000
              Re: linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-09 20:39 -0700
    Re: linux raid vs hw raid Robert Riches <spamtrap42@jacob21819.net> - 2011-04-10 03:47 +0000
      Re: linux raid vs hw raid Balwinder S Dheeman <bsd.SANSPAM@anu.homelinux.net> - 2011-04-10 11:11 +0530
        Re: linux raid vs hw raid Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> - 2011-04-09 23:29 -0700
          Re: linux raid vs hw raid Balwinder S Dheeman <bsd.SANSPAM@anu.homelinux.net> - 2011-04-10 14:05 +0530
        Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-10 20:16 +1000
          Re: linux raid vs hw raid Tim Watts <tw@dionic.net> - 2011-04-10 11:28 +0100
          Re: linux raid vs hw raid Balwinder S Dheeman <bsd.SANSPAM@anu.homelinux.net> - 2011-04-10 19:43 +0530
        Re: linux raid vs hw raid Robert Riches <spamtrap42@jacob21819.net> - 2011-04-12 03:44 +0000
          Re: linux raid vs hw raid Balwinder S Dheeman <bsd.SANSPAM@anu.homelinux.net> - 2011-04-12 13:56 +0530
      Re: linux raid vs hw raid Grant <omg@grrr.id.au> - 2011-04-10 20:09 +1000
        Re: linux raid vs hw raid Robert Riches <spamtrap42@jacob21819.net> - 2011-04-12 03:37 +0000

csiph-web