Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #230279

Re: recommendations for supported, affordable hardware raid controller.

Show all headers | View raw

[Multipart message — attachments visible in raw view] - view raw

On Sb, 02 ian 21, 13:35:06, David Christensen wrote:
> On 2021-01-02 03:24, Andrei POPESCU wrote:
> 
> > http://www.unixsheikh.com/articles/battle-testing-data-integrity-verification-with-zfs-btrfs-and-mdadm-dm-integrity.html
> 
> That looks interesting.  Thanks for the link.  :-)
> 
> 
> On 2021-01-02 08:08, Richard Hector wrote:
> > On 3/01/21 12:24 am, Andrei POPESCU wrote:
> > > In case of data corruption (system crash, power outage, user error,
> > > or even just a HDD "hiccup") plain md without the dm-integrity
> > > layer won't even be able to tell which is the good data and will
> > > overwrite your good data with bad data. Silently.
> 
> > I've had crashes and power outages and never noticed any problems,
> > not that that means they won't happen (or even that they haven't
> > happened). Does a journalling filesystem on top not cover that?
> 
> AIUI a journaling filesystem provides a two-step process to achieve atomic
> writes of multiple sectors to disk -- e.g. a process wants to put some data
> into a block here (say, a file), a block there (say, a directory), etc., and
> consistency of the on-disk data structures must be preserved.  The journal
> provides a two-step process whereby everything is written to the journal,
> then everything is written to disk.

That would mean all data is written to the disk twice and would make a 
journaling file system twice as slow compared to a non-journaling file 
system; the journal is typically on the same storage.

Even if you put the journal on another storage, having the data written 
there in parallel would basically result in a sort of RAID ;)

> If either step is interrupted, the
> filesystem driver will detect the failure and respond.  When done, either
> all of the blocks have been updated on disk or none of the blocks on disk
> have been changed.

My understanding of [1] is that the journal only keeps track of metadata 
and/or data *updates*.

In case of a crash it can only tell you whether the write was 
"completed". It has no way to know whether the data on disk is actually 
correct.

[1] https://en.wikipedia.org/wiki/Journaling_file_system

When using a non-checksumming file system we are trusting the storage to 
either return correct data or report a read error (bad block) in order 
to restore it from the "other" copies (in case of RAID) or, worst case, 
from backup.

If the storage returns *wrong* data instead a non-checksumming file 
system will never notice as it has no concept of what the correct data 
should be. 

A RAID by itself is also unable to distinguish between correct and 
incorrect data and may even overwrite the good data with the bad data.

If we are lucky the error is noticed *and* the backups contain the 
correct data (the backup may have been created from the corrupt data).

Without good integrity checks it is also very difficult to tell how 
probable such issues are.


Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser

Back to linux.debian.user | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

recommendations for supported, affordable hardware raid controller. Steven Mainor <steve@degga.net> - 2021-01-01 19:10 +0100
  Re: recommendations for supported, affordable hardware raid  controller. "Alexander V. Makartsev" <avbetev@gmail.com> - 2021-01-01 20:00 +0100
  Re: recommendations for supported, affordable hardware raid  controller. Dan Ritter <dsr@randomstring.org> - 2021-01-01 20:00 +0100
    Re: recommendations for supported, affordable hardware raid controller. Sven Hartge <sven@svenhartge.de> - 2021-01-01 20:40 +0100
  Re: recommendations for supported, affordable hardware raid  controller. David Christensen <dpchrist@holgerdanske.com> - 2021-01-01 21:10 +0100
    Re: recommendations for supported, affordable hardware raid  controller. Steven Mainor <steve@degga.net> - 2021-01-02 09:30 +0100
      Re: recommendations for supported, affordable hardware raid  controller. Steven Mainor <steve@degga.net> - 2021-01-02 09:50 +0100
  Re: recommendations for supported, affordable hardware raid  controller. Michael Stone <mstone@debian.org> - 2021-01-01 21:20 +0100
  Re: recommendations for supported, affordable hardware raid  controller. Steven Mainor <steve@degga.net> - 2021-01-02 09:20 +0100
    Re: recommendations for supported, affordable hardware raid  controller. David Christensen <dpchrist@holgerdanske.com> - 2021-01-02 10:50 +0100
      Re: recommendations for supported, affordable hardware raid  controller. Andrei POPESCU <andreimpopescu@gmail.com> - 2021-01-02 12:30 +0100
        Re: recommendations for supported, affordable hardware raid  controller. Richard Hector <richard@walnut.gen.nz> - 2021-01-02 17:10 +0100
          Re: recommendations for supported, affordable hardware raid controller. Gene Heskett <gheskett@shentel.net> - 2021-01-02 17:50 +0100
        Re: recommendations for supported, affordable hardware raid  controller. David Christensen <dpchrist@holgerdanske.com> - 2021-01-02 22:40 +0100
          Re: recommendations for supported, affordable hardware raid  controller. Andrei POPESCU <andreimpopescu@gmail.com> - 2021-01-03 10:30 +0100
            Re: recommendations for supported, affordable hardware raid  controller. David Christensen <dpchrist@holgerdanske.com> - 2021-01-03 22:50 +0100
              Re: recommendations for supported, affordable hardware raid  controller. Andrei POPESCU <andreimpopescu@gmail.com> - 2021-01-04 08:30 +0100
            Re: recommendations for supported, affordable hardware raid controller. Stefan Monnier <monnier@iro.umontreal.ca> - 2021-01-03 23:20 +0100
            Re: recommendations for supported, affordable hardware raid  controller. Michael Stone <mstone@debian.org> - 2021-01-04 02:00 +0100
              Re: recommendations for supported, affordable hardware raid  controller. Andrei POPESCU <andreimpopescu@gmail.com> - 2021-01-04 08:40 +0100
                Re: recommendations for supported, affordable hardware raid  controller. Michael Stone <mstone@debian.org> - 2021-01-04 14:40 +0100
    Re: recommendations for supported, affordable hardware raid controller. Sven Hartge <sven@svenhartge.de> - 2021-01-02 12:50 +0100
      Re: recommendations for supported, affordable hardware raid controller. Nicholas Geovanis <nickgeovanis@gmail.com> - 2021-01-02 16:30 +0100
        Re: recommendations for supported, affordable hardware raid controller. Sven Hartge <sven@svenhartge.de> - 2021-01-02 17:30 +0100
        Re: recommendations for supported, affordable hardware raid  controller. Alain D D Williams <addw@phcomp.co.uk> - 2021-01-02 18:40 +0100
  Re: recommendations for supported, affordable hardware raid controller. deloptes <deloptes@gmail.com> - 2021-01-02 10:00 +0100

csiph-web