Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1162926 > unrolled thread

Re: clustered MD

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: clustered MD Neil Brown <neilb@suse.de> - 2015-06-11 01:00 +0200
    Re: clustered MD Goldwyn Rodrigues <rgoldwyn@suse.com> - 2015-06-15 00:20 +0200
      Re: clustered MD NeilBrown <neilb@suse.com> - 2015-06-23 03:40 +0200

#1162926 — Re: clustered MD

On Wed, 10 Jun 2015 16:07:44 -0500
David Teigland <teigland@redhat.com> wrote:

> On Thu, Jun 11, 2015 at 06:31:31AM +1000, Neil Brown wrote:
> > What is your interest in this?  I'm always happy for open discussion and
> > varied input, but it would help to know to what extent you are a stake
> > holder?
> 
> Using the dlm correctly is non-trivial and should be reviewed.
> If the dlm is misused, some part of that may fall in my lap, if
> only so far as having to debug problems to distinguish between dlm
> bugs or md-cluster bugs.  This has been learned the hard way.
> 
> I have yet to find time to look up the previous review discussion.
> I will be more than happy if I find the dlm usage has already been
> thoroughly reviewed.

The DLM usage is the part that I am least comfortable with and I would
certainly welcome review.   There was a recent discussion of some issue that I
haven't had a chance to go over yet, but apart from that it has mostly just
been a few developers trying to figure out what we need and how that can be
implemented.

There are (as I recall) two main aspects of the DLM usage.
One is fairly idiomatic locking of the multiple write-intend bitmaps.
Each bitmap can be "active" or "idle".  When "idle" all bits are clear.
When "active", one node will usually have an exclusive lock.  If/when that node
dies, all other nodes must find out and at least one takes remedial action.
Once the remedial action is taken the bitmap becomes idle.  In that state
a new node can claim it.  When that happens all other nodes must find out so
they transition to the "watching an active bitmap" state.
This seems to fit well with the shared/exclusive reclaimable locks of DLM.

The other usage is to provide synchronous broadcast message passing between
nodes.  When one nodes makes a configuration change it needs to tell all other
nodes and wait for them to acknowledge before the change (such as adding a
spare) is committed.  There is a small collections of locks which represent
different states in a broadcast/acknowledge protocol.
This is the part I'm least confident of, but it seems to make sense and seems
to work.


Separately:

> Reading those messages again I see what you mean, they don't sound very
> nice, so sorry about that.  I'll repeat the one positive note, which is
> that the brief things I've noticed make it look much better than the dm
> approach from several years ago.

Thanks :-)
In part this effort is a response to "clvm" - which is a completely adequate
solution of clustering when you just need volume management (growing and
shrinking and striping volumes) but doesn't extend very well to RAID.

Look forward to any review comments you find time for:-)

Thanks,
NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1164929

On 06/12/2015 01:46 PM, David Teigland wrote:
> When a node fails, its dirty areas get special treatment from other nodes
> using the area_resyncing() function.  Should the suspend_list be created
> before any reads or writes from the file system are processed by md?  It
> seems to me that gfs journal recovery could read/write to dirty regions
> (from the failed node) before md was finished setting up the suspend_list.
> md could probably prevent that by using the recover_prep() dlm callback to
> set a flag that would block any i/o that arrived before the suspend_list
> was ready.
>
> .

Yes, we should call mddev_suspend() in recover_prep() and mddev_resume() 
after suspend_list is created. Thanks for pointing it out.

-- 
Goldwyn
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1170331

On Sun, 14 Jun 2015 17:19:31 -0500
Goldwyn Rodrigues <rgoldwyn@suse.com> wrote:

> 
> 
> On 06/12/2015 01:46 PM, David Teigland wrote:
> > When a node fails, its dirty areas get special treatment from other nodes
> > using the area_resyncing() function.  Should the suspend_list be created
> > before any reads or writes from the file system are processed by md?  It
> > seems to me that gfs journal recovery could read/write to dirty regions
> > (from the failed node) before md was finished setting up the suspend_list.
> > md could probably prevent that by using the recover_prep() dlm callback to
> > set a flag that would block any i/o that arrived before the suspend_list
> > was ready.
> >
> > .
> 
> Yes, we should call mddev_suspend() in recover_prep() and mddev_resume() 
> after suspend_list is created. Thanks for pointing it out.
> 

The only thing that nodes need to be careful of between the time when
some other node disappears and when that disappearance has been
completely handled is reads.
md/raid1 must ensure that  if/when the filesystem reads from a region
that the missing node was writing to, that the filesystem sees
consistent data - on all nodes.

So it needs to suspend read-balancing while it is uncertain.

Once the bitmap from the node has been loaded, the normal protection
against read-balancing in a "dirty" region is sufficient.  While
waiting for the bitmap to be loaded, the safe thing to do would be to
disable read-balancing completely.

So I think that recover_prep() should set a flag which disables all
read balancing, and recover_done() (or similar) should clear that flag.
Probably there should be one flag for each other node.

Calling mddev_suspend to suspect all IO is over-kill.  Suspending all
read balancing is all that is needed.

Thanks,
NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web