Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1521743 > unrolled thread

Re: [PATCH v4] staging: lustre: mdc: manage number of modify RPCs in flight

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH v4] staging: lustre: mdc: manage number of modify RPCs in  flight Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2016-11-14 16:20 +0100
    Re: [PATCH v4] staging: lustre: mdc: manage number of modify RPCs  in flight James Simmons <jsimmons@infradead.org> - 2016-11-14 18:00 +0100
      Re: [PATCH v4] staging: lustre: mdc: manage number of modify RPCs in  flight Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2016-11-14 18:30 +0100
        Re: [PATCH v4] staging: lustre: mdc: manage number of modify RPCs  in flight James Simmons <jsimmons@infradead.org> - 2016-11-14 18:30 +0100
          Re: [PATCH v4] staging: lustre: mdc: manage number of modify RPCs in  flight Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2016-11-15 11:10 +0100

#1521743 — Re: [PATCH v4] staging: lustre: mdc: manage number of modify RPCs in flight

On Thu, Nov 10, 2016 at 10:51:13AM -0500, James Simmons wrote:
> From: Gregoire Pichon <gregoire.pichon@bull.net>
> 
> This patch is the main client part of a new feature that supports
> multiple modify metadata RPCs in parallel. Its goal is to improve
> metadata operations performance of a single client, while maintening
> the consistency of MDT reply reconstruction and MDT recovery
> mechanisms.
> 
> It allows to manage the number of modify RPCs in flight within
> the client obd structure and to assign a virtual index (the tag) to
> each modify RPC to help server side cleaning of reply data.
> 
> The mdc component uses this feature to send multiple modify RPCs
> in parallel.

Is this a new feature?  Why should we take this now and not just wait
until the code is out of staging?

thanks,

greg k-h

[toc] | [next] | [standalone]

#1521864 — Re: [PATCH v4] staging: lustre: mdc: manage number of modify RPCs in flight

> On Thu, Nov 10, 2016 at 10:51:13AM -0500, James Simmons wrote:
> > From: Gregoire Pichon <gregoire.pichon@bull.net>
> > 
> > This patch is the main client part of a new feature that supports
> > multiple modify metadata RPCs in parallel. Its goal is to improve
> > metadata operations performance of a single client, while maintening
> > the consistency of MDT reply reconstruction and MDT recovery
> > mechanisms.
> > 
> > It allows to manage the number of modify RPCs in flight within
> > the client obd structure and to assign a virtual index (the tag) to
> > each modify RPC to help server side cleaning of reply data.
> > 
> > The mdc component uses this feature to send multiple modify RPCs
> > in parallel.
> 
> Is this a new feature?  Why should we take this now and not just wait
> until the code is out of staging?

Yes on the server side. So the problem on our meta data servers couldn't
handle writing mulitiple bits of data to the back end disk at ths same
time.

One client side the issue was the metadata operations were being 
serialized by a mutex in the MDC layer. That is what this patch fixed.
So for the client it would be a performance improvement patch. 

[toc] | [prev] | [next] | [standalone]

#1521902

On Mon, Nov 14, 2016 at 04:59:48PM +0000, James Simmons wrote:
> 
> > On Thu, Nov 10, 2016 at 10:51:13AM -0500, James Simmons wrote:
> > > From: Gregoire Pichon <gregoire.pichon@bull.net>
> > > 
> > > This patch is the main client part of a new feature that supports
> > > multiple modify metadata RPCs in parallel. Its goal is to improve
> > > metadata operations performance of a single client, while maintening
> > > the consistency of MDT reply reconstruction and MDT recovery
> > > mechanisms.
> > > 
> > > It allows to manage the number of modify RPCs in flight within
> > > the client obd structure and to assign a virtual index (the tag) to
> > > each modify RPC to help server side cleaning of reply data.
> > > 
> > > The mdc component uses this feature to send multiple modify RPCs
> > > in parallel.
> > 
> > Is this a new feature?  Why should we take this now and not just wait
> > until the code is out of staging?
> 
> Yes on the server side. So the problem on our meta data servers couldn't
> handle writing mulitiple bits of data to the back end disk at ths same
> time.
> 
> One client side the issue was the metadata operations were being 
> serialized by a mutex in the MDC layer. That is what this patch fixed.
> So for the client it would be a performance improvement patch. 

So, it's a "performance" patch, which isn't functionality, so why should
we merge this to staging now?  Why aren't people working on the known
coding issues to get this out of staging instead of working on
performance stuff?

thanks,

greg k-h

[toc] | [prev] | [next] | [standalone]

#1521910 — Re: [PATCH v4] staging: lustre: mdc: manage number of modify RPCs in flight

> On Mon, Nov 14, 2016 at 04:59:48PM +0000, James Simmons wrote:
> > 
> > > On Thu, Nov 10, 2016 at 10:51:13AM -0500, James Simmons wrote:
> > > > From: Gregoire Pichon <gregoire.pichon@bull.net>
> > > > 
> > > > This patch is the main client part of a new feature that supports
> > > > multiple modify metadata RPCs in parallel. Its goal is to improve
> > > > metadata operations performance of a single client, while maintening
> > > > the consistency of MDT reply reconstruction and MDT recovery
> > > > mechanisms.
> > > > 
> > > > It allows to manage the number of modify RPCs in flight within
> > > > the client obd structure and to assign a virtual index (the tag) to
> > > > each modify RPC to help server side cleaning of reply data.
> > > > 
> > > > The mdc component uses this feature to send multiple modify RPCs
> > > > in parallel.
> > > 
> > > Is this a new feature?  Why should we take this now and not just wait
> > > until the code is out of staging?
> > 
> > Yes on the server side. So the problem on our meta data servers couldn't
> > handle writing mulitiple bits of data to the back end disk at ths same
> > time.
> > 
> > One client side the issue was the metadata operations were being 
> > serialized by a mutex in the MDC layer. That is what this patch fixed.
> > So for the client it would be a performance improvement patch. 
> 
> So, it's a "performance" patch, which isn't functionality, so why should
> we merge this to staging now?  Why aren't people working on the known
> coding issues to get this out of staging instead of working on
> performance stuff?

Because the primary goal which the people at my company, not Intel by 
the way, wanted was to get this to what is running in production 
environments so people would actually use the staging client. When it was 
old and broken no one would touch it with a 10 foot pole. The currently 
supported verison in production is lustre 2.8.0 so we are only about 30 
patches away from reaching the goal. I'm going to respond to your other
email in length about leaving staging. It would be really nice for the
user base if we can reach that goal.

[toc] | [prev] | [next] | [standalone]

#1522517

On Mon, Nov 14, 2016 at 05:27:00PM +0000, James Simmons wrote:
> 
> > On Mon, Nov 14, 2016 at 04:59:48PM +0000, James Simmons wrote:
> > > 
> > > > On Thu, Nov 10, 2016 at 10:51:13AM -0500, James Simmons wrote:
> > > > > From: Gregoire Pichon <gregoire.pichon@bull.net>
> > > > > 
> > > > > This patch is the main client part of a new feature that supports
> > > > > multiple modify metadata RPCs in parallel. Its goal is to improve
> > > > > metadata operations performance of a single client, while maintening
> > > > > the consistency of MDT reply reconstruction and MDT recovery
> > > > > mechanisms.
> > > > > 
> > > > > It allows to manage the number of modify RPCs in flight within
> > > > > the client obd structure and to assign a virtual index (the tag) to
> > > > > each modify RPC to help server side cleaning of reply data.
> > > > > 
> > > > > The mdc component uses this feature to send multiple modify RPCs
> > > > > in parallel.
> > > > 
> > > > Is this a new feature?  Why should we take this now and not just wait
> > > > until the code is out of staging?
> > > 
> > > Yes on the server side. So the problem on our meta data servers couldn't
> > > handle writing mulitiple bits of data to the back end disk at ths same
> > > time.
> > > 
> > > One client side the issue was the metadata operations were being 
> > > serialized by a mutex in the MDC layer. That is what this patch fixed.
> > > So for the client it would be a performance improvement patch. 
> > 
> > So, it's a "performance" patch, which isn't functionality, so why should
> > we merge this to staging now?  Why aren't people working on the known
> > coding issues to get this out of staging instead of working on
> > performance stuff?
> 
> Because the primary goal which the people at my company, not Intel by 
> the way, wanted was to get this to what is running in production 
> environments so people would actually use the staging client. When it was 
> old and broken no one would touch it with a 10 foot pole. The currently 
> supported verison in production is lustre 2.8.0 so we are only about 30 
> patches away from reaching the goal. I'm going to respond to your other
> email in length about leaving staging. It would be really nice for the
> user base if we can reach that goal.

Ugh, ok, I'll take this, but PLEASE start working to get this tree out
of staging and stop working on adding new features.  The longer you
delay, the more you have to deal with this external/internal tree mess.

greg k-h

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web