Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1321788 > unrolled thread

RE: [PATCH v2] dm pref-path: provides preferred path load balance policy

Started by"Nalla, Ravikanth" <ravikanth.nalla@hpe.com>
First post2016-01-29 15:20 +0100
Last post2016-01-31 00:40 +0100
Articles 5 — 3 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  RE: [PATCH v2] dm pref-path: provides preferred path load balance  policy "Nalla, Ravikanth" <ravikanth.nalla@hpe.com> - 2016-01-29 15:20 +0100
    Re: [PATCH v2] dm pref-path: provides preferred path load balance  policy "Benjamin Marzinski" <bmarzins@redhat.com> - 2016-01-29 19:00 +0100
      Re: [PATCH v2] dm pref-path: provides preferred path load balance  policy Hannes Reinecke <hare@suse.de> - 2016-01-30 09:40 +0100
        Re: [PATCH v2] dm pref-path: provides preferred path load balance  policy "Benjamin Marzinski" <bmarzins@redhat.com> - 2016-01-30 21:30 +0100
          Re: [dm-devel] [PATCH v2] dm pref-path: provides preferred path load  balance policy "Benjamin Marzinski" <bmarzins@redhat.com> - 2016-01-31 00:40 +0100

#1321788 — RE: [PATCH v2] dm pref-path: provides preferred path load balance policy

From"Nalla, Ravikanth" <ravikanth.nalla@hpe.com>
Date2016-01-29 15:20 +0100
SubjectRE: [PATCH v2] dm pref-path: provides preferred path load balance policy
Message-ID<qWi43-4jW-47@gated-at.bofh.it>
Hi Mike, Hannes, Ben

On 1/22/2016 10:29 PM, Mike Snitzer wrote:

 [Hannes please fix your mail client, seems you dropped all the original CCs]

On Fri, Jan 22 2016 at  8:42am -0500,
Hannes Reinecke <hare@suse.de> wrote:

> On 01/22/2016 02:31 PM, Ravikanth Nalla wrote:
> > 
> > Functionality provided in this module is verified on wide variety of 
> > servers ( with 2 CPU sockets, 4 CPU sockets and 8 CPU sockets).
> > Additionally in some specific multipathing configurations involving 
> > varied path speeds, proposed preferred path policy provided some 
> > performance improvements over existing round-robin and service-time 
> > load balance policies.
> > 
> Shouldn't service-time provide similar functionality?
> After all, for all scenarios described above the preferred path would 
> have a lower service time, so they should be selected automatically, 
> no?

Yes you are right that in case if the user is preferring a path because of its path speed, it will have a lower service time and so behavior of service time policy will be similar to the preferred path policy that we proposed. However another reason for proposing this policy is a scenario where user want to totally eliminate a particular path which is flaky and behaving in a unpredictable manner. In this case, service time policy may still schedule I/O on this path as randomly it may demonstrate better service time but the overall I/O performance over a period of time could be affected and so in this scenario selecting a known good preferred path will be beneficial was our thinking. In fact when we did comparative testing our policy with service time in a setup with varying path speeds ( 8Gig FC and 4Gig FC) we saw the following results which showed that our policy fared marginally better than the service-time policy.

service-time:
    io/sec    MB/sec    (msec)    Max Response Time 
   =======    ======    ======   =============
   1383.2    1450.3    23.132      44.7 

pref-path:
    io/sec    MB/sec    (msec)    Max Response Time 
   =======    ======    ======   =============
   1444.3    1514.5    22.152      37.4

> Yes, I'm thinking the same thing.  In fact the service-time also has the ability to specify 'relative_throughput'
> (see: Documentation/device-mapper/dm-service-time.txt).

Thanks for suggestion to look at the 'relative_throughput' feature associated with service-time. After your comment, I could see from the code how this feature works but unfortunately we are not able to find documentation on how to specify this in the multipath.conf file. May be we are not looking at the right place and if you have further pointers on how to use this feature, that will be helpful.

> I'm also missing why different path groups couldn't be used for fast vs slow paths...
> Basically: Ravikanth we need _proof_ that you've exhausted the capabilities of the existing path selector policies (in conjunction with path groups)

I assume you are referring to specifying following in multipath.conf file ( prio_args having the path to be preferred )

multipath{
	prio "weightedpath"
	prio_args "devname sdp 1"
	path_grouping_policy group_by_prio
}

Yes with this way it  that it is possible to lock the path to a specific path similar to what we have proposed. However having used this feature, it appears that it is not that intuitive to use it and also need multiple configuration parameters to be specified in the conf file. Hence the main intention with which we  provided this feature was to  provide a more intuitive way to specify a user preferred path.

>Not understanding why you verified "some performance improvements" but stopped short of showing them.

The performance improvements that we observed are shown above. Actually we never intended this to be a performance feature as our intention was to provide users with an easier option to specify a preferred path in scenarios like flaky SAN environments and so that's why we did not mention it earlier.

>On 1/22/2016 10:36 PM, Benjamin Marzinski wrote:

> This seems like a problem that has already been solved with path groups.
> If the path(s) in your preferred path group are there, multipath will use them.  If not, then it will use your less preferred path(s), and load balance across them > how ever you choose with the path_selectors.

> I admit that we don't have a path prioritizer that does a good job of allowing users to manually pick a specific path to prefer.  But it seems to me that there is > >where we should be solving the issue.

Yes as  mentioned , it appears that we will be able to achieve the same result using the above multipath{...} configuration. However as you mentioned I felt that it is not that user friendly in specify the path to prefer. So when you mentioned about solving the problem there, could you please clarify on what you had in mind and is there anything specific from our implementation that can be used there ?

[toc] | [next] | [standalone]

#1321924

From"Benjamin Marzinski" <bmarzins@redhat.com>
Date2016-01-29 19:00 +0100
Message-ID<qWluV-6FF-1@gated-at.bofh.it>
In reply to#1321788
On Fri, Jan 29, 2016 at 02:10:52PM +0000, Nalla, Ravikanth wrote:
> Hi Mike, Hannes, Ben
> > This seems like a problem that has already been solved with path groups.
> > If the path(s) in your preferred path group are there, multipath will use them.  If not, then it will use your less preferred path(s), and load balance across them > how ever you choose with the path_selectors.
> 
> > I admit that we don't have a path prioritizer that does a good job of allowing users to manually pick a specific path to prefer.  But it seems to me that there is > >where we should be solving the issue.
> 
> Yes as  mentioned , it appears that we will be able to achieve the same result using the above multipath{...} configuration. However as you mentioned I felt that it is not that user friendly in specify the path to prefer. So when you mentioned about solving the problem there, could you please clarify on what you had in mind and is there anything specific from our implementation that can be used there ?
> 

There are two changes that I'm working on.

1. I'm adding an option for the alua prioritizer so that setting the
ALUA TPG Preferred Bit will cause the alau prioritizer to put that path
in a group by itself (with the highest priority). Currently if the
preferred bit is set for an active/optimized path, and there are other
active/optimized paths, they are all grouped together, and there is no
way to change that. So, for people with ALUA enabled hardware, they can
just enable the option, and set the Preferred Bit.

2. For people that need to be able to control the exact priority, I'm
redoing the weighted handler to allow better ways to specify the paths
in a presistent manner.  It won't be as simple as the alua method, but
it will be actually usable, unlike it's current state.

-Ben

[toc] | [prev] | [next] | [standalone]

#1322292

FromHannes Reinecke <hare@suse.de>
Date2016-01-30 09:40 +0100
Message-ID<qWzex-8sE-3@gated-at.bofh.it>
In reply to#1321924
On 01/29/2016 06:50 PM, Benjamin Marzinski wrote:
> On Fri, Jan 29, 2016 at 02:10:52PM +0000, Nalla, Ravikanth wrote:
>> Hi Mike, Hannes, Ben
>>> This seems like a problem that has already been solved with path groups.
>>> If the path(s) in your preferred path group are there, multipath will
 >>> use them.  If not, then it will use your less preferred path(s), and
 >>> load balance across them > how ever you choose with the path_selectors.
>>>
>>> I admit that we don't have a path prioritizer that does a good job of
 >>> allowing users to manually pick a specific path to prefer.  But it 
seems
 >>> to me that there is > >where we should be solving the issue.
>>>
>> Yes as  mentioned , it appears that we will be able to achieve the same
 >> result using the above multipath{...} configuration. However as you
 >> mentioned I felt that it is not that user friendly in specify the path
 >> to prefer. So when you mentioned about solving the problem there, could
 >> you please clarify on what you had in mind and is there anything 
specific
 >> from our implementation that can be used there ?
>>
>
> There are two changes that I'm working on.
>
> 1. I'm adding an option for the alua prioritizer so that setting the
> ALUA TPG Preferred Bit will cause the alau prioritizer to put that path
> in a group by itself (with the highest priority). Currently if the
> preferred bit is set for an active/optimized path, and there are other
> active/optimized paths, they are all grouped together, and there is no
> way to change that. So, for people with ALUA enabled hardware, they can
> just enable the option, and set the Preferred Bit.
>
Hmm? I was under the distinct impression that it's exactly the other way 
round; at least in my code I have this:

                 switch(aas) {
                         case AAS_OPTIMIZED:
                                 rc = 50;
                                 break;
                         case AAS_NON_OPTIMIZED:
                                 rc = 10;
                                 break;
                         case AAS_LBA_DEPENDENT:
                                 rc = 5;
                                 break;
                         case AAS_STANDBY:
                                 rc = 1;
                                 break;
                         default:
                                 rc = 0;
                 }
                 if (priopath && aas != AAS_OPTIMIZED)
                         rc += 80;

ie any path with the 'prio' bit set will be getting a differen priority 
than those without. Consequently they'll be grouped into different 
priority groups.
I'd be surprised if your code is different, but what do I know ...

> 2. For people that need to be able to control the exact priority, I'm
> redoing the weighted handler to allow better ways to specify the paths
> in a presistent manner.  It won't be as simple as the alua method, but
> it will be actually usable, unlike it's current state.
>
That, however, is greatly appreciated :-)

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

[toc] | [prev] | [next] | [standalone]

#1322502

From"Benjamin Marzinski" <bmarzins@redhat.com>
Date2016-01-30 21:30 +0100
Message-ID<qWKjE-IE-1@gated-at.bofh.it>
In reply to#1322292
On Sat, Jan 30, 2016 at 09:32:53AM +0100, Hannes Reinecke wrote:
> On 01/29/2016 06:50 PM, Benjamin Marzinski wrote:
> >On Fri, Jan 29, 2016 at 02:10:52PM +0000, Nalla, Ravikanth wrote:
> >>Hi Mike, Hannes, Ben
> >>>This seems like a problem that has already been solved with path groups.
> >>>If the path(s) in your preferred path group are there, multipath will
> >>> use them.  If not, then it will use your less preferred path(s), and
> >>> load balance across them > how ever you choose with the path_selectors.
> >>>
> >>>I admit that we don't have a path prioritizer that does a good job of
> >>> allowing users to manually pick a specific path to prefer.  But it seems
> >>> to me that there is > >where we should be solving the issue.
> >>>
> >>Yes as  mentioned , it appears that we will be able to achieve the same
> >> result using the above multipath{...} configuration. However as you
> >> mentioned I felt that it is not that user friendly in specify the path
> >> to prefer. So when you mentioned about solving the problem there, could
> >> you please clarify on what you had in mind and is there anything specific
> >> from our implementation that can be used there ?
> >>
> >
> >There are two changes that I'm working on.
> >
> >1. I'm adding an option for the alua prioritizer so that setting the
> >ALUA TPG Preferred Bit will cause the alau prioritizer to put that path
> >in a group by itself (with the highest priority). Currently if the
> >preferred bit is set for an active/optimized path, and there are other
> >active/optimized paths, they are all grouped together, and there is no
> >way to change that. So, for people with ALUA enabled hardware, they can
> >just enable the option, and set the Preferred Bit.
> >
> Hmm? I was under the distinct impression that it's exactly the other way
> round; at least in my code I have this:
> 
>                 switch(aas) {
>                         case AAS_OPTIMIZED:
>                                 rc = 50;
>                                 break;
>                         case AAS_NON_OPTIMIZED:
>                                 rc = 10;
>                                 break;
>                         case AAS_LBA_DEPENDENT:
>                                 rc = 5;
>                                 break;
>                         case AAS_STANDBY:
>                                 rc = 1;
>                                 break;
>                         default:
>                                 rc = 0;
>                 }
>                 if (priopath && aas != AAS_OPTIMIZED)
>                         rc += 80;
> 
> ie any path with the 'prio' bit set will be getting a differen priority than
> those without. Consequently they'll be grouped into different priority
> groups.
> I'd be surprised if your code is different, but what do I know ...

No. That's only true if the path doesn't have AAS_OPTIMIZED set.  So if
you have a non-optimized path with the pref bit set, it will be in a
group by itself. If the path is AAS_OPTIMIZED, the pref bit is ignored.

Like I mentioned before, you are the one who changed this

commit b330bf8a5e6a29b51af0d8b4088e0d8554e5cfb4
Author: Hannes Reinecke <hare@suse.de>
Date:   Tue Jul 16 09:12:54 2013 +0200

    alua: Do not add preferred path priority for active/optimized
    
    When a path is in active/optimized we should disregard the
    'preferred path' bit when calculating the priority.
    Otherwise we'll end up with having two different priorities
    (one for 'active/optimized (preferred)' and one for
    'active/optimized (non-preferred)').
    Which will result in two different path groups and a
    sub-optimal path usage.
    
    Signed-off-by: Hannes Reinecke <hare@suse.de>


Before this commit, it always used the pref bit. Again, like I said
before, I'm saying that this was the wrong thing to do.  The Spec is
pretty vague on what you should expect to happen when you set to pref
bit.  When the path was in a group by itself, I got complaints. Now that
the path is is a group with other active/optimized paths, I get
complaints.  The only answer is to allow the user to say what they want
the pref bit to mean.

-Ben 

 
> >2. For people that need to be able to control the exact priority, I'm
> >redoing the weighted handler to allow better ways to specify the paths
> >in a presistent manner.  It won't be as simple as the alua method, but
> >it will be actually usable, unlike it's current state.
> >
> That, however, is greatly appreciated :-)
> 
> Cheers,
> 
> Hannes
> -- 
> Dr. Hannes Reinecke		      zSeries & Storage
> hare@suse.de			      +49 911 74053 688
> SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
> GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

[toc] | [prev] | [next] | [standalone]

#1322525 — Re: [dm-devel] [PATCH v2] dm pref-path: provides preferred path load balance policy

From"Benjamin Marzinski" <bmarzins@redhat.com>
Date2016-01-31 00:40 +0100
SubjectRe: [dm-devel] [PATCH v2] dm pref-path: provides preferred path load balance policy
Message-ID<qWNhw-2Mj-5@gated-at.bofh.it>
In reply to#1322502
On Sat, Jan 30, 2016 at 02:25:25PM -0600, Benjamin Marzinski wrote:
> Before this commit, it always used the pref bit. Again, like I said
> before, I'm saying that this was the wrong thing to do.  The Spec is

Oops. I meant: "I'm NOT saying that this was the wrong thing to do".

I am also fine with changing the default back to making the pref bit
always create it's own path group. As long there is a way for users to
get either behavior, I'm happy.

-Ben

> pretty vague on what you should expect to happen when you set to pref
> bit.  When the path was in a group by itself, I got complaints. Now that
> the path is is a group with other active/optimized paths, I get
> complaints.  The only answer is to allow the user to say what they want
> the pref bit to mean.
> 
> -Ben

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web