Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1234993 > unrolled thread

Re: [lkp] [nfsd] 4aac1bf05b: -2.9% fsmark.files_per_sec

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [lkp] [nfsd] 4aac1bf05b: -2.9% fsmark.files_per_sec Jeff Layton <jeff.layton@primarydata.com> - 2015-09-29 13:50 +0200
    Re: [lkp] [nfsd] 4aac1bf05b: -2.9% fsmark.files_per_sec "Huang\, Ying" <ying.huang@linux.intel.com> - 2015-09-30 01:30 +0200
      Re: [lkp] [nfsd] 4aac1bf05b: -2.9% fsmark.files_per_sec Jeff Layton <jeff.layton@primarydata.com> - 2015-09-30 02:10 +0200
        Re: [lkp] [nfsd] 4aac1bf05b: -2.9% fsmark.files_per_sec "Huang\, Ying" <ying.huang@linux.intel.com> - 2015-09-30 10:40 +0200
          Re: [lkp] [nfsd] 4aac1bf05b: -2.9% fsmark.files_per_sec Jeff Layton <jeff.layton@primarydata.com> - 2015-09-30 12:10 +0200
            Re: [lkp] [nfsd] 4aac1bf05b: -2.9% fsmark.files_per_sec Dave Chinner <david@fromorbit.com> - 2015-10-01 01:20 +0200
              Re: [lkp] [nfsd] 4aac1bf05b: -2.9% fsmark.files_per_sec "J. Bruce Fields" <bfields@fieldses.org> - 2015-10-01 03:00 +0200
              Re: [lkp] [nfsd] 4aac1bf05b: -2.9% fsmark.files_per_sec Jeff Layton <jeff.layton@primarydata.com> - 2015-10-01 13:40 +0200

#1234993 — Re: [lkp] [nfsd] 4aac1bf05b: -2.9% fsmark.files_per_sec

On Mon, 28 Sep 2015 14:49:32 +0800
kernel test robot <ying.huang@intel.com> wrote:

> FYI, we noticed the below changes on
> 
> =========================================================================================
> tbox_group/testcase/rootfs/kconfig/compiler/cpufreq_governor/iterations/nr_threads/disk/fs/fs2/filesize/test_size/sync_method/nr_directories/nr_files_per_directory:
>   lkp-ne04/fsmark/debian-x86_64-2015-02-07.cgz/x86_64-rhel/gcc-4.9/performance/1x/32t/1HDD/xfs/nfsv4/5K/400M/fsyncBeforeClose/16d/256fpd
> 
> commit: 
>   cd2d35ff27c4fda9ba73b0aa84313e8e20ce4d2c
>   4aac1bf05b053a201a4b392dd9a684fb2b7e6103
> 

A question...

I think my tree should now contain a fix for this, but with a
performance regression like this it's difficult to know for sure.

Is there some (automated) way to request that the KTR redo this test?
If not, will I get a note saying "problem seems to now be fixed" or do
I just take a lack of further emails from the KTR about this as a sign
that it's resolved?

Thanks!
-- 
Jeff Layton <jeff.layton@primarydata.com>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1235632

Jeff Layton <jeff.layton@primarydata.com> writes:

> On Mon, 28 Sep 2015 14:49:32 +0800
> kernel test robot <ying.huang@intel.com> wrote:
>
>> FYI, we noticed the below changes on
>> 
>> =========================================================================================
>> tbox_group/testcase/rootfs/kconfig/compiler/cpufreq_governor/iterations/nr_threads/disk/fs/fs2/filesize/test_size/sync_method/nr_directories/nr_files_per_directory:
>>   lkp-ne04/fsmark/debian-x86_64-2015-02-07.cgz/x86_64-rhel/gcc-4.9/performance/1x/32t/1HDD/xfs/nfsv4/5K/400M/fsyncBeforeClose/16d/256fpd
>> 
>> commit: 
>>   cd2d35ff27c4fda9ba73b0aa84313e8e20ce4d2c
>>   4aac1bf05b053a201a4b392dd9a684fb2b7e6103
>> 
>
> A question...
>
> I think my tree should now contain a fix for this, but with a
> performance regression like this it's difficult to know for sure.
>
> Is there some (automated) way to request that the KTR redo this test?
> If not, will I get a note saying "problem seems to now be fixed" or do
> I just take a lack of further emails from the KTR about this as a sign
> that it's resolved?

Can you provide the branch name and commit ID for your tree with fix?  I
can confirm whether it is fixed for you.

Best Regards,
Huang, Ying
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1235647

On Wed, 30 Sep 2015 07:27:54 +0800
"Huang\, Ying" <ying.huang@linux.intel.com> wrote:

> Jeff Layton <jeff.layton@primarydata.com> writes:
> 
> > On Mon, 28 Sep 2015 14:49:32 +0800
> > kernel test robot <ying.huang@intel.com> wrote:
> >
> >> FYI, we noticed the below changes on
> >> 
> >> =========================================================================================
> >> tbox_group/testcase/rootfs/kconfig/compiler/cpufreq_governor/iterations/nr_threads/disk/fs/fs2/filesize/test_size/sync_method/nr_directories/nr_files_per_directory:
> >>   lkp-ne04/fsmark/debian-x86_64-2015-02-07.cgz/x86_64-rhel/gcc-4.9/performance/1x/32t/1HDD/xfs/nfsv4/5K/400M/fsyncBeforeClose/16d/256fpd
> >> 
> >> commit: 
> >>   cd2d35ff27c4fda9ba73b0aa84313e8e20ce4d2c
> >>   4aac1bf05b053a201a4b392dd9a684fb2b7e6103
> >> 
> >
> > A question...
> >
> > I think my tree should now contain a fix for this, but with a
> > performance regression like this it's difficult to know for sure.
> >
> > Is there some (automated) way to request that the KTR redo this test?
> > If not, will I get a note saying "problem seems to now be fixed" or do
> > I just take a lack of further emails from the KTR about this as a sign
> > that it's resolved?
> 
> Can you provide the branch name and commit ID for your tree with fix?  I
> can confirm whether it is fixed for you.
> 
Sure:

git://git.samba.org/jlayton/linux.git nfsd-4.4

The tip commit is ed3d7c1e01a76f5ecc7444067704a82af4c2f76e.

Many thanks!
-- 
Jeff Layton <jeff.layton@primarydata.com>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1235896

Jeff Layton <jeff.layton@primarydata.com> writes:

> On Wed, 30 Sep 2015 07:27:54 +0800
> "Huang\, Ying" <ying.huang@linux.intel.com> wrote:
>
>> Jeff Layton <jeff.layton@primarydata.com> writes:
>> 
>> > On Mon, 28 Sep 2015 14:49:32 +0800
>> > kernel test robot <ying.huang@intel.com> wrote:
>> >
>> >> FYI, we noticed the below changes on
>> >> 
>> >> =========================================================================================
>> >> tbox_group/testcase/rootfs/kconfig/compiler/cpufreq_governor/iterations/nr_threads/disk/fs/fs2/filesize/test_size/sync_method/nr_directories/nr_files_per_directory:
>> >>   lkp-ne04/fsmark/debian-x86_64-2015-02-07.cgz/x86_64-rhel/gcc-4.9/performance/1x/32t/1HDD/xfs/nfsv4/5K/400M/fsyncBeforeClose/16d/256fpd
>> >> 
>> >> commit: 
>> >>   cd2d35ff27c4fda9ba73b0aa84313e8e20ce4d2c
>> >>   4aac1bf05b053a201a4b392dd9a684fb2b7e6103
>> >> 
>> >
>> > A question...
>> >
>> > I think my tree should now contain a fix for this, but with a
>> > performance regression like this it's difficult to know for sure.
>> >
>> > Is there some (automated) way to request that the KTR redo this test?
>> > If not, will I get a note saying "problem seems to now be fixed" or do
>> > I just take a lack of further emails from the KTR about this as a sign
>> > that it's resolved?
>> 
>> Can you provide the branch name and commit ID for your tree with fix?  I
>> can confirm whether it is fixed for you.
>> 
> Sure:
>
> git://git.samba.org/jlayton/linux.git nfsd-4.4
>
> The tip commit is ed3d7c1e01a76f5ecc7444067704a82af4c2f76e.
>

It seems that the regression is fixed at that commit.  Thanks!

=========================================================================================
tbox_group/testcase/rootfs/kconfig/compiler/cpufreq_governor/iterations/nr_threads/disk/fs/fs2/filesize/test_size/sync_method/nr_directories/nr_files_per_directory:
  lkp-ne04/fsmark/debian-x86_64-2015-02-07.cgz/x86_64-rhel/gcc-4.9/performance/1x/32t/1HDD/xfs/nfsv4/5K/400M/fsyncBeforeClose/16d/256fpd

commit: 
  cd2d35ff27c4fda9ba73b0aa84313e8e20ce4d2c
  4aac1bf05b053a201a4b392dd9a684fb2b7e6103
  ed3d7c1e01a76f5ecc7444067704a82af4c2f76e

cd2d35ff27c4fda9 4aac1bf05b053a201a4b392dd9 ed3d7c1e01a76f5ecc74440677 
---------------- -------------------------- -------------------------- 
         %stddev     %change         %stddev     %change         %stddev
             \          |                \          |                \  
  14415356 ±  0%      +2.6%   14788625 ±  1%      +4.1%   15008301 ±  0%  fsmark.app_overhead
    441.60 ±  0%      -2.9%     428.80 ±  0%      -0.4%     439.68 ±  0%  fsmark.files_per_sec
    185.78 ±  0%      +2.9%     191.26 ±  0%      +0.3%     186.37 ±  0%  fsmark.time.elapsed_time
    185.78 ±  0%      +2.9%     191.26 ±  0%      +0.3%     186.37 ±  0%  fsmark.time.elapsed_time.max
     97472 ±  0%      -2.8%      94713 ±  0%      -0.8%      96657 ±  0%  fsmark.time.involuntary_context_switches

Best Regards,
Huang, Ying
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1235992

On Wed, 30 Sep 2015 16:35:58 +0800
"Huang\, Ying" <ying.huang@linux.intel.com> wrote:

> Jeff Layton <jeff.layton@primarydata.com> writes:
> 
> > On Wed, 30 Sep 2015 07:27:54 +0800
> > "Huang\, Ying" <ying.huang@linux.intel.com> wrote:
> >
> >> Jeff Layton <jeff.layton@primarydata.com> writes:
> >> 
> >> > On Mon, 28 Sep 2015 14:49:32 +0800
> >> > kernel test robot <ying.huang@intel.com> wrote:
> >> >
> >> >> FYI, we noticed the below changes on
> >> >> 
> >> >> =========================================================================================
> >> >> tbox_group/testcase/rootfs/kconfig/compiler/cpufreq_governor/iterations/nr_threads/disk/fs/fs2/filesize/test_size/sync_method/nr_directories/nr_files_per_directory:
> >> >>   lkp-ne04/fsmark/debian-x86_64-2015-02-07.cgz/x86_64-rhel/gcc-4.9/performance/1x/32t/1HDD/xfs/nfsv4/5K/400M/fsyncBeforeClose/16d/256fpd
> >> >> 
> >> >> commit: 
> >> >>   cd2d35ff27c4fda9ba73b0aa84313e8e20ce4d2c
> >> >>   4aac1bf05b053a201a4b392dd9a684fb2b7e6103
> >> >> 
> >> >
> >> > A question...
> >> >
> >> > I think my tree should now contain a fix for this, but with a
> >> > performance regression like this it's difficult to know for sure.
> >> >
> >> > Is there some (automated) way to request that the KTR redo this test?
> >> > If not, will I get a note saying "problem seems to now be fixed" or do
> >> > I just take a lack of further emails from the KTR about this as a sign
> >> > that it's resolved?
> >> 
> >> Can you provide the branch name and commit ID for your tree with fix?  I
> >> can confirm whether it is fixed for you.
> >> 
> > Sure:
> >
> > git://git.samba.org/jlayton/linux.git nfsd-4.4
> >
> > The tip commit is ed3d7c1e01a76f5ecc7444067704a82af4c2f76e.
> >
> 
> It seems that the regression is fixed at that commit.  Thanks!
> 
> =========================================================================================
> tbox_group/testcase/rootfs/kconfig/compiler/cpufreq_governor/iterations/nr_threads/disk/fs/fs2/filesize/test_size/sync_method/nr_directories/nr_files_per_directory:
>   lkp-ne04/fsmark/debian-x86_64-2015-02-07.cgz/x86_64-rhel/gcc-4.9/performance/1x/32t/1HDD/xfs/nfsv4/5K/400M/fsyncBeforeClose/16d/256fpd
> 
> commit: 
>   cd2d35ff27c4fda9ba73b0aa84313e8e20ce4d2c
>   4aac1bf05b053a201a4b392dd9a684fb2b7e6103
>   ed3d7c1e01a76f5ecc7444067704a82af4c2f76e
> 
> cd2d35ff27c4fda9 4aac1bf05b053a201a4b392dd9 ed3d7c1e01a76f5ecc74440677 
> ---------------- -------------------------- -------------------------- 
>          %stddev     %change         %stddev     %change         %stddev
>              \          |                \          |                \  
>   14415356 ±  0%      +2.6%   14788625 ±  1%      +4.1%   15008301 ±  0%  fsmark.app_overhead
>     441.60 ±  0%      -2.9%     428.80 ±  0%      -0.4%     439.68 ±  0%  fsmark.files_per_sec
>     185.78 ±  0%      +2.9%     191.26 ±  0%      +0.3%     186.37 ±  0%  fsmark.time.elapsed_time
>     185.78 ±  0%      +2.9%     191.26 ±  0%      +0.3%     186.37 ±  0%  fsmark.time.elapsed_time.max
>      97472 ±  0%      -2.8%      94713 ±  0%      -0.8%      96657 ±  0%  fsmark.time.involuntary_context_switches
> 
> Best Regards,
> Huang, Ying

Thanks for testing it and catching the problem in the first place!

FWIW, the problem seems to have been bad hash distribution generated by
hash_ptr on struct inode pointers. When the cache had ~10000 entries in
it total, one of the hash chains had almost 2000 entries. When I
switched to hashing on inode->i_ino, the distribution was much better.

I'm not sure if it was just rotten luck or there is something about
inode pointers that makes hash_ptr generate a lot of duplicates. That
really could use more investigation...

-- 
Jeff Layton <jeff.layton@primarydata.com>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1236851

On Wed, Sep 30, 2015 at 06:03:59AM -0400, Jeff Layton wrote:
> Thanks for testing it and catching the problem in the first place!
> 
> FWIW, the problem seems to have been bad hash distribution generated by
> hash_ptr on struct inode pointers. When the cache had ~10000 entries in
> it total, one of the hash chains had almost 2000 entries. When I
> switched to hashing on inode->i_ino, the distribution was much better.
> 
> I'm not sure if it was just rotten luck or there is something about
> inode pointers that makes hash_ptr generate a lot of duplicates. That
> really could use more investigation...

Inode pointers have no entropy in the lower 9-10 bits because of
their size, and being allocated from a slab they are all going to
have the same set of values in the next 3-4 bits (i.e. offset into
the slab page which is defined by sizeof(inode)).  Pointers also
have very similar upper bits, too, because they are all in kernel
memory.

hash_64 trys to fold all the entropy from the lower bits into into
the upper bits and then takes the result from the upper bits. Hence
if there is no entropy in either the lower or upper bits to start
with, then the hash may not end up with much entropy in it at all...

FWIW, see fs/inode.c::hash() to see how the fs code hashes inode
numbers (called from insert_inode_hash()). It's very different
because because inode numbers have the majority of their entropy in
the lower bits and (usually) none in the upper bits...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1236886

On Thu, Oct 01, 2015 at 09:17:42AM +1000, Dave Chinner wrote:
> Inode pointers have no entropy in the lower 9-10 bits because of
> their size, and being allocated from a slab they are all going to
> have the same set of values in the next 3-4 bits (i.e. offset into
> the slab page which is defined by sizeof(inode)).  Pointers also
> have very similar upper bits, too, because they are all in kernel
> memory.
> 
> hash_64 trys to fold all the entropy from the lower bits into into
> the upper bits and then takes the result from the upper bits. Hence
> if there is no entropy in either the lower or upper bits to start
> with, then the hash may not end up with much entropy in it at all...

So we have something hash_ptr() that turns out to be terrible at hashing
pointers?  Argh.

(I understand you're saying this isn't necessarily the case for all
pointers, but inode pointers on their own seem likely to be a common
case, and there must be many more that are similar.)

--b.

> 
> FWIW, see fs/inode.c::hash() to see how the fs code hashes inode
> numbers (called from insert_inode_hash()). It's very different
> because because inode numbers have the majority of their entropy in
> the lower bits and (usually) none in the upper bits...
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1237298

On Thu, 1 Oct 2015 09:17:42 +1000
Dave Chinner <david@fromorbit.com> wrote:

> On Wed, Sep 30, 2015 at 06:03:59AM -0400, Jeff Layton wrote:
> > Thanks for testing it and catching the problem in the first place!
> > 
> > FWIW, the problem seems to have been bad hash distribution generated by
> > hash_ptr on struct inode pointers. When the cache had ~10000 entries in
> > it total, one of the hash chains had almost 2000 entries. When I
> > switched to hashing on inode->i_ino, the distribution was much better.
> > 
> > I'm not sure if it was just rotten luck or there is something about
> > inode pointers that makes hash_ptr generate a lot of duplicates. That
> > really could use more investigation...
> 
> Inode pointers have no entropy in the lower 9-10 bits because of
> their size, and being allocated from a slab they are all going to
> have the same set of values in the next 3-4 bits (i.e. offset into
> the slab page which is defined by sizeof(inode)).  Pointers also
> have very similar upper bits, too, because they are all in kernel
> memory.
> 
> hash_64 trys to fold all the entropy from the lower bits into into
> the upper bits and then takes the result from the upper bits. Hence
> if there is no entropy in either the lower or upper bits to start
> with, then the hash may not end up with much entropy in it at all...
> 
> FWIW, see fs/inode.c::hash() to see how the fs code hashes inode
> numbers (called from insert_inode_hash()). It's very different
> because because inode numbers have the majority of their entropy in
> the lower bits and (usually) none in the upper bits...
> 

Thanks for the explanation, Dave. That makes sense.

In hindsight I should have looked at how the vfs code hashes inodes in
its hashtable. Given that we're basically creating "shadow" inode
structures here that would probably work fairly well.
-- 
Jeff Layton <jeff.layton@primarydata.com>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web