Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1418925 > unrolled thread

[PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs in own kmem_caches

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  [PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs in own kmem_caches James Simmons <jsimmons@infradead.org> - 2016-06-10 00:50 +0200
    Re: [PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs in  own kmem_caches Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2016-06-10 03:30 +0200
      Re: [PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs  in own kmem_caches "Faccini, Bruno" <bruno.faccini@intel.com> - 2016-06-10 17:30 +0200
        Re: [PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs in  own kmem_caches Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2016-06-10 18:40 +0200
          Re: [PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs  in own kmem_caches James Simmons <jsimmons@infradead.org> - 2016-06-15 05:10 +0200
            Re: [PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs in  own kmem_caches Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2016-06-18 05:40 +0200

#1418925 — [PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs in own kmem_caches

From: Bruno Faccini <bruno.faccini@intel.com>

As part of LU-3848 and LU-4330, it has been discovered that LNET
MEs and small MDs (<=128 Bytes) are allocated in <size-128> kmem_cache
and thus can suffer quite frequent corruptions, from other modules or
Kernel parts, that occur there. To avoid this, MEs and small-MDs
specific kmem_cache have been created.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-4430
Reviewed-on: http://review.whamcloud.com/18586
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 .../staging/lustre/include/linux/lnet/lib-lnet.h   |   36 ++++++++++++++--
 drivers/staging/lustre/lnet/lnet/api-ni.c          |   45 ++++++++++++++++++++
 2 files changed, 77 insertions(+), 4 deletions(-)

diff --git a/drivers/staging/lustre/include/linux/lnet/lib-lnet.h b/drivers/staging/lustre/include/linux/lnet/lib-lnet.h
index 513a822..51ad729 100644
--- a/drivers/staging/lustre/include/linux/lnet/lib-lnet.h
+++ b/drivers/staging/lustre/include/linux/lnet/lib-lnet.h
@@ -178,6 +178,11 @@ lnet_net_lock_current(void)
 
 #define MAX_PORTALS		64
 
+#define LNET_SMALL_MD_SIZE	offsetof(lnet_libmd_t, md_iov.iov[1])
+extern struct kmem_cache *lnet_mes_cachep;	/* MEs kmem_cache */
+extern struct kmem_cache *lnet_small_mds_cachep;/* <= LNET_SMALL_MD_SIZE bytes
+						 * MDs kmem_cache
+						 */
 static inline lnet_eq_t *
 lnet_eq_alloc(void)
 {
@@ -208,7 +213,19 @@ lnet_md_alloc(lnet_md_t *umd)
 		size = offsetof(lnet_libmd_t, md_iov.iov[niov]);
 	}
 
-	LIBCFS_ALLOC(md, size);
+	if (size <= LNET_SMALL_MD_SIZE) {
+		md = kmem_cache_alloc(lnet_small_mds_cachep,
+				      GFP_NOFS | __GFP_ZERO);
+		if (md) {
+			CDEBUG(D_MALLOC, "slab-alloced 'md' of size %u at %p.\n",
+			       size, md);
+		} else {
+			CDEBUG(D_MALLOC, "failed to allocate 'md' of size %u\n",
+			       size);
+		}
+	} else {
+		LIBCFS_ALLOC(md, size);
+	}
 
 	if (md) {
 		/* Set here in case of early free */
@@ -230,7 +247,12 @@ lnet_md_free(lnet_libmd_t *md)
 	else
 		size = offsetof(lnet_libmd_t, md_iov.iov[md->md_niov]);
 
-	LIBCFS_FREE(md, size);
+	if (size <= LNET_SMALL_MD_SIZE) {
+		CDEBUG(D_MALLOC, "slab-freed 'md' at %p.\n", md);
+		kmem_cache_free(lnet_small_mds_cachep, md);
+	} else {
+		LIBCFS_FREE(md, size);
+	}
 }
 
 static inline lnet_me_t *
@@ -238,14 +260,20 @@ lnet_me_alloc(void)
 {
 	lnet_me_t *me;
 
-	LIBCFS_ALLOC(me, sizeof(*me));
+	me = kmem_cache_alloc(lnet_mes_cachep, GFP_NOFS | __GFP_ZERO);
+	if (me)
+		CDEBUG(D_MALLOC, "slab-alloced 'me' at %p.\n", me);
+	else
+		CDEBUG(D_MALLOC, "failed to allocate 'me'\n");
+
 	return me;
 }
 
 static inline void
 lnet_me_free(lnet_me_t *me)
 {
-	LIBCFS_FREE(me, sizeof(*me));
+	CDEBUG(D_MALLOC, "slab-freed 'me' at %p.\n", me);
+	kmem_cache_free(lnet_mes_cachep, me);
 }
 
 static inline lnet_msg_t *
diff --git a/drivers/staging/lustre/lnet/lnet/api-ni.c b/drivers/staging/lustre/lnet/lnet/api-ni.c
index fe0dbe7..9db0ff1 100644
--- a/drivers/staging/lustre/lnet/lnet/api-ni.c
+++ b/drivers/staging/lustre/lnet/lnet/api-ni.c
@@ -103,6 +103,46 @@ lnet_init_locks(void)
 	mutex_init(&the_lnet.ln_api_mutex);
 }
 
+struct kmem_cache *lnet_mes_cachep;		/* MEs kmem_cache */
+struct kmem_cache *lnet_small_mds_cachep;	/* <= LNET_SMALL_MD_SIZE bytes
+						 *  MDs kmem_cache
+						 */
+static int
+lnet_descriptor_setup(void)
+{
+	/*
+	 * create specific kmem_cache for MEs and small MDs (i.e., originally
+	 * allocated in <size-xxx> kmem_cache).
+	 */
+	lnet_mes_cachep = kmem_cache_create("lnet_MEs", sizeof(lnet_me_t),
+					    0, 0, NULL);
+	if (!lnet_mes_cachep)
+		return -ENOMEM;
+
+	lnet_small_mds_cachep = kmem_cache_create("lnet_small_MDs",
+						  LNET_SMALL_MD_SIZE, 0, 0,
+						  NULL);
+	if (!lnet_small_mds_cachep)
+		return -ENOMEM;
+
+	return 0;
+}
+
+static void
+lnet_descriptor_cleanup(void)
+{
+
+	if (lnet_small_mds_cachep) {
+		kmem_cache_destroy(lnet_small_mds_cachep);
+		lnet_small_mds_cachep = NULL;
+	}
+
+	if (lnet_mes_cachep) {
+		kmem_cache_destroy(lnet_mes_cachep);
+		lnet_mes_cachep = NULL;
+	}
+}
+
 static int
 lnet_create_remote_nets_table(void)
 {
@@ -553,6 +593,10 @@ lnet_prepare(lnet_pid_t requested_pid)
 	INIT_LIST_HEAD(&the_lnet.ln_drop_rules);
 	INIT_LIST_HEAD(&the_lnet.ln_delay_rules);
 
+	rc = lnet_descriptor_setup();
+	if (rc)
+		goto failed;
+
 	rc = lnet_create_remote_nets_table();
 	if (rc)
 		goto failed;
@@ -652,6 +696,7 @@ lnet_unprepare(void)
 		the_lnet.ln_counters = NULL;
 	}
 	lnet_destroy_remote_nets_table();
+	lnet_descriptor_cleanup();
 
 	return 0;
 }
-- 
1.7.1

[toc] | [next] | [standalone]

#1418978 — Re: [PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs in own kmem_caches

On Thu, Jun 09, 2016 at 06:45:46PM -0400, James Simmons wrote:
> From: Bruno Faccini <bruno.faccini@intel.com>
> 
> As part of LU-3848 and LU-4330, it has been discovered that LNET
> MEs and small MDs (<=128 Bytes) are allocated in <size-128> kmem_cache
> and thus can suffer quite frequent corruptions, from other modules or
> Kernel parts, that occur there. To avoid this, MEs and small-MDs
> specific kmem_cache have been created.

What?  Who corrupts them?  That shouldn't be possible, and on some
systems, even if you do ask for a separate slab, it will be merged
togther with others of the same size.  So this patch doesn't do all that
much.

I think you are having some other problem here, changing to a separate
memory cache shouldn't solve corruption issues.

sorry,

greg k-h

[toc] | [prev] | [next] | [standalone]

#1419519 — Re: [PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs in own kmem_caches

Hello,
The intent of this patch is not to solve the corruptions for sure, but only to avoid the concerned MEs/small-MDs LNet structs to be quite frequently impacted due to their high allocation/free rate. This may also possibly help to save cycles due to high usage and contention when using a generic kmem_cache (when they stay separate from others, thanks for the precision!).
Bye,
Bruno.

> Le Jun 10, 2016 à 03:28, Greg Kroah-Hartman <gregkh@linuxfoundation.org> a écrit :
> 
> On Thu, Jun 09, 2016 at 06:45:46PM -0400, James Simmons wrote:
>> From: Bruno Faccini <bruno.faccini@intel.com>
>> 
>> As part of LU-3848 and LU-4330, it has been discovered that LNET
>> MEs and small MDs (<=128 Bytes) are allocated in <size-128> kmem_cache
>> and thus can suffer quite frequent corruptions, from other modules or
>> Kernel parts, that occur there. To avoid this, MEs and small-MDs
>> specific kmem_cache have been created.
> 
> What?  Who corrupts them?  That shouldn't be possible, and on some
> systems, even if you do ask for a separate slab, it will be merged
> togther with others of the same size.  So this patch doesn't do all that
> much.
> 
> I think you are having some other problem here, changing to a separate
> memory cache shouldn't solve corruption issues.
> 
> sorry,
> 
> greg k-h

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[toc] | [prev] | [next] | [standalone]

#1419575 — Re: [PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs in own kmem_caches

A: No.
Q: Should I include quotations after my reply?

http://daringfireball.net/2007/07/on_top

On Fri, Jun 10, 2016 at 03:25:28PM +0000, Faccini, Bruno wrote:
> Hello,
> The intent of this patch is not to solve the corruptions for sure, but
> only to avoid the concerned MEs/small-MDs LNet structs to be quite
> frequently impacted due to their high allocation/free rate.

But that's not what the patch description said :(

And again, putting them in a separate cache is not going to save much of
anything, given that your caches might have been merged together anyway.

> This may also possibly help to save cycles due to high usage and
> contention when using a generic kmem_cache (when they stay separate
> from others, thanks for the precision!).

Have you measured this?

This isn't applicable for 4.7-rc at this time, _unless_ it fixes a bug,
which is why I pushed back on this.  If you want your own cache for
these variables, fine, I don't care, but that makes it a 4.8-rc1 patch
instead.

hope that helps explain things better,

greg k-h

[toc] | [prev] | [next] | [standalone]

#1422523 — Re: [PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs in own kmem_caches

> > This may also possibly help to save cycles due to high usage and
> > contention when using a generic kmem_cache (when they stay separate
> > from others, thanks for the precision!).
> 
> Have you measured this?
> 
> This isn't applicable for 4.7-rc at this time, _unless_ it fixes a bug,
> which is why I pushed back on this.  If you want your own cache for
> these variables, fine, I don't care, but that makes it a 4.8-rc1 patch
> instead.
> 
> hope that helps explain things better,

As a side question when is the window to push patches of this class?
Is it when 4.7-rc7 is merged to staging? 

[toc] | [prev] | [next] | [standalone]

#1425627 — Re: [PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs in own kmem_caches

On Wed, Jun 15, 2016 at 04:02:40AM +0100, James Simmons wrote:
> 
> > > This may also possibly help to save cycles due to high usage and
> > > contention when using a generic kmem_cache (when they stay separate
> > > from others, thanks for the precision!).
> > 
> > Have you measured this?
> > 
> > This isn't applicable for 4.7-rc at this time, _unless_ it fixes a bug,
> > which is why I pushed back on this.  If you want your own cache for
> > these variables, fine, I don't care, but that makes it a 4.8-rc1 patch
> > instead.
> > 
> > hope that helps explain things better,
> 
> As a side question when is the window to push patches of this class?
> Is it when 4.7-rc7 is merged to staging? 

You can send them to me anytime, I'll queue them up in my "-next" branch
to be merged in the next merge window.  Like I do for almost all lustre
patches that aren't bugfixes or regressions.

But I think you have a bigger problem here that you need to debug, using
a separate cache isn't going to solve that bug, only postpone you
finding it...

thanks,

greg k-h

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web