Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1450659 > unrolled thread

[RFC PATCH] mm/hugetlb: Avoid soft lockup in set_max_huge_pages()

Back to article view | Back to linux.kernel

Contents

  [RFC PATCH] mm/hugetlb: Avoid soft lockup in set_max_huge_pages() Jia He <hejianet@gmail.com> - 2016-07-26 17:50 +0200
    Re: [RFC PATCH] mm/hugetlb: Avoid soft lockup in set_max_huge_pages() Dave Hansen <dave.hansen@linux.intel.com> - 2016-07-26 18:00 +0200
      Re: [RFC PATCH] mm/hugetlb: Avoid soft lockup in set_max_huge_pages() hejianet <hejianet@gmail.com> - 2016-07-26 18:40 +0200
      Re: [RFC PATCH] mm/hugetlb: Avoid soft lockup in set_max_huge_pages() hejianet <hejianet@gmail.com> - 2016-07-27 03:50 +0200
        Re: [RFC PATCH] mm/hugetlb: Avoid soft lockup in set_max_huge_pages() Dave Hansen <dave.hansen@linux.intel.com> - 2016-07-27 17:30 +0200

#1450659 — [RFC PATCH] mm/hugetlb: Avoid soft lockup in set_max_huge_pages()

In large memory(32TB) powerpc servers, we watched several soft lockup under
stress tests.
The call trace are as follows:
1.
get_page_from_freelist+0x2d8/0xd50  
__alloc_pages_nodemask+0x180/0xc20  
alloc_fresh_huge_page+0xb0/0x190    
set_max_huge_pages+0x164/0x3b0      

2.
prep_new_huge_page+0x5c/0x100             
alloc_fresh_huge_page+0xc8/0x190          
set_max_huge_pages+0x164/0x3b0

This patch is to fix such soft lockup. I thouhgt it is safe to call 
cond_resched() because alloc_fresh_gigantic_page and alloc_fresh_huge_page 
are out of spin_lock/unlock section.

Signed-off-by: Jia He <hejianet@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
Cc: Mike Kravetz <mike.kravetz@oracle.com>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>

---
 mm/hugetlb.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index addfe4ac..d51759d 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1146,6 +1146,10 @@ static int alloc_fresh_gigantic_page(struct hstate *h,
 
 	for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) {
 		page = alloc_fresh_gigantic_page_node(h, node);
+
+		/* yield cpu */
+		cond_resched();
+
 		if (page)
 			return 1;
 	}
@@ -1381,6 +1385,10 @@ static int alloc_fresh_huge_page(struct hstate *h, nodemask_t *nodes_allowed)
 
 	for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) {
 		page = alloc_fresh_huge_page_node(h, node);
+
+		/* yield cpu */
+		cond_resched();
+
 		if (page) {
 			ret = 1;
 			break;
-- 
2.5.0

[toc] | [next] | [standalone]

#1450664

On 07/26/2016 08:44 AM, Jia He wrote:
> This patch is to fix such soft lockup. I thouhgt it is safe to call 
> cond_resched() because alloc_fresh_gigantic_page and alloc_fresh_huge_page 
> are out of spin_lock/unlock section.

Yikes.  So the call site for both the things you patch is this:

>         while (count > persistent_huge_pages(h)) {
...
>                 spin_unlock(&hugetlb_lock);
>                 if (hstate_is_gigantic(h))
>                         ret = alloc_fresh_gigantic_page(h, nodes_allowed);
>                 else
>                         ret = alloc_fresh_huge_page(h, nodes_allowed);
>                 spin_lock(&hugetlb_lock);

and you choose to patch both of the alloc_*() functions.  Why not just
fix it at the common call site?  Seems like that
spin_lock(&hugetlb_lock) could be a cond_resched_lock() which would fix
both cases.

Also, putting that cond_resched() inside the for_each_node*() loop is an
odd choice.  It seems to indicate that the loops can take a long time,
which really isn't the case.  The _loop_ isn't long, right?

[toc] | [prev] | [next] | [standalone]

#1450674

On 7/26/16 11:58 PM, Dave Hansen wrote:
> On 07/26/2016 08:44 AM, Jia He wrote:
>> This patch is to fix such soft lockup. I thouhgt it is safe to call
>> cond_resched() because alloc_fresh_gigantic_page and alloc_fresh_huge_page
>> are out of spin_lock/unlock section.
> Yikes.  So the call site for both the things you patch is this:
>
>>          while (count > persistent_huge_pages(h)) {
> ...
>>                  spin_unlock(&hugetlb_lock);
>>                  if (hstate_is_gigantic(h))
>>                          ret = alloc_fresh_gigantic_page(h, nodes_allowed);
>>                  else
>>                          ret = alloc_fresh_huge_page(h, nodes_allowed);
>>                  spin_lock(&hugetlb_lock);
> and you choose to patch both of the alloc_*() functions.  Why not just
> fix it at the common call site?  Seems like that
> spin_lock(&hugetlb_lock) could be a cond_resched_lock() which would fix
> both cases.
>
> Also, putting that cond_resched() inside the for_each_node*() loop is an
> odd choice.  It seems to indicate that the loops can take a long time,
> which really isn't the case.  The _loop_ isn't long, right?
Yes,thanks for the suggestions
Will send out V2 later

B.R.

[toc] | [prev] | [next] | [standalone]

#1450999

Hi Dave

On 7/26/16 11:58 PM, Dave Hansen wrote:
> On 07/26/2016 08:44 AM, Jia He wrote:
>> This patch is to fix such soft lockup. I thouhgt it is safe to call
>> cond_resched() because alloc_fresh_gigantic_page and alloc_fresh_huge_page
>> are out of spin_lock/unlock section.
> Yikes.  So the call site for both the things you patch is this:
>
>>          while (count > persistent_huge_pages(h)) {
> ...
>>                  spin_unlock(&hugetlb_lock);
>>                  if (hstate_is_gigantic(h))
>>                          ret = alloc_fresh_gigantic_page(h, nodes_allowed);
>>                  else
>>                          ret = alloc_fresh_huge_page(h, nodes_allowed);
>>                  spin_lock(&hugetlb_lock);
> and you choose to patch both of the alloc_*() functions.  Why not just
> fix it at the common call site?  Seems like that
> spin_lock(&hugetlb_lock) could be a cond_resched_lock() which would fix
> both cases.
I agree to move the cond_resched() to a common site in set_max_huge_pages().
But do you mean the spin_lock in this while loop can be replaced by
cond_resched_lock?
IIUC, cond_resched_lock = spin_unlock+cond_resched+spin_lock.
So could you please explain more details about it? Thanks.

B.R.
Justin
> Also, putting that cond_resched() inside the for_each_node*() loop is an
> odd choice.  It seems to indicate that the loops can take a long time,
> which really isn't the case.  The _loop_ isn't long, right?
>

[toc] | [prev] | [next] | [standalone]

#1451357

On 07/26/2016 06:39 PM, hejianet wrote:
>>>
>> and you choose to patch both of the alloc_*() functions.  Why not just
>> fix it at the common call site?  Seems like that
>> spin_lock(&hugetlb_lock) could be a cond_resched_lock() which would fix
>> both cases.
> I agree to move the cond_resched() to a common site in 
> set_max_huge_pages(). But do you mean the spin_lock in this while
> loop can be replaced by cond_resched_lock? IIUC, cond_resched_lock =
> spin_unlock+cond_resched+spin_lock. So could you please explain more
> details about it? Thanks.

Ahh, good point.  A plain cond_resched() outside the lock is probably
sufficient here.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web