Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1611021 > unrolled thread

Re: lockdep warning: console vs. mem hotplug

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: lockdep warning: console vs. mem hotplug Michal Hocko <mhocko@kernel.org> - 2017-03-28 16:30 +0200
    Re: lockdep warning: console vs. mem hotplug Petr Mladek <pmladek@suse.com> - 2017-03-28 18:10 +0200
      Re: lockdep warning: console vs. mem hotplug Michal Hocko <mhocko@kernel.org> - 2017-03-29 09:40 +0200
        Re: lockdep warning: console vs. mem hotplug Petr Mladek <pmladek@suse.com> - 2017-03-30 11:50 +0200
    Re: lockdep warning: console vs. mem hotplug Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2017-03-30 06:10 +0200

#1611021 — Re: lockdep warning: console vs. mem hotplug

On Sat 25-03-17 09:04:42, Sergey Senozhatsky wrote:
> On (03/21/17 13:44), Sergey Senozhatsky wrote:
> [..]
> > so we probably can
> > 
> > 
> > 1) move pr_info() out of zone->lock in __offline_isolated_pages().
> >    meh...
> > 
> > 
> > 2) switch to printk_deferred() in __offline_isolated_pages().
> >    meh.. there might a bunch of other printks done from under zone->lock.
> > 
> > 
> > 3) move add_timer() out of sclp_con_lock console in sclp_console_write().
> >    well, there can be other consoles that do something similar.
> > 
> > 
> > 4) ... something smart.
> 
> 
> Sebastian, does this change make lockdep happy?
> 
> it removes console drivers from the __offline_isolated_pages(). not the
> best solution I can think of, but the simplest one.
> 
> ---
> 
>  mm/page_alloc.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index f749b7ff7c50..eb61e6ab5f4f 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -7705,7 +7705,7 @@ __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn)
>  		BUG_ON(!PageBuddy(page));
>  		order = page_order(page);
>  #ifdef CONFIG_DEBUG_VM
> -		pr_info("remove from free list %lx %d %lx\n",
> +		printk_deferred(KERN_INFO "remove from free list %lx %d %lx\n",
>  			pfn, 1 << order, end_pfn);
>  #endif
>  		list_del(&page->lru);

I believe this is not a proper fix. Although this code is ugly and maybe
it doesn't really need zone->lock because that should be the page
allocator internal thing the problem is that printk shouldn't impose
such a subtle dependency on locks. Why does the timer needs to allocate
at all?

-- 
Michal Hocko
SUSE Labs

[toc] | [next] | [standalone]

#1611147

On Tue 2017-03-28 16:22:27, Michal Hocko wrote:
> On Sat 25-03-17 09:04:42, Sergey Senozhatsky wrote:
> > On (03/21/17 13:44), Sergey Senozhatsky wrote:
> > [..]
> > > so we probably can
> > > 
> > > 
> > > 1) move pr_info() out of zone->lock in __offline_isolated_pages().
> > >    meh...
> > > 
> > > 
> > > 2) switch to printk_deferred() in __offline_isolated_pages().
> > >    meh.. there might a bunch of other printks done from under zone->lock.
> > > 
> > > 
> > > 3) move add_timer() out of sclp_con_lock console in sclp_console_write().
> > >    well, there can be other consoles that do something similar.
> > > 
> > > 
> > > 4) ... something smart.
> > 
> > 
> > Sebastian, does this change make lockdep happy?
> > 
> > it removes console drivers from the __offline_isolated_pages(). not the
> > best solution I can think of, but the simplest one.
> > 
> > ---
> > 
> >  mm/page_alloc.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > index f749b7ff7c50..eb61e6ab5f4f 100644
> > --- a/mm/page_alloc.c
> > +++ b/mm/page_alloc.c
> > @@ -7705,7 +7705,7 @@ __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn)
> >  		BUG_ON(!PageBuddy(page));
> >  		order = page_order(page);
> >  #ifdef CONFIG_DEBUG_VM
> > -		pr_info("remove from free list %lx %d %lx\n",
> > +		printk_deferred(KERN_INFO "remove from free list %lx %d %lx\n",
> >  			pfn, 1 << order, end_pfn);
> >  #endif
> >  		list_del(&page->lru);
> 
> I believe this is not a proper fix. Although this code is ugly and maybe
> it doesn't really need zone->lock because that should be the page
> allocator internal thing the problem is that printk shouldn't impose
> such a subtle dependency on locks. Why does the timer needs to allocate
> at all?

printk/console use timers to postpone flushing of buffers. There are
often more consequent printks. The code wants to wait a bit and flush
them together eventually. At the same time, it wants to set a deadline
for the flushing. It makes sure that they will get flushed in a
reasonable time even when the buffer is not full. It is questionable
but it makes some sense.

In each case, the timer code is used also by scheduler and we probably
need to use scheduler from printk.


Regarding the timer code. The problem seems to be with static
timers. They call debug_object_init() when the timer is used
for the first time. See the special handling of not-found
objects in debug_object_activate().

Now, __debug_object_init() calls fill_pool() that allocates
the memory. A solution would be to either use static
struct kmem_cache for statically defined timers and
avoid the allocation. Or we should call fill_pool()
asynchronously from a safe context.

What do you think?

Best Regards,
Petr

[toc] | [prev] | [next] | [standalone]

#1611659

On Tue 28-03-17 18:00:16, Petr Mladek wrote:
> On Tue 2017-03-28 16:22:27, Michal Hocko wrote:
> > On Sat 25-03-17 09:04:42, Sergey Senozhatsky wrote:
> > > On (03/21/17 13:44), Sergey Senozhatsky wrote:
> > > [..]
> > > > so we probably can
> > > > 
> > > > 
> > > > 1) move pr_info() out of zone->lock in __offline_isolated_pages().
> > > >    meh...
> > > > 
> > > > 
> > > > 2) switch to printk_deferred() in __offline_isolated_pages().
> > > >    meh.. there might a bunch of other printks done from under zone->lock.
> > > > 
> > > > 
> > > > 3) move add_timer() out of sclp_con_lock console in sclp_console_write().
> > > >    well, there can be other consoles that do something similar.
> > > > 
> > > > 
> > > > 4) ... something smart.
> > > 
> > > 
> > > Sebastian, does this change make lockdep happy?
> > > 
> > > it removes console drivers from the __offline_isolated_pages(). not the
> > > best solution I can think of, but the simplest one.
> > > 
> > > ---
> > > 
> > >  mm/page_alloc.c | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > > index f749b7ff7c50..eb61e6ab5f4f 100644
> > > --- a/mm/page_alloc.c
> > > +++ b/mm/page_alloc.c
> > > @@ -7705,7 +7705,7 @@ __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn)
> > >  		BUG_ON(!PageBuddy(page));
> > >  		order = page_order(page);
> > >  #ifdef CONFIG_DEBUG_VM
> > > -		pr_info("remove from free list %lx %d %lx\n",
> > > +		printk_deferred(KERN_INFO "remove from free list %lx %d %lx\n",
> > >  			pfn, 1 << order, end_pfn);
> > >  #endif
> > >  		list_del(&page->lru);
> > 
> > I believe this is not a proper fix. Although this code is ugly and maybe
> > it doesn't really need zone->lock because that should be the page
> > allocator internal thing the problem is that printk shouldn't impose
> > such a subtle dependency on locks. Why does the timer needs to allocate
> > at all?
> 
> printk/console use timers to postpone flushing of buffers. There are
> often more consequent printks. The code wants to wait a bit and flush
> them together eventually. At the same time, it wants to set a deadline
> for the flushing. It makes sure that they will get flushed in a
> reasonable time even when the buffer is not full. It is questionable
> but it makes some sense.
> 
> In each case, the timer code is used also by scheduler and we probably
> need to use scheduler from printk.
> 
> Regarding the timer code. The problem seems to be with static
> timers. They call debug_object_init() when the timer is used
> for the first time. See the special handling of not-found
> objects in debug_object_activate().

Thanks for the clarification!

> Now, __debug_object_init() calls fill_pool() that allocates
> the memory. A solution would be to either use static
> struct kmem_cache

I am not sure what do you mean by that.

> for statically defined timers and avoid the allocation. Or we should
> call fill_pool() asynchronously from a safe context.

I think we should avoid the allocation completely. It is GFP_ATOMIC and
so likely to fail under heavy memory pressure. Async fill will make it
slightly more complicated but still unreliable.
 
> What do you think?

Why cannot we simply embed this debugging data into the timer itself?
It will make the structure larger (I didn't check how much) but this is
an opt in feature so it should be acceptable.

A subtle dependecny on the allocator is really bad and we should get rid
of it.
-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1612796

On Wed 2017-03-29 09:31:47, Michal Hocko wrote:
> On Tue 28-03-17 18:00:16, Petr Mladek wrote:
> > On Tue 2017-03-28 16:22:27, Michal Hocko wrote:
> > > On Sat 25-03-17 09:04:42, Sergey Senozhatsky wrote:
> > > > On (03/21/17 13:44), Sergey Senozhatsky wrote:
> > > > [..]
> > > > > so we probably can
> > > > > 
> > > > > 
> > > > > 1) move pr_info() out of zone->lock in __offline_isolated_pages().
> > > > >    meh...
> > > > > 
> > > > > 
> > > > > 2) switch to printk_deferred() in __offline_isolated_pages().
> > > > >    meh.. there might a bunch of other printks done from under zone->lock.
> > > > > 
> > > > > 
> > > > > 3) move add_timer() out of sclp_con_lock console in sclp_console_write().
> > > > >    well, there can be other consoles that do something similar.
> > > > > 
> > > > > 
> > > > > 4) ... something smart.
> > > > 
> > > > 
> > Regarding the timer code. The problem seems to be with static
> > timers. They call debug_object_init() when the timer is used
> > for the first time. See the special handling of not-found
> > objects in debug_object_activate().
> 
> Thanks for the clarification!
> 
> > Now, __debug_object_init() calls fill_pool() that allocates
> > the memory. A solution would be to either use static
> > struct kmem_cache
> 
> I am not sure what do you mean by that.

The problem is when the timer is defined using
DEFINE_TIMER() and initialized using __TIMER_INITIALIZER().

I had an unclear idea about extending the macro to define
also the needed structure that might later be used by
debug_object_init() instead of taking it from the pool.

I am sorry for the confusion with struct kmem_cache.
It is the type of the pointer that is passed when
allocating the memory in fill_pool(). It seems
we actually need struct debug_obj. But it might
be even more complicated.


> > for statically defined timers and avoid the allocation. Or we should
> > call fill_pool() asynchronously from a safe context.
> 
> I think we should avoid the allocation completely. It is GFP_ATOMIC and
> so likely to fail under heavy memory pressure. Async fill will make it
> slightly more complicated but still unreliable.
>  
> > What do you think?
> 
> Why cannot we simply embed this debugging data into the timer itself?
> It will make the structure larger (I didn't check how much) but this is
> an opt in feature so it should be acceptable.

This sounds reasonable to me.

> A subtle dependecny on the allocator is really bad and
> we should get rid of it.

Yup.

Best Regards,
Petr

[toc] | [prev] | [next] | [standalone]

#1612547

On (03/28/17 16:22), Michal Hocko wrote:
[..]
> > Sebastian, does this change make lockdep happy?
> > 
> > it removes console drivers from the __offline_isolated_pages(). not the
> > best solution I can think of, but the simplest one.
> > 
> > ---
> > 
> >  mm/page_alloc.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > index f749b7ff7c50..eb61e6ab5f4f 100644
> > --- a/mm/page_alloc.c
> > +++ b/mm/page_alloc.c
> > @@ -7705,7 +7705,7 @@ __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn)
> >  		BUG_ON(!PageBuddy(page));
> >  		order = page_order(page);
> >  #ifdef CONFIG_DEBUG_VM
> > -		pr_info("remove from free list %lx %d %lx\n",
> > +		printk_deferred(KERN_INFO "remove from free list %lx %d %lx\n",
> >  			pfn, 1 << order, end_pfn);
> >  #endif
> >  		list_del(&page->lru);
> 
> I believe this is not a proper fix.

oh, absolutely. I hate it. didn't really propose it as a fix. mostly
did it just to verify that there are no other lock inversions behind
the one that has been reported (lockdep turns off itself when it
detects the first lock dependency inversion).

> Although this code is ugly and maybe it doesn't really need zone->lock
> because that should be the page allocator internal thing the problem is
> that printk shouldn't impose such a subtle dependency on locks. Why does
> the timer needs to allocate at all?

I believe Petr has answered your questions. sorry for the delay.

	-ss

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web