Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1637889 > unrolled thread

get/put_cpu() usage in block/blk-mq.c

Back to article view | Back to linux.kernel

Contents

  get/put_cpu() usage in block/blk-mq.c Mike Galbraith <efault@gmx.de> - 2017-05-09 08:10 +0200
    Re: get/put_cpu() usage in block/blk-mq.c Mike Galbraith <efault@gmx.de> - 2017-05-09 17:10 +0200
      Re: get/put_cpu() usage in block/blk-mq.c Mike Galbraith <efault@gmx.de> - 2017-05-09 17:50 +0200
        Re: get/put_cpu() usage in block/blk-mq.c Mike Galbraith <efault@gmx.de> - 2017-05-09 18:30 +0200
          Re: get/put_cpu() usage in block/blk-mq.c Mike Galbraith <efault@gmx.de> - 2017-05-09 19:00 +0200

#1637889 — get/put_cpu() usage in block/blk-mq.c

Hi Jens,

I was about to fix up this splat..

[  445.022141] loop: module loaded
[  445.078116] BUG: using smp_processor_id() in preemptible [00000000] code: loop0/3801
[  445.085873] caller is debug_smp_processor_id+0x17/0x20
[  445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: G            E   4.12.0-default #40
[  445.098838] Hardware name: IBM System x3550 M3 -[7944K3G]-/69Y5698     , BIOS -[D6E150AUS-1.10]- 12/15/2010
[  445.108564] Call Trace:
[  445.111016]  dump_stack+0x65/0x89
[  445.114330]  check_preemption_disabled+0xde/0xf0
[  445.118945]  debug_smp_processor_id+0x17/0x20
[  445.123300]  blk_stat_add+0xb0/0x130
[  445.126876]  __blk_mq_complete_request+0xb5/0x150
[  445.131575]  blk_mq_complete_request+0x16/0x20
[  445.136020]  loop_queue_work+0x5f/0xaa0 [loop]
[  445.140461]  ? _raw_spin_unlock_irq+0x21/0x40
[  445.144816]  ? finish_task_switch+0x85/0x270
[  445.149085]  ? __schedule+0x291/0x8c0
[  445.152747]  kthread_worker_fn+0xc2/0x1d0
[  445.156754]  kthread+0x114/0x150
[  445.159983]  ? __kthread_init_worker+0xb0/0xb0
[  445.164424]  ? kthread_park+0x60/0x60
[  445.168085]  ret_from_fork+0x2c/0x40

..but then noticed that __blk_mq_run_hw_queue() is also called under
get_cpu(), but recently grew a might_sleep(), so maybe wants some block
layer eyeballs.

	-Mike

[toc] | [next] | [standalone]

#1638182

On Tue, 2017-05-09 at 08:53 -0600, Jens Axboe wrote:
> On 05/09/2017 12:07 AM, Mike Galbraith wrote:
> > Hi Jens,
> > 
> > I was about to fix up this splat..
> > 
> > [  445.022141] loop: module loaded
> > [  445.078116] BUG: using smp_processor_id() in preemptible [00000000] code: loop0/3801
> > [  445.085873] caller is debug_smp_processor_id+0x17/0x20
> > [  445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: G            E   4.12.0-default #40
> > [  445.098838] Hardware name: IBM System x3550 M3 -[7944K3G]-/69Y5698     , BIOS -[D6E150AUS-1.10]- 12/15/2010
> > [  445.108564] Call Trace:
> > [  445.111016]  dump_stack+0x65/0x89
> > [  445.114330]  check_preemption_disabled+0xde/0xf0
> > [  445.118945]  debug_smp_processor_id+0x17/0x20
> > [  445.123300]  blk_stat_add+0xb0/0x130
> > [  445.126876]  __blk_mq_complete_request+0xb5/0x150
> > [  445.131575]  blk_mq_complete_request+0x16/0x20
> > [  445.136020]  loop_queue_work+0x5f/0xaa0 [loop]
> > [  445.140461]  ? _raw_spin_unlock_irq+0x21/0x40
> > [  445.144816]  ? finish_task_switch+0x85/0x270
> > [  445.149085]  ? __schedule+0x291/0x8c0
> > [  445.152747]  kthread_worker_fn+0xc2/0x1d0
> > [  445.156754]  kthread+0x114/0x150
> > [  445.159983]  ? __kthread_init_worker+0xb0/0xb0
> > [  445.164424]  ? kthread_park+0x60/0x60
> > [  445.168085]  ret_from_fork+0x2c/0x40
> 
> Is it from this_cpu_ptr() in blk_stat_add()?

Yeah.

> > ..but then noticed that __blk_mq_run_hw_queue() is also called under
> > get_cpu(), but recently grew a might_sleep(), so maybe wants some block
> > layer eyeballs.
> 
> That part is fine. The might sleep depends on BLOCKING being set in
> the hardware queue flags. And if that is set, then we don't call
> it under get_cpu().

Ah, right.

	-Mike

[toc] | [prev] | [next] | [standalone]

#1638205

On Tue, 2017-05-09 at 09:15 -0600, Jens Axboe wrote:
> On 05/09/2017 09:13 AM, Jens Axboe wrote:
> > On 05/09/2017 09:04 AM, Mike Galbraith wrote:
> > > On Tue, 2017-05-09 at 08:53 -0600, Jens Axboe wrote:
> > > > On 05/09/2017 12:07 AM, Mike Galbraith wrote:
> > > > > Hi Jens,
> > > > > 
> > > > > I was about to fix up this splat..
> > > > > 
> > > > > [  445.022141] loop: module loaded
> > > > > [  445.078116] BUG: using smp_processor_id() in preemptible
> > > > > [00000000] code: loop0/3801
> > > > > [  445.085873] caller is debug_smp_processor_id+0x17/0x20
> > > > > [  445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: G       
> > > > >      E   4.12.0-default #40
> > > > > [  445.098838] Hardware name: IBM System x3550 M3 -[7944K3G]
> > > > > -/69Y5698     , BIOS -[D6E150AUS-1.10]- 12/15/2010
> > > > > [  445.108564] Call Trace:
> > > > > [  445.111016]  dump_stack+0x65/0x89
> > > > > [  445.114330]  check_preemption_disabled+0xde/0xf0
> > > > > [  445.118945]  debug_smp_processor_id+0x17/0x20
> > > > > [  445.123300]  blk_stat_add+0xb0/0x130
> > > > > [  445.126876]  __blk_mq_complete_request+0xb5/0x150
> > > > > [  445.131575]  blk_mq_complete_request+0x16/0x20
> > > > > [  445.136020]  loop_queue_work+0x5f/0xaa0 [loop]
> > > > > [  445.140461]  ? _raw_spin_unlock_irq+0x21/0x40
> > > > > [  445.144816]  ? finish_task_switch+0x85/0x270
> > > > > [  445.149085]  ? __schedule+0x291/0x8c0
> > > > > [  445.152747]  kthread_worker_fn+0xc2/0x1d0
> > > > > [  445.156754]  kthread+0x114/0x150
> > > > > [  445.159983]  ? __kthread_init_worker+0xb0/0xb0
> > > > > [  445.164424]  ? kthread_park+0x60/0x60
> > > > > [  445.168085]  ret_from_fork+0x2c/0x40
> > > > 
> > > > Is it from this_cpu_ptr() in blk_stat_add()?
> > > 
> > > Yeah.
> > 
> > So why is this complaining, doesn't rcu_read_lock() disable
> > preemption?
> 
> Ah, I guess it doesn't if PREEMPT_RCU is set. How about the below?

Should do it.  I was about to run LTP (where it turned up) again
anyway, I'll add this.  No news is good news, what you should hear.

> diff --git a/block/blk-stat.c b/block/blk-stat.c
> index 6c2f40940439..022f5925a427 100644
> --- a/block/blk-stat.c
> +++ b/block/blk-stat.c
> @@ -96,13 +96,16 @@ void blk_stat_add(struct request *rq)
>  
>  	rcu_read_lock();
>  	list_for_each_entry_rcu(cb, &q->stats->callbacks, list) {
> -		if (blk_stat_is_active(cb)) {
> -			bucket = cb->bucket_fn(rq);
> -			if (bucket < 0)
> -				continue;
> -			stat = &this_cpu_ptr(cb->cpu_stat)[bucket];
> -			__blk_stat_add(stat, value);
> -		}
> +		if (!blk_stat_is_active(cb))
> +			continue;
> +
> +		bucket = cb->bucket_fn(rq);
> +		if (bucket < 0)
> +			continue;
> +
> +		stat = &get_cpu_var(cb->cpu_stat)[bucket];
> +		__blk_stat_add(stat, value);
> +		put_cpu_var(cb->cpu_stat);
>  	}
>  	rcu_read_unlock();
>  }
> 

[toc] | [prev] | [next] | [standalone]

#1638273

On Tue, 2017-05-09 at 09:45 -0600, Jens Axboe wrote:

> > > > > > Is it from this_cpu_ptr() in blk_stat_add()?
> > > > > 
> > > > > Yeah.
> > > > 
> > > > So why is this complaining, doesn't rcu_read_lock() disable
> > > > preemption?
> > > 
> > > Ah, I guess it doesn't if PREEMPT_RCU is set. How about the
> > > below?
> > 
> > Should do it.  I was about to run LTP (where it turned up) again
> > anyway, I'll add this.  No news is good news, what you should hear.
> 
> Thanks, let me know, so I can add your tested-by (or whatever you
> prefer).

Sure, but s/get_cpu_var/get_cpu_ptr first.

	-MIke

[toc] | [prev] | [next] | [standalone]

#1638286

On Tue, 2017-05-09 at 10:32 -0600, Jens Axboe wrote:

> > Sure, but s/get_cpu_var/get_cpu_ptr first.
> 
> Doh.

Ditto, box OTOH spotted the booboo instantly :)

	-Mike

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web