Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1238035 > unrolled thread

[PATCH] SCSI: Fix hard lockup in scsi_remove_target()

Back to article view | Back to linux.kernel

Contents

  [PATCH] SCSI: Fix hard lockup in scsi_remove_target() Johannes Thumshirn <jthumshirn@suse.de> - 2015-10-02 10:30 +0200
    Re: [PATCH] SCSI: Fix hard lockup in scsi_remove_target() Hannes Reinecke <hare@suse.de> - 2015-10-02 10:30 +0200
    Re: [PATCH] SCSI: Fix hard lockup in scsi_remove_target() Christoph Hellwig <hch@infradead.org> - 2015-10-02 14:50 +0200
      Re: [PATCH] SCSI: Fix hard lockup in scsi_remove_target() Johannes Thumshirn <jthumshirn@suse.de> - 2015-10-02 15:10 +0200
        Re: [PATCH] SCSI: Fix hard lockup in scsi_remove_target() Christoph Hellwig <hch@infradead.org> - 2015-10-02 15:20 +0200

#1238035 — [PATCH] SCSI: Fix hard lockup in scsi_remove_target()

Removing a SCSI target via scsi_remove_target() suspected to be racy. When a
sibling get's removed from the list it can occassionly happen that one CPU is
stuck endlessly looping around this code block

list_for_each_entry(starget, &shost->__targets, siblings) {
        if (starget->state == STARGET_DEL)
                continue;

Resulting in the following hard lockup.

Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0
[...]
Call Trace:
 [<ffffffff8100471d>] dump_trace+0x7d/0x2d0
 [<ffffffff81004a04>] show_stack_log_lvl+0x94/0x170
 [<ffffffff81005cc1>] show_stack+0x21/0x50
 [<ffffffff8151aa75>] dump_stack+0x41/0x51
 [<ffffffff8151545a>] panic+0xc8/0x1d7
 [<ffffffff810fbdda>] watchdog_overflow_callback+0xba/0xc0
 [<ffffffff811336c8>] __perf_event_overflow+0x88/0x240
 [<ffffffff8101e3aa>] intel_pmu_handle_irq+0x1fa/0x3e0
 [<ffffffff81522836>] perf_event_nmi_handler+0x26/0x40
 [<ffffffff81521fcd>] nmi_handle.isra.2+0x8d/0x180
 [<ffffffff815221e6>] do_nmi+0x126/0x3c0
 [<ffffffff8152159b>] end_repeat_nmi+0x1a/0x1e
 [<ffffffffa00212e8>] scsi_remove_target+0x68/0x240 [scsi_mod]
 [<ffffffff81072742>] process_one_work+0x172/0x420
 [<ffffffff810733ba>] worker_thread+0x11a/0x3c0
 [<ffffffff81079d34>] kthread+0xb4/0xc0
 [<ffffffff81528cd8>] ret_from_fork+0x58/0x90

This patch decouples the list traversal for targets and the reaping of SCSI
targets by moving to be removed targets to a separate reap list. Entries in
this list can then be removed by the SCSI layer in a lockless manner.

This was discovered by a partner in a 24h stress test.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 drivers/scsi/scsi_sysfs.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/scsi/scsi_sysfs.c b/drivers/scsi/scsi_sysfs.c
index b333389..5d92cf56 100644
--- a/drivers/scsi/scsi_sysfs.c
+++ b/drivers/scsi/scsi_sysfs.c
@@ -1158,31 +1158,31 @@ static void __scsi_remove_target(struct scsi_target *starget)
 void scsi_remove_target(struct device *dev)
 {
 	struct Scsi_Host *shost = dev_to_shost(dev->parent);
-	struct scsi_target *starget, *last = NULL;
+	struct scsi_target *starget, *tmp;
 	unsigned long flags;
+	LIST_HEAD(reap_list);
 
 	/* remove targets being careful to lookup next entry before
 	 * deleting the last
 	 */
 	spin_lock_irqsave(shost->host_lock, flags);
-	list_for_each_entry(starget, &shost->__targets, siblings) {
+	list_for_each_entry_safe(starget, tmp, &shost->__targets, siblings) {
 		if (starget->state == STARGET_DEL)
 			continue;
 		if (starget->dev.parent == dev || &starget->dev == dev) {
 			/* assuming new targets arrive at the end */
 			kref_get(&starget->reap_ref);
 			spin_unlock_irqrestore(shost->host_lock, flags);
-			if (last)
-				scsi_target_reap(last);
-			last = starget;
+
 			__scsi_remove_target(starget);
+			list_move_tail(&starget->siblings, &reap_list);
 			spin_lock_irqsave(shost->host_lock, flags);
 		}
 	}
 	spin_unlock_irqrestore(shost->host_lock, flags);
 
-	if (last)
-		scsi_target_reap(last);
+	list_for_each_entry_safe(starget, tmp, &reap_list, siblings)
+		scsi_target_reap(starget);
 }
 EXPORT_SYMBOL(scsi_remove_target);
 
-- 
2.5.0

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1238037

On 10/02/2015 10:21 AM, Johannes Thumshirn wrote:
> Removing a SCSI target via scsi_remove_target() suspected to be racy. When a
> sibling get's removed from the list it can occassionly happen that one CPU is
> stuck endlessly looping around this code block
> 
> list_for_each_entry(starget, &shost->__targets, siblings) {
>         if (starget->state == STARGET_DEL)
>                 continue;
> 
> Resulting in the following hard lockup.
> 
> Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0
> [...]
> Call Trace:
>  [<ffffffff8100471d>] dump_trace+0x7d/0x2d0
>  [<ffffffff81004a04>] show_stack_log_lvl+0x94/0x170
>  [<ffffffff81005cc1>] show_stack+0x21/0x50
>  [<ffffffff8151aa75>] dump_stack+0x41/0x51
>  [<ffffffff8151545a>] panic+0xc8/0x1d7
>  [<ffffffff810fbdda>] watchdog_overflow_callback+0xba/0xc0
>  [<ffffffff811336c8>] __perf_event_overflow+0x88/0x240
>  [<ffffffff8101e3aa>] intel_pmu_handle_irq+0x1fa/0x3e0
>  [<ffffffff81522836>] perf_event_nmi_handler+0x26/0x40
>  [<ffffffff81521fcd>] nmi_handle.isra.2+0x8d/0x180
>  [<ffffffff815221e6>] do_nmi+0x126/0x3c0
>  [<ffffffff8152159b>] end_repeat_nmi+0x1a/0x1e
>  [<ffffffffa00212e8>] scsi_remove_target+0x68/0x240 [scsi_mod]
>  [<ffffffff81072742>] process_one_work+0x172/0x420
>  [<ffffffff810733ba>] worker_thread+0x11a/0x3c0
>  [<ffffffff81079d34>] kthread+0xb4/0xc0
>  [<ffffffff81528cd8>] ret_from_fork+0x58/0x90
> 
> This patch decouples the list traversal for targets and the reaping of SCSI
> targets by moving to be removed targets to a separate reap list. Entries in
> this list can then be removed by the SCSI layer in a lockless manner.
> 
> This was discovered by a partner in a 24h stress test.
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> ---
>  drivers/scsi/scsi_sysfs.c | 14 +++++++-------
>  1 file changed, 7 insertions(+), 7 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare@suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		               zSeries & Storage
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1238167

> -	list_for_each_entry(starget, &shost->__targets, siblings) {
> +	list_for_each_entry_safe(starget, tmp, &shost->__targets, siblings) {
>  		if (starget->state == STARGET_DEL)
>  			continue;
>  		if (starget->dev.parent == dev || &starget->dev == dev) {
>  			/* assuming new targets arrive at the end */

Now that the last variable is gone this comments isn't needed.

>  			kref_get(&starget->reap_ref);
>  			spin_unlock_irqrestore(shost->host_lock, flags);
> -			if (last)
> -				scsi_target_reap(last);
> -			last = starget;
> +
>  			__scsi_remove_target(starget);
> +			list_move_tail(&starget->siblings, &reap_list);
>  			spin_lock_irqsave(shost->host_lock, flags);
>  		}

What makes the list_move save after dropping host_lock?  I think this
needs to be changed to not drop the host_lock and change
__scsi_remove_target to expect host_lock held to be safe. 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1238179

Christoph Hellwig <hch@infradead.org> writes:

>> -	list_for_each_entry(starget, &shost->__targets, siblings) {
>> +	list_for_each_entry_safe(starget, tmp, &shost->__targets, siblings) {
>>  		if (starget->state == STARGET_DEL)
>>  			continue;
>>  		if (starget->dev.parent == dev || &starget->dev == dev) {
>>  			/* assuming new targets arrive at the end */
>
> Now that the last variable is gone this comments isn't needed.

Yep, you're right. I'll remove it.

>
>>  			kref_get(&starget->reap_ref);
>>  			spin_unlock_irqrestore(shost->host_lock, flags);
>> -			if (last)
>> -				scsi_target_reap(last);
>> -			last = starget;
>> +
>>  			__scsi_remove_target(starget);
>> +			list_move_tail(&starget->siblings, &reap_list);
>>  			spin_lock_irqsave(shost->host_lock, flags);
>>  		}
>
> What makes the list_move save after dropping host_lock?  I think this
> needs to be changed to not drop the host_lock and change
> __scsi_remove_target to expect host_lock held to be safe. 

Having the list_move() outside of the host_lock was purely by
accident. Interestingly the stressing didn't mind it. But yes you're
right, __scsi_remove_target() should be made host_lock() save for being
called under the host_lock.

Regarding the list move, does it look OK for you (i.e. do we still need
it after reworking __scsi_remove_target())? IMHO yes, but I only have
half a year of experience in this area).

> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Johannes Thumshirn                                          Storage
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1238188

On Fri, Oct 02, 2015 at 03:09:33PM +0200, Johannes Thumshirn wrote:
> Having the list_move() outside of the host_lock was purely by
> accident. Interestingly the stressing didn't mind it. But yes you're
> right, __scsi_remove_target() should be made host_lock() save for being
> called under the host_lock.
> 
> Regarding the list move, does it look OK for you (i.e. do we still need
> it after reworking __scsi_remove_target())? IMHO yes, but I only have
> half a year of experience in this area).

Hi Johannes,

your general approach looks fine, we just need to fix up the locking as
far as I can tell.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web