Groups > linux.kernel > #1684001 > unrolled thread

[PATCH v3 0/7] Enhance libsas hotplug feature

Started by	Yijing Wang <wangyijing@huawei.com>
First post	2017-07-10 09:10 +0200
Last post	2017-07-14 10:30 +0200
Articles	18 — 5 participants

Back to article view | Back to linux.kernel

  [PATCH v3 0/7] Enhance libsas hotplug feature Yijing Wang <wangyijing@huawei.com> - 2017-07-10 09:10 +0200
    [PATCH v3 4/7] libsas: add sas event wait-complete support Yijing Wang <wangyijing@huawei.com> - 2017-07-10 09:10 +0200
      Re: [PATCH v3 4/7] libsas: add sas event wait-complete support Hannes Reinecke <hare@suse.de> - 2017-07-14 09:00 +0200
        Re: [PATCH v3 4/7] libsas: add sas event wait-complete support wangyijing <wangyijing@huawei.com> - 2017-07-14 09:50 +0200
        Re: [PATCH v3 4/7] libsas: add sas event wait-complete support John Garry <john.garry@huawei.com> - 2017-07-14 10:50 +0200
    [PATCH v3 6/7] libsas: add wait-complete support to sync discovery event Yijing Wang <wangyijing@huawei.com> - 2017-07-10 09:10 +0200
      Re: [PATCH v3 6/7] libsas: add wait-complete support to sync  discovery event John Garry <john.garry@huawei.com> - 2017-07-12 16:00 +0200
        Re: [PATCH v3 6/7] libsas: add wait-complete support to sync  discovery event wangyijing <wangyijing@huawei.com> - 2017-07-13 04:30 +0200
      Re: [PATCH v3 6/7] libsas: add wait-complete support to sync  discovery event Hannes Reinecke <hare@suse.de> - 2017-07-14 09:00 +0200
    [PATCH v3 3/7] libsas: Use new workqueue to run sas event Yijing Wang <wangyijing@huawei.com> - 2017-07-10 09:10 +0200
      Re: [PATCH v3 3/7] libsas: Use new workqueue to run sas event Hannes Reinecke <hare@suse.de> - 2017-07-14 08:50 +0200
    Re: [PATCH v3 0/7] Enhance libsas hotplug feature John Garry <john.garry@huawei.com> - 2017-07-12 12:10 +0200
      Re: [PATCH v3 0/7] Enhance libsas hotplug feature Johannes Thumshirn <jthumshirn@suse.de> - 2017-07-12 14:00 +0200
      Re: [PATCH v3 0/7] Enhance libsas hotplug feature wangyijing <wangyijing@huawei.com> - 2017-07-13 03:30 +0200
      Re: [PATCH v3 0/7] Enhance libsas hotplug feature wangyijing <wangyijing@huawei.com> - 2017-07-13 03:40 +0200
        Re: [PATCH v3 0/7] Enhance libsas hotplug feature John Garry <john.garry@huawei.com> - 2017-07-13 10:10 +0200
          Re: [PATCH v3 0/7] Enhance libsas hotplug feature wangyijing <wangyijing@huawei.com> - 2017-07-13 10:50 +0200
    Re: [PATCH v3 0/7] Enhance libsas hotplug feature wangyijing <wangyijing@huawei.com> - 2017-07-14 10:30 +0200

#1684001 — [PATCH v3 0/7] Enhance libsas hotplug feature

From	Yijing Wang <wangyijing@huawei.com>
Date	2017-07-10 09:10 +0200
Subject	[PATCH v3 0/7] Enhance libsas hotplug feature
Message-ID	<u1AZr-1Sl-3@gated-at.bofh.it>

This patchset is based Johannes's patch
"scsi: sas: scsi_queue_work can fail, so make callers aware"

Now the libsas hotplug has some issues, Dan Williams report
a similar bug here before
https://www.mail-archive.com/linux-scsi@vger.kernel.org/msg39187.html

The issues we have found
1. if LLDD burst reports lots of phy-up/phy-down sas events, some events
   may lost because a same sas events is pending now, finally libsas topo
   may different the hardware.
2. receive a phy down sas event, libsas call sas_deform_port to remove
   devices, it would first delete the sas port, then put a destruction
   discovery event in a new work, and queue it at the tail of workqueue,
   once the sas port be deleted, its children device will be deleted too,
   when the destruction work start, it will found the target device has
   been removed, and report a sysfs warnning.
3. since a hotplug process will be devided into several works, if a phy up
   sas event insert into phydown works, like
   destruction work  ---> PORTE_BYTES_DMAED (sas_form_port) ---->PHYE_LOSS_OF_SIGNAL
   the hot remove flow would broken by PORTE_BYTES_DMAED event, it's not
   we expected, and issues would occur.

The first patch fix the sas events lost, and the second one introudce wait-complete
to fix the hotplug order issues.

v2->v3: some code improvements suggested by Johannes and John,
	    split v2 patch 2 into several small pathes.
v1->v2: some code improvements suggested by John Garry

Yijing Wang (7):
  libsas: Use static sas event pool to appease sas event lost
  libsas: remove unused port_gone_completion
  libsas: Use new workqueue to run sas event
  libsas: add sas event wait-complete support
  libsas: add a new workqueue to run probe/destruct discovery event
  libsas: add wait-complete support to sync discovery event
  libsas: release disco mutex during waiting in sas_ex_discover_end_dev

 drivers/scsi/libsas/sas_discover.c |  58 +++++++---
 drivers/scsi/libsas/sas_event.c    | 212 ++++++++++++++++++++++++++++++++-----
 drivers/scsi/libsas/sas_expander.c |  22 +++-
 drivers/scsi/libsas/sas_init.c     |  21 ++--
 drivers/scsi/libsas/sas_internal.h |  64 +++++++++++
 drivers/scsi/libsas/sas_phy.c      |  48 +++------
 drivers/scsi/libsas/sas_port.c     |  22 ++--
 include/scsi/libsas.h              |  27 +++--
 8 files changed, 373 insertions(+), 101 deletions(-)

-- 
2.5.0

[toc] | [next] | [standalone]

#1684002 — [PATCH v3 4/7] libsas: add sas event wait-complete support

From	Yijing Wang <wangyijing@huawei.com>
Date	2017-07-10 09:10 +0200
Subject	[PATCH v3 4/7] libsas: add sas event wait-complete support
Message-ID	<u1AZs-1Sl-33@gated-at.bofh.it>
In reply to	#1684001

Introduce wait-complete for libsas sas event processing,
execute sas port create/destruct in sync.

Signed-off-by: Yijing Wang <wangyijing@huawei.com>
CC: John Garry <john.garry@huawei.com>
CC: Johannes Thumshirn <jthumshirn@suse.de>
CC: Ewan Milne <emilne@redhat.com>
CC: Christoph Hellwig <hch@lst.de>
CC: Tomas Henzl <thenzl@redhat.com>
CC: Dan Williams <dan.j.williams@intel.com>
---
 drivers/scsi/libsas/sas_discover.c | 41 ++++++++++++++++++++++++++++----------
 drivers/scsi/libsas/sas_internal.h | 34 +++++++++++++++++++++++++++++++
 drivers/scsi/libsas/sas_port.c     |  4 ++++
 include/scsi/libsas.h              |  5 ++++-
 4 files changed, 72 insertions(+), 12 deletions(-)

diff --git a/drivers/scsi/libsas/sas_discover.c b/drivers/scsi/libsas/sas_discover.c
index 60de662..5d4a3a8 100644
--- a/drivers/scsi/libsas/sas_discover.c
+++ b/drivers/scsi/libsas/sas_discover.c
@@ -525,16 +525,43 @@ static void sas_revalidate_domain(struct work_struct *work)
 	mutex_unlock(&ha->disco_mutex);
 }
 
+static const work_func_t sas_event_fns[DISC_NUM_EVENTS] = {
+	[DISCE_DISCOVER_DOMAIN] = sas_discover_domain,
+	[DISCE_REVALIDATE_DOMAIN] = sas_revalidate_domain,
+	[DISCE_PROBE] = sas_probe_devices,
+	[DISCE_SUSPEND] = sas_suspend_devices,
+	[DISCE_RESUME] = sas_resume_devices,
+	[DISCE_DESTRUCT] = sas_destruct_devices,
+};
+
+/* a simple wrapper for sas discover event funtions */
+static void sas_discover_common_fn(struct work_struct *work)
+{
+	struct sas_discovery_event *ev = to_sas_discovery_event(work);
+	struct asd_sas_port *port = ev->port;
+
+	sas_event_fns[ev->type](work);
+	sas_port_put(port);
+}
+
+
 /* ---------- Events ---------- */
 
 static void sas_chain_work(struct sas_ha_struct *ha, struct sas_work *sw)
 {
+	int ret;
+	struct sas_discovery_event *ev = to_sas_discovery_event(&sw->work);
+	struct asd_sas_port *port = ev->port;
+
 	/* chained work is not subject to SA_HA_DRAINING or
 	 * SAS_HA_REGISTERED, because it is either submitted in the
 	 * workqueue, or known to be submitted from a context that is
 	 * not racing against draining
 	 */
-	scsi_queue_work(ha->core.shost, &sw->work);
+	sas_port_get(port);
+	ret = scsi_queue_work(ha->core.shost, &sw->work);
+	if (ret != 1)
+		sas_port_put(port);
 }
 
 static void sas_chain_event(int event, unsigned long *pending,
@@ -575,18 +602,10 @@ void sas_init_disc(struct sas_discovery *disc, struct asd_sas_port *port)
 {
 	int i;
 
-	static const work_func_t sas_event_fns[DISC_NUM_EVENTS] = {
-		[DISCE_DISCOVER_DOMAIN] = sas_discover_domain,
-		[DISCE_REVALIDATE_DOMAIN] = sas_revalidate_domain,
-		[DISCE_PROBE] = sas_probe_devices,
-		[DISCE_SUSPEND] = sas_suspend_devices,
-		[DISCE_RESUME] = sas_resume_devices,
-		[DISCE_DESTRUCT] = sas_destruct_devices,
-	};
-
 	disc->pending = 0;
 	for (i = 0; i < DISC_NUM_EVENTS; i++) {
-		INIT_SAS_WORK(&disc->disc_work[i].work, sas_event_fns[i]);
+		INIT_SAS_WORK(&disc->disc_work[i].work, sas_discover_common_fn);
 		disc->disc_work[i].port = port;
+		disc->disc_work[i].type = i;
 	}
 }
diff --git a/drivers/scsi/libsas/sas_internal.h b/drivers/scsi/libsas/sas_internal.h
index f03ce64..890b5d26 100644
--- a/drivers/scsi/libsas/sas_internal.h
+++ b/drivers/scsi/libsas/sas_internal.h
@@ -100,6 +100,40 @@ void sas_free_device(struct kref *kref);
 extern const work_func_t sas_phy_event_fns[PHY_NUM_EVENTS];
 extern const work_func_t sas_port_event_fns[PORT_NUM_EVENTS];
 
+static void sas_complete_event(struct kref *kref)
+{
+	struct asd_sas_port *port = container_of(kref, struct asd_sas_port, ref);
+
+	complete_all(&port->completion);
+}
+
+static inline void sas_port_put(struct asd_sas_port *port)
+{
+	if (port->is_sync)
+		kref_put(&port->ref, sas_complete_event);
+}
+
+static inline void sas_port_wait_init(struct asd_sas_port *port)
+{
+	init_completion(&port->completion);
+	kref_init(&port->ref);
+	port->is_sync = true;
+}
+
+static inline void sas_port_wait_completion(
+		struct asd_sas_port *port)
+{
+	sas_port_put(port);
+	wait_for_completion(&port->completion);
+	port->is_sync = false;
+}
+
+static inline void sas_port_get(struct asd_sas_port *port)
+{
+	if (port && port->is_sync)
+		kref_get(&port->ref);
+}
+
 #ifdef CONFIG_SCSI_SAS_HOST_SMP
 extern int sas_smp_host_handler(struct Scsi_Host *shost, struct request *req,
 				struct request *rsp);
diff --git a/drivers/scsi/libsas/sas_port.c b/drivers/scsi/libsas/sas_port.c
index 9326628..d589adb 100644
--- a/drivers/scsi/libsas/sas_port.c
+++ b/drivers/scsi/libsas/sas_port.c
@@ -191,7 +191,9 @@ static void sas_form_port(struct asd_sas_phy *phy)
 	if (si->dft->lldd_port_formed)
 		si->dft->lldd_port_formed(phy);
 
+	sas_port_wait_init(port);
 	sas_discover_event(phy->port, DISCE_DISCOVER_DOMAIN);
+	sas_port_wait_completion(port);
 }
 
 /**
@@ -218,7 +220,9 @@ void sas_deform_port(struct asd_sas_phy *phy, int gone)
 		dev->pathways--;
 
 	if (port->num_phys == 1) {
+		sas_port_wait_init(port);
 		sas_unregister_domain_devices(port, gone);
+		sas_port_wait_completion(port);
 		sas_port_delete(port->port);
 		port->port = NULL;
 	} else {
diff --git a/include/scsi/libsas.h b/include/scsi/libsas.h
index a01ca42..c2ef05e 100644
--- a/include/scsi/libsas.h
+++ b/include/scsi/libsas.h
@@ -242,6 +242,7 @@ static inline void INIT_SAS_WORK(struct sas_work *sw, void (*fn)(struct work_str
 struct sas_discovery_event {
 	struct sas_work work;
 	struct asd_sas_port *port;
+	enum discover_event	type;
 };
 
 static inline struct sas_discovery_event *to_sas_discovery_event(struct work_struct *work)
@@ -273,7 +274,9 @@ struct asd_sas_port {
 
 	struct sas_work work;
 	int suspended;
-
+	struct kref ref;
+	struct completion completion;
+	bool is_sync;
 /* public: */
 	int id;
 
-- 
2.5.0

[toc] | [prev] | [next] | [standalone]

#1687121 — Re: [PATCH v3 4/7] libsas: add sas event wait-complete support

From	Hannes Reinecke <hare@suse.de>
Date	2017-07-14 09:00 +0200
Subject	Re: [PATCH v3 4/7] libsas: add sas event wait-complete support
Message-ID	<u32JY-88H-23@gated-at.bofh.it>
In reply to	#1684002

On 07/10/2017 09:06 AM, Yijing Wang wrote:
> Introduce wait-complete for libsas sas event processing,
> execute sas port create/destruct in sync.
> 
> Signed-off-by: Yijing Wang <wangyijing@huawei.com>
> CC: John Garry <john.garry@huawei.com>
> CC: Johannes Thumshirn <jthumshirn@suse.de>
> CC: Ewan Milne <emilne@redhat.com>
> CC: Christoph Hellwig <hch@lst.de>
> CC: Tomas Henzl <thenzl@redhat.com>
> CC: Dan Williams <dan.j.williams@intel.com>
> ---
>  drivers/scsi/libsas/sas_discover.c | 41 ++++++++++++++++++++++++++++----------
>  drivers/scsi/libsas/sas_internal.h | 34 +++++++++++++++++++++++++++++++
>  drivers/scsi/libsas/sas_port.c     |  4 ++++
>  include/scsi/libsas.h              |  5 ++++-
>  4 files changed, 72 insertions(+), 12 deletions(-)
> 
> diff --git a/drivers/scsi/libsas/sas_discover.c b/drivers/scsi/libsas/sas_discover.c
> index 60de662..5d4a3a8 100644
> --- a/drivers/scsi/libsas/sas_discover.c
> +++ b/drivers/scsi/libsas/sas_discover.c
> @@ -525,16 +525,43 @@ static void sas_revalidate_domain(struct work_struct *work)
>  	mutex_unlock(&ha->disco_mutex);
>  }
>  
> +static const work_func_t sas_event_fns[DISC_NUM_EVENTS] = {
> +	[DISCE_DISCOVER_DOMAIN] = sas_discover_domain,
> +	[DISCE_REVALIDATE_DOMAIN] = sas_revalidate_domain,
> +	[DISCE_PROBE] = sas_probe_devices,
> +	[DISCE_SUSPEND] = sas_suspend_devices,
> +	[DISCE_RESUME] = sas_resume_devices,
> +	[DISCE_DESTRUCT] = sas_destruct_devices,
> +};
> +
> +/* a simple wrapper for sas discover event funtions */
> +static void sas_discover_common_fn(struct work_struct *work)
> +{
> +	struct sas_discovery_event *ev = to_sas_discovery_event(work);
> +	struct asd_sas_port *port = ev->port;
> +
> +	sas_event_fns[ev->type](work);
> +	sas_port_put(port);
> +}
> +
> +
>  /* ---------- Events ---------- */
>  
>  static void sas_chain_work(struct sas_ha_struct *ha, struct sas_work *sw)
>  {
> +	int ret;
> +	struct sas_discovery_event *ev = to_sas_discovery_event(&sw->work);
> +	struct asd_sas_port *port = ev->port;
> +
>  	/* chained work is not subject to SA_HA_DRAINING or
>  	 * SAS_HA_REGISTERED, because it is either submitted in the
>  	 * workqueue, or known to be submitted from a context that is
>  	 * not racing against draining
>  	 */
> -	scsi_queue_work(ha->core.shost, &sw->work);
> +	sas_port_get(port);
> +	ret = scsi_queue_work(ha->core.shost, &sw->work);
> +	if (ret != 1)
> +		sas_port_put(port);
>  }
>  
>  static void sas_chain_event(int event, unsigned long *pending,
> @@ -575,18 +602,10 @@ void sas_init_disc(struct sas_discovery *disc, struct asd_sas_port *port)
>  {
>  	int i;
>  
> -	static const work_func_t sas_event_fns[DISC_NUM_EVENTS] = {
> -		[DISCE_DISCOVER_DOMAIN] = sas_discover_domain,
> -		[DISCE_REVALIDATE_DOMAIN] = sas_revalidate_domain,
> -		[DISCE_PROBE] = sas_probe_devices,
> -		[DISCE_SUSPEND] = sas_suspend_devices,
> -		[DISCE_RESUME] = sas_resume_devices,
> -		[DISCE_DESTRUCT] = sas_destruct_devices,
> -	};
> -
>  	disc->pending = 0;
>  	for (i = 0; i < DISC_NUM_EVENTS; i++) {
> -		INIT_SAS_WORK(&disc->disc_work[i].work, sas_event_fns[i]);
> +		INIT_SAS_WORK(&disc->disc_work[i].work, sas_discover_common_fn);
>  		disc->disc_work[i].port = port;
> +		disc->disc_work[i].type = i;
>  	}
>  }
> diff --git a/drivers/scsi/libsas/sas_internal.h b/drivers/scsi/libsas/sas_internal.h
> index f03ce64..890b5d26 100644
> --- a/drivers/scsi/libsas/sas_internal.h
> +++ b/drivers/scsi/libsas/sas_internal.h
> @@ -100,6 +100,40 @@ void sas_free_device(struct kref *kref);
>  extern const work_func_t sas_phy_event_fns[PHY_NUM_EVENTS];
>  extern const work_func_t sas_port_event_fns[PORT_NUM_EVENTS];
>  
> +static void sas_complete_event(struct kref *kref)
> +{
> +	struct asd_sas_port *port = container_of(kref, struct asd_sas_port, ref);
> +
> +	complete_all(&port->completion);
> +}
> +
> +static inline void sas_port_put(struct asd_sas_port *port)
> +{
> +	if (port->is_sync)
> +		kref_put(&port->ref, sas_complete_event);
> +}
> +
> +static inline void sas_port_wait_init(struct asd_sas_port *port)
> +{
> +	init_completion(&port->completion);
> +	kref_init(&port->ref);
> +	port->is_sync = true;
> +}
> +
> +static inline void sas_port_wait_completion(
> +		struct asd_sas_port *port)
> +{
> +	sas_port_put(port);
> +	wait_for_completion(&port->completion);
> +	port->is_sync = false;
> +}
> +
> +static inline void sas_port_get(struct asd_sas_port *port)
> +{
> +	if (port && port->is_sync)
> +		kref_get(&port->ref);
> +}
> +
>  #ifdef CONFIG_SCSI_SAS_HOST_SMP
>  extern int sas_smp_host_handler(struct Scsi_Host *shost, struct request *req,
>  				struct request *rsp);
> diff --git a/drivers/scsi/libsas/sas_port.c b/drivers/scsi/libsas/sas_port.c
> index 9326628..d589adb 100644
> --- a/drivers/scsi/libsas/sas_port.c
> +++ b/drivers/scsi/libsas/sas_port.c
> @@ -191,7 +191,9 @@ static void sas_form_port(struct asd_sas_phy *phy)
>  	if (si->dft->lldd_port_formed)
>  		si->dft->lldd_port_formed(phy);
>  
> +	sas_port_wait_init(port);
>  	sas_discover_event(phy->port, DISCE_DISCOVER_DOMAIN);
> +	sas_port_wait_completion(port);
>  }
>  
>  /**
> @@ -218,7 +220,9 @@ void sas_deform_port(struct asd_sas_phy *phy, int gone)
>  		dev->pathways--;
>  
>  	if (port->num_phys == 1) {
> +		sas_port_wait_init(port);
>  		sas_unregister_domain_devices(port, gone);
> +		sas_port_wait_completion(port);
>  		sas_port_delete(port->port);
>  		port->port = NULL;
>  	} else {

I would rather use the standard on-stack completion here;
like this:

	DECLARE_COMPLETION_ONSTACK(complete);
	port->completion = &complete;
	sas_unregister_domain_devices(port, gone);
	wait_for_completion(&complete);
	sas_port_delete(port->port);

which would simplify the above helpers to:

static inline void sas_port_put(struct asd_sas_port *port)
{
	if (port->completion)
		kref_put(&port->ref, sas_complete_event);
}

and you could do away with the 'is_sync' helper.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

[toc] | [prev] | [next] | [standalone]

#1687129 — Re: [PATCH v3 4/7] libsas: add sas event wait-complete support

From	wangyijing <wangyijing@huawei.com>
Date	2017-07-14 09:50 +0200
Subject	Re: [PATCH v3 4/7] libsas: add sas event wait-complete support
Message-ID	<u33wl-ht-7@gated-at.bofh.it>
In reply to	#1687121


在 2017/7/14 14:51, Hannes Reinecke 写道:
> On 07/10/2017 09:06 AM, Yijing Wang wrote:
>> Introduce wait-complete for libsas sas event processing,
>> execute sas port create/destruct in sync.
>>
>> Signed-off-by: Yijing Wang <wangyijing@huawei.com>
>> CC: John Garry <john.garry@huawei.com>
>> CC: Johannes Thumshirn <jthumshirn@suse.de>
>> CC: Ewan Milne <emilne@redhat.com>
>> CC: Christoph Hellwig <hch@lst.de>
>> CC: Tomas Henzl <thenzl@redhat.com>
>> CC: Dan Williams <dan.j.williams@intel.com>
>> ---
>>  drivers/scsi/libsas/sas_discover.c | 41 ++++++++++++++++++++++++++++----------
>>  drivers/scsi/libsas/sas_internal.h | 34 +++++++++++++++++++++++++++++++
>>  drivers/scsi/libsas/sas_port.c     |  4 ++++
>>  include/scsi/libsas.h              |  5 ++++-
>>  4 files changed, 72 insertions(+), 12 deletions(-)
>>
>> diff --git a/drivers/scsi/libsas/sas_discover.c b/drivers/scsi/libsas/sas_discover.c
>> index 60de662..5d4a3a8 100644
>> --- a/drivers/scsi/libsas/sas_discover.c
>> +++ b/drivers/scsi/libsas/sas_discover.c
>> @@ -525,16 +525,43 @@ static void sas_revalidate_domain(struct work_struct *work)
>>  	mutex_unlock(&ha->disco_mutex);
>>  }
>>  
>> +static const work_func_t sas_event_fns[DISC_NUM_EVENTS] = {
>> +	[DISCE_DISCOVER_DOMAIN] = sas_discover_domain,
>> +	[DISCE_REVALIDATE_DOMAIN] = sas_revalidate_domain,
>> +	[DISCE_PROBE] = sas_probe_devices,
>> +	[DISCE_SUSPEND] = sas_suspend_devices,
>> +	[DISCE_RESUME] = sas_resume_devices,
>> +	[DISCE_DESTRUCT] = sas_destruct_devices,
>> +};
>> +
>> +/* a simple wrapper for sas discover event funtions */
>> +static void sas_discover_common_fn(struct work_struct *work)
>> +{
>> +	struct sas_discovery_event *ev = to_sas_discovery_event(work);
>> +	struct asd_sas_port *port = ev->port;
>> +
>> +	sas_event_fns[ev->type](work);
>> +	sas_port_put(port);
>> +}
>> +
>> +
>>  /* ---------- Events ---------- */
>>  
>>  static void sas_chain_work(struct sas_ha_struct *ha, struct sas_work *sw)
>>  {
>> +	int ret;
>> +	struct sas_discovery_event *ev = to_sas_discovery_event(&sw->work);
>> +	struct asd_sas_port *port = ev->port;
>> +
>>  	/* chained work is not subject to SA_HA_DRAINING or
>>  	 * SAS_HA_REGISTERED, because it is either submitted in the
>>  	 * workqueue, or known to be submitted from a context that is
>>  	 * not racing against draining
>>  	 */
>> -	scsi_queue_work(ha->core.shost, &sw->work);
>> +	sas_port_get(port);
>> +	ret = scsi_queue_work(ha->core.shost, &sw->work);
>> +	if (ret != 1)
>> +		sas_port_put(port);
>>  }
>>  
>>  static void sas_chain_event(int event, unsigned long *pending,
>> @@ -575,18 +602,10 @@ void sas_init_disc(struct sas_discovery *disc, struct asd_sas_port *port)
>>  {
>>  	int i;
>>  
>> -	static const work_func_t sas_event_fns[DISC_NUM_EVENTS] = {
>> -		[DISCE_DISCOVER_DOMAIN] = sas_discover_domain,
>> -		[DISCE_REVALIDATE_DOMAIN] = sas_revalidate_domain,
>> -		[DISCE_PROBE] = sas_probe_devices,
>> -		[DISCE_SUSPEND] = sas_suspend_devices,
>> -		[DISCE_RESUME] = sas_resume_devices,
>> -		[DISCE_DESTRUCT] = sas_destruct_devices,
>> -	};
>> -
>>  	disc->pending = 0;
>>  	for (i = 0; i < DISC_NUM_EVENTS; i++) {
>> -		INIT_SAS_WORK(&disc->disc_work[i].work, sas_event_fns[i]);
>> +		INIT_SAS_WORK(&disc->disc_work[i].work, sas_discover_common_fn);
>>  		disc->disc_work[i].port = port;
>> +		disc->disc_work[i].type = i;
>>  	}
>>  }
>> diff --git a/drivers/scsi/libsas/sas_internal.h b/drivers/scsi/libsas/sas_internal.h
>> index f03ce64..890b5d26 100644
>> --- a/drivers/scsi/libsas/sas_internal.h
>> +++ b/drivers/scsi/libsas/sas_internal.h
>> @@ -100,6 +100,40 @@ void sas_free_device(struct kref *kref);
>>  extern const work_func_t sas_phy_event_fns[PHY_NUM_EVENTS];
>>  extern const work_func_t sas_port_event_fns[PORT_NUM_EVENTS];
>>  
>> +static void sas_complete_event(struct kref *kref)
>> +{
>> +	struct asd_sas_port *port = container_of(kref, struct asd_sas_port, ref);
>> +
>> +	complete_all(&port->completion);
>> +}
>> +
>> +static inline void sas_port_put(struct asd_sas_port *port)
>> +{
>> +	if (port->is_sync)
>> +		kref_put(&port->ref, sas_complete_event);
>> +}
>> +
>> +static inline void sas_port_wait_init(struct asd_sas_port *port)
>> +{
>> +	init_completion(&port->completion);
>> +	kref_init(&port->ref);
>> +	port->is_sync = true;
>> +}
>> +
>> +static inline void sas_port_wait_completion(
>> +		struct asd_sas_port *port)
>> +{
>> +	sas_port_put(port);
>> +	wait_for_completion(&port->completion);
>> +	port->is_sync = false;
>> +}
>> +
>> +static inline void sas_port_get(struct asd_sas_port *port)
>> +{
>> +	if (port && port->is_sync)
>> +		kref_get(&port->ref);
>> +}
>> +
>>  #ifdef CONFIG_SCSI_SAS_HOST_SMP
>>  extern int sas_smp_host_handler(struct Scsi_Host *shost, struct request *req,
>>  				struct request *rsp);
>> diff --git a/drivers/scsi/libsas/sas_port.c b/drivers/scsi/libsas/sas_port.c
>> index 9326628..d589adb 100644
>> --- a/drivers/scsi/libsas/sas_port.c
>> +++ b/drivers/scsi/libsas/sas_port.c
>> @@ -191,7 +191,9 @@ static void sas_form_port(struct asd_sas_phy *phy)
>>  	if (si->dft->lldd_port_formed)
>>  		si->dft->lldd_port_formed(phy);
>>  
>> +	sas_port_wait_init(port);
>>  	sas_discover_event(phy->port, DISCE_DISCOVER_DOMAIN);
>> +	sas_port_wait_completion(port);
>>  }
>>  
>>  /**
>> @@ -218,7 +220,9 @@ void sas_deform_port(struct asd_sas_phy *phy, int gone)
>>  		dev->pathways--;
>>  
>>  	if (port->num_phys == 1) {
>> +		sas_port_wait_init(port);
>>  		sas_unregister_domain_devices(port, gone);
>> +		sas_port_wait_completion(port);
>>  		sas_port_delete(port->port);
>>  		port->port = NULL;
>>  	} else {
> 
> I would rather use the standard on-stack completion here;
> like this:
> 
> 	DECLARE_COMPLETION_ONSTACK(complete);
> 	port->completion = &complete;
> 	sas_unregister_domain_devices(port, gone);
> 	wait_for_completion(&complete);
> 	sas_port_delete(port->port);
> 
> which would simplify the above helpers to:
> 
> static inline void sas_port_put(struct asd_sas_port *port)
> {
> 	if (port->completion)
> 		kref_put(&port->ref, sas_complete_event);
> }
> 
> and you could do away with the 'is_sync' helper.

It looks better, thanks!

> 
> Cheers,
> 
> Hannes
>

[toc] | [prev] | [next] | [standalone]

#1687162 — Re: [PATCH v3 4/7] libsas: add sas event wait-complete support

From	John Garry <john.garry@huawei.com>
Date	2017-07-14 10:50 +0200
Subject	Re: [PATCH v3 4/7] libsas: add sas event wait-complete support
Message-ID	<u34sp-T0-7@gated-at.bofh.it>
In reply to	#1687121

On 14/07/2017 07:51, Hannes Reinecke wrote:
>>  #ifdef CONFIG_SCSI_SAS_HOST_SMP
>> >  extern int sas_smp_host_handler(struct Scsi_Host *shost, struct request *req,
>> >  				struct request *rsp);
>> > diff --git a/drivers/scsi/libsas/sas_port.c b/drivers/scsi/libsas/sas_port.c
>> > index 9326628..d589adb 100644
>> > --- a/drivers/scsi/libsas/sas_port.c
>> > +++ b/drivers/scsi/libsas/sas_port.c
>> > @@ -191,7 +191,9 @@ static void sas_form_port(struct asd_sas_phy *phy)
>> >  	if (si->dft->lldd_port_formed)
>> >  		si->dft->lldd_port_formed(phy);
>> >
>> > +	sas_port_wait_init(port);
>> >  	sas_discover_event(phy->port, DISCE_DISCOVER_DOMAIN);
>> > +	sas_port_wait_completion(port);
>> >  }
>> >
>> >  /**
>> > @@ -218,7 +220,9 @@ void sas_deform_port(struct asd_sas_phy *phy, int gone)
>> >  		dev->pathways--;
>> >

Hannes thanks for checking.

>> >  	if (port->num_phys == 1) {
>> > +		sas_port_wait_init(port);
>> >  		sas_unregister_domain_devices(port, gone);
>> > +		sas_port_wait_completion(port);
>> >  		sas_port_delete(port->port);
>> >  		port->port = NULL;
>> >  	} else {
> I would rather use the standard on-stack completion here;
> like this:
>
> 	DECLARE_COMPLETION_ONSTACK(complete);
> 	port->completion = &complete;
> 	sas_unregister_domain_devices(port, gone);
> 	wait_for_completion(&complete);
> 	sas_port_delete(port->port);
>
> which would simplify the above helpers to:
>
> static inline void sas_port_put(struct asd_sas_port *port)
> {
> 	if (port->completion)
> 		kref_put(&port->ref, sas_complete_event);
> }
>
> and you could do away with the 'is_sync' helper.
>

I did wonder if we could avoid using completion altogether and just 
flush the respective queue which the work item is being processed in. 
But, due to the intricacy of SCSI/ATA EH, and since we still use shost 
workqueue for the libsas hotplug processing, maybe it best to keep it 
straightforward and keep using completions.

Anyway, The idea to declare the completion on the stack seems sound. 
And, for patch 6/7, I don't think the is_sync element is even required 
without any change to declaration of completion in struct 
sas_discovery_event.

John

> Cheers,
>
> Hannes
> --

[toc] | [prev] | [next] | [standalone]

#1684003 — [PATCH v3 6/7] libsas: add wait-complete support to sync discovery event

From	Yijing Wang <wangyijing@huawei.com>
Date	2017-07-10 09:10 +0200
Subject	[PATCH v3 6/7] libsas: add wait-complete support to sync discovery event
Message-ID	<u1AZs-1Sl-31@gated-at.bofh.it>
In reply to	#1684001

Introduce a sync flag to tag discovery event whether need to
sync execute, per-event wait-complete ensure sync.

Signed-off-by: Yijing Wang <wangyijing@huawei.com>
CC: John Garry <john.garry@huawei.com>
CC: Johannes Thumshirn <jthumshirn@suse.de>
CC: Ewan Milne <emilne@redhat.com>
CC: Christoph Hellwig <hch@lst.de>
CC: Tomas Henzl <thenzl@redhat.com>
CC: Dan Williams <dan.j.williams@intel.com>
---
 drivers/scsi/libsas/sas_discover.c |  8 ++++++--
 drivers/scsi/libsas/sas_expander.c | 12 +++++++++++-
 drivers/scsi/libsas/sas_internal.h | 27 +++++++++++++++++++++++++++
 include/scsi/libsas.h              |  2 ++
 4 files changed, 46 insertions(+), 3 deletions(-)

diff --git a/drivers/scsi/libsas/sas_discover.c b/drivers/scsi/libsas/sas_discover.c
index a25d648..d68f8dd 100644
--- a/drivers/scsi/libsas/sas_discover.c
+++ b/drivers/scsi/libsas/sas_discover.c
@@ -378,6 +378,7 @@ void sas_unregister_dev(struct asd_sas_port *port, struct domain_device *dev)
 		list_del_init(&dev->disco_list_node);
 		sas_rphy_free(dev->rphy);
 		sas_unregister_common_dev(port, dev);
+		sas_disc_cancel_sync(&port->disc.disc_work[DISCE_DESTRUCT]);
 		return;
 	}
 
@@ -541,6 +542,7 @@ static void sas_discover_common_fn(struct work_struct *work)
 	struct asd_sas_port *port = ev->port;
 
 	sas_event_fns[ev->type](work);
+	sas_disc_wakeup(ev);
 	sas_port_put(port);
 }
 
@@ -571,8 +573,10 @@ static void sas_chain_work(struct sas_ha_struct *ha, struct sas_work *sw)
 	else
 		ret = scsi_queue_work(ha->core.shost, &sw->work);
 
-	if (ret != 1)
+	if (ret != 1) {
 		sas_port_put(port);
+		sas_disc_cancel_sync(ev);
+	}
 }
 
 static void sas_chain_event(int event, unsigned long *pending,
@@ -592,9 +596,9 @@ int sas_discover_event(struct asd_sas_port *port, enum discover_event ev)
 {
 	struct sas_discovery *disc;
 
+	disc = &port->disc;
 	if (!port)
 		return 0;
-	disc = &port->disc;
 
 	BUG_ON(ev >= DISC_NUM_EVENTS);
 
diff --git a/drivers/scsi/libsas/sas_expander.c b/drivers/scsi/libsas/sas_expander.c
index 570b2cb..9d26c28 100644
--- a/drivers/scsi/libsas/sas_expander.c
+++ b/drivers/scsi/libsas/sas_expander.c
@@ -822,14 +822,18 @@ static struct domain_device *sas_ex_discover_end_dev(
 
 		list_add_tail(&child->disco_list_node, &parent->port->disco_list);
 
+		sas_disc_wait_init(child->port, DISCE_PROBE);
 		res = sas_discover_sata(child);
 		if (res) {
+			sas_disc_cancel_sync(&child->port->disc.disc_work[DISCE_PROBE]);
 			SAS_DPRINTK("sas_discover_sata() for device %16llx at "
 				    "%016llx:0x%x returned 0x%x\n",
 				    SAS_ADDR(child->sas_addr),
 				    SAS_ADDR(parent->sas_addr), phy_id, res);
 			goto out_list_del;
 		}
+		sas_disc_wait_completion(child->port, DISCE_PROBE);
+
 	} else
 #endif
 	  if (phy->attached_tproto & SAS_PROTOCOL_SSP) {
@@ -847,14 +851,17 @@ static struct domain_device *sas_ex_discover_end_dev(
 
 		list_add_tail(&child->disco_list_node, &parent->port->disco_list);
 
+		sas_disc_wait_init(child->port, DISCE_PROBE);
 		res = sas_discover_end_dev(child);
 		if (res) {
+			sas_disc_cancel_sync(&child->port->disc.disc_work[DISCE_PROBE]);
 			SAS_DPRINTK("sas_discover_end_dev() for device %16llx "
 				    "at %016llx:0x%x returned 0x%x\n",
 				    SAS_ADDR(child->sas_addr),
 				    SAS_ADDR(parent->sas_addr), phy_id, res);
 			goto out_list_del;
 		}
+		sas_disc_wait_completion(child->port, DISCE_PROBE);
 	} else {
 		SAS_DPRINTK("target proto 0x%x at %016llx:0x%x not handled\n",
 			    phy->attached_tproto, SAS_ADDR(parent->sas_addr),
@@ -1890,8 +1897,11 @@ static void sas_unregister_devs_sas_addr(struct domain_device *parent,
 				if (child->dev_type == SAS_EDGE_EXPANDER_DEVICE ||
 				    child->dev_type == SAS_FANOUT_EXPANDER_DEVICE)
 					sas_unregister_ex_tree(parent->port, child);
-				else
+				else {
+					sas_disc_wait_init(parent->port, DISCE_DESTRUCT);
 					sas_unregister_dev(parent->port, child);
+					sas_disc_wait_completion(parent->port, DISCE_DESTRUCT);
+				}
 				found = child;
 				break;
 			}
diff --git a/drivers/scsi/libsas/sas_internal.h b/drivers/scsi/libsas/sas_internal.h
index 890b5d26..09a9b10 100644
--- a/drivers/scsi/libsas/sas_internal.h
+++ b/drivers/scsi/libsas/sas_internal.h
@@ -134,6 +134,33 @@ static inline void sas_port_get(struct asd_sas_port *port)
 		kref_get(&port->ref);
 }
 
+static inline void sas_disc_cancel_sync(struct sas_discovery_event *event)
+{
+	event->is_sync = false;
+}
+
+static inline void sas_disc_wakeup(struct sas_discovery_event *event)
+{
+	if (event->is_sync)
+		complete(&event->completion);
+}
+
+static inline void sas_disc_wait_init(struct asd_sas_port *port,
+		enum discover_event event)
+{
+	port->disc.disc_work[event].is_sync = true;
+	init_completion(&port->disc.disc_work[event].completion);
+}
+
+static inline void sas_disc_wait_completion(struct asd_sas_port *port,
+		enum discover_event event)
+{
+	if (port->disc.disc_work[event].is_sync) {
+		wait_for_completion(&port->disc.disc_work[event].completion);
+		port->disc.disc_work[event].is_sync = false;
+	}
+}
+
 #ifdef CONFIG_SCSI_SAS_HOST_SMP
 extern int sas_smp_host_handler(struct Scsi_Host *shost, struct request *req,
 				struct request *rsp);
diff --git a/include/scsi/libsas.h b/include/scsi/libsas.h
index 4bcb9fe..21e9fb140 100644
--- a/include/scsi/libsas.h
+++ b/include/scsi/libsas.h
@@ -243,6 +243,8 @@ struct sas_discovery_event {
 	struct sas_work work;
 	struct asd_sas_port *port;
 	enum discover_event	type;
+	bool is_sync;
+	struct completion completion;
 };
 
 static inline struct sas_discovery_event *to_sas_discovery_event(struct work_struct *work)
-- 
2.5.0

[toc] | [prev] | [next] | [standalone]

#1685795 — Re: [PATCH v3 6/7] libsas: add wait-complete support to sync discovery event

From	John Garry <john.garry@huawei.com>
Date	2017-07-12 16:00 +0200
Subject	Re: [PATCH v3 6/7] libsas: add wait-complete support to sync discovery event
Message-ID	<u2qlk-t8-15@gated-at.bofh.it>
In reply to	#1684003

On 10/07/2017 08:06, Yijing Wang wrote:
>
>  static void sas_chain_event(int event, unsigned long *pending,
> @@ -592,9 +596,9 @@ int sas_discover_event(struct asd_sas_port *port, enum discover_event ev)
>  {
>  	struct sas_discovery *disc;
>
> +	disc = &port->disc;
>  	if (!port)
>  		return 0;
> -	disc = &port->disc;
>
>  	BUG_ON(ev >= DISC_NUM_EVENTS);
>
> diff --git a/drivers/scsi/libsas/sas_expander.c b/drivers/scsi/libsas/sas_expander.c
> index 570b2cb..9d26c28 1

I was just looking through the code and I noticed this, above. Is there 
a specific reason to move the NULL check, or was it modified accidentally?

I mean, if port is NULL I don't think we would get as far as checking it 
as we would have already de-referenced it.

[toc] | [prev] | [next] | [standalone]

#1686190 — Re: [PATCH v3 6/7] libsas: add wait-complete support to sync discovery event

From	wangyijing <wangyijing@huawei.com>
Date	2017-07-13 04:30 +0200
Subject	Re: [PATCH v3 6/7] libsas: add wait-complete support to sync discovery event
Message-ID	<u2C38-84q-13@gated-at.bofh.it>
In reply to	#1685795


在 2017/7/12 21:51, John Garry 写道:
> On 10/07/2017 08:06, Yijing Wang wrote:
>>
>>  static void sas_chain_event(int event, unsigned long *pending,
>> @@ -592,9 +596,9 @@ int sas_discover_event(struct asd_sas_port *port, enum discover_event ev)
>>  {
>>      struct sas_discovery *disc;
>>
>> +    disc = &port->disc;
>>      if (!port)
>>          return 0;
>> -    disc = &port->disc;
>>
>>      BUG_ON(ev >= DISC_NUM_EVENTS);
>>
>> diff --git a/drivers/scsi/libsas/sas_expander.c b/drivers/scsi/libsas/sas_expander.c
>> index 570b2cb..9d26c28 1
> 
> I was just looking through the code and I noticed this, above. Is there a specific reason to move the NULL check, or was it modified accidentally?
> 
> I mean, if port is NULL I don't think we would get as far as checking it as we would have already de-referenced it.

Oh, sorry, it's a accidental change, good catch, thanks!

> 
> 
> 
> .
>

[toc] | [prev] | [next] | [standalone]

#1687116 — Re: [PATCH v3 6/7] libsas: add wait-complete support to sync discovery event

From	Hannes Reinecke <hare@suse.de>
Date	2017-07-14 09:00 +0200
Subject	Re: [PATCH v3 6/7] libsas: add wait-complete support to sync discovery event
Message-ID	<u32JY-88H-15@gated-at.bofh.it>
In reply to	#1684003

On 07/10/2017 09:06 AM, Yijing Wang wrote:
> Introduce a sync flag to tag discovery event whether need to
> sync execute, per-event wait-complete ensure sync.
> 
> Signed-off-by: Yijing Wang <wangyijing@huawei.com>
> CC: John Garry <john.garry@huawei.com>
> CC: Johannes Thumshirn <jthumshirn@suse.de>
> CC: Ewan Milne <emilne@redhat.com>
> CC: Christoph Hellwig <hch@lst.de>
> CC: Tomas Henzl <thenzl@redhat.com>
> CC: Dan Williams <dan.j.williams@intel.com>
> ---
>  drivers/scsi/libsas/sas_discover.c |  8 ++++++--
>  drivers/scsi/libsas/sas_expander.c | 12 +++++++++++-
>  drivers/scsi/libsas/sas_internal.h | 27 +++++++++++++++++++++++++++
>  include/scsi/libsas.h              |  2 ++
>  4 files changed, 46 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/scsi/libsas/sas_discover.c b/drivers/scsi/libsas/sas_discover.c
> index a25d648..d68f8dd 100644
> --- a/drivers/scsi/libsas/sas_discover.c
> +++ b/drivers/scsi/libsas/sas_discover.c
> @@ -378,6 +378,7 @@ void sas_unregister_dev(struct asd_sas_port *port, struct domain_device *dev)
>  		list_del_init(&dev->disco_list_node);
>  		sas_rphy_free(dev->rphy);
>  		sas_unregister_common_dev(port, dev);
> +		sas_disc_cancel_sync(&port->disc.disc_work[DISCE_DESTRUCT]);
>  		return;
>  	}
>  
> @@ -541,6 +542,7 @@ static void sas_discover_common_fn(struct work_struct *work)
>  	struct asd_sas_port *port = ev->port;
>  
>  	sas_event_fns[ev->type](work);
> +	sas_disc_wakeup(ev);
>  	sas_port_put(port);
>  }
>  
> @@ -571,8 +573,10 @@ static void sas_chain_work(struct sas_ha_struct *ha, struct sas_work *sw)
>  	else
>  		ret = scsi_queue_work(ha->core.shost, &sw->work);
>  
> -	if (ret != 1)
> +	if (ret != 1) {
>  		sas_port_put(port);
> +		sas_disc_cancel_sync(ev);
> +	}
>  }
>  
>  static void sas_chain_event(int event, unsigned long *pending,
> @@ -592,9 +596,9 @@ int sas_discover_event(struct asd_sas_port *port, enum discover_event ev)
>  {
>  	struct sas_discovery *disc;
>  
> +	disc = &port->disc;
>  	if (!port)
>  		return 0;
> -	disc = &port->disc;
>  
>  	BUG_ON(ev >= DISC_NUM_EVENTS);
>  
> diff --git a/drivers/scsi/libsas/sas_expander.c b/drivers/scsi/libsas/sas_expander.c
> index 570b2cb..9d26c28 100644
> --- a/drivers/scsi/libsas/sas_expander.c
> +++ b/drivers/scsi/libsas/sas_expander.c
> @@ -822,14 +822,18 @@ static struct domain_device *sas_ex_discover_end_dev(
>  
>  		list_add_tail(&child->disco_list_node, &parent->port->disco_list);
>  
> +		sas_disc_wait_init(child->port, DISCE_PROBE);
>  		res = sas_discover_sata(child);
>  		if (res) {
> +			sas_disc_cancel_sync(&child->port->disc.disc_work[DISCE_PROBE]);
>  			SAS_DPRINTK("sas_discover_sata() for device %16llx at "
>  				    "%016llx:0x%x returned 0x%x\n",
>  				    SAS_ADDR(child->sas_addr),
>  				    SAS_ADDR(parent->sas_addr), phy_id, res);
>  			goto out_list_del;
>  		}
> +		sas_disc_wait_completion(child->port, DISCE_PROBE);
> +
>  	} else
>  #endif
>  	  if (phy->attached_tproto & SAS_PROTOCOL_SSP) {
> @@ -847,14 +851,17 @@ static struct domain_device *sas_ex_discover_end_dev(
>  
>  		list_add_tail(&child->disco_list_node, &parent->port->disco_list);
>  
> +		sas_disc_wait_init(child->port, DISCE_PROBE);
>  		res = sas_discover_end_dev(child);
>  		if (res) {
> +			sas_disc_cancel_sync(&child->port->disc.disc_work[DISCE_PROBE]);
>  			SAS_DPRINTK("sas_discover_end_dev() for device %16llx "
>  				    "at %016llx:0x%x returned 0x%x\n",
>  				    SAS_ADDR(child->sas_addr),
>  				    SAS_ADDR(parent->sas_addr), phy_id, res);
>  			goto out_list_del;
>  		}
> +		sas_disc_wait_completion(child->port, DISCE_PROBE);
>  	} else {
>  		SAS_DPRINTK("target proto 0x%x at %016llx:0x%x not handled\n",
>  			    phy->attached_tproto, SAS_ADDR(parent->sas_addr),
> @@ -1890,8 +1897,11 @@ static void sas_unregister_devs_sas_addr(struct domain_device *parent,
>  				if (child->dev_type == SAS_EDGE_EXPANDER_DEVICE ||
>  				    child->dev_type == SAS_FANOUT_EXPANDER_DEVICE)
>  					sas_unregister_ex_tree(parent->port, child);
> -				else
> +				else {
> +					sas_disc_wait_init(parent->port, DISCE_DESTRUCT);
>  					sas_unregister_dev(parent->port, child);
> +					sas_disc_wait_completion(parent->port, DISCE_DESTRUCT);
> +				}
>  				found = child;
>  				break;
>  			}
> diff --git a/drivers/scsi/libsas/sas_internal.h b/drivers/scsi/libsas/sas_internal.h
> index 890b5d26..09a9b10 100644
> --- a/drivers/scsi/libsas/sas_internal.h
> +++ b/drivers/scsi/libsas/sas_internal.h
> @@ -134,6 +134,33 @@ static inline void sas_port_get(struct asd_sas_port *port)
>  		kref_get(&port->ref);
>  }
>  
> +static inline void sas_disc_cancel_sync(struct sas_discovery_event *event)
> +{
> +	event->is_sync = false;
> +}
> +
> +static inline void sas_disc_wakeup(struct sas_discovery_event *event)
> +{
> +	if (event->is_sync)
> +		complete(&event->completion);
> +}
> +
> +static inline void sas_disc_wait_init(struct asd_sas_port *port,
> +		enum discover_event event)
> +{
> +	port->disc.disc_work[event].is_sync = true;
> +	init_completion(&port->disc.disc_work[event].completion);
> +}
> +
> +static inline void sas_disc_wait_completion(struct asd_sas_port *port,
> +		enum discover_event event)
> +{
> +	if (port->disc.disc_work[event].is_sync) {
> +		wait_for_completion(&port->disc.disc_work[event].completion);
> +		port->disc.disc_work[event].is_sync = false;
> +	}
> +}
> +
>  #ifdef CONFIG_SCSI_SAS_HOST_SMP
>  extern int sas_smp_host_handler(struct Scsi_Host *shost, struct request *req,
>  				struct request *rsp);
> diff --git a/include/scsi/libsas.h b/include/scsi/libsas.h
> index 4bcb9fe..21e9fb140 100644
> --- a/include/scsi/libsas.h
> +++ b/include/scsi/libsas.h
> @@ -243,6 +243,8 @@ struct sas_discovery_event {
>  	struct sas_work work;
>  	struct asd_sas_port *port;
>  	enum discover_event	type;
> +	bool is_sync;
> +	struct completion completion;
>  };
>  
>  static inline struct sas_discovery_event *to_sas_discovery_event(struct work_struct *work)
> 
Some comments as the earlier patch: please use DECLARE_COMPETION_ONSTACK
here.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

[toc] | [prev] | [next] | [standalone]

#1684004 — [PATCH v3 3/7] libsas: Use new workqueue to run sas event

From	Yijing Wang <wangyijing@huawei.com>
Date	2017-07-10 09:10 +0200
Subject	[PATCH v3 3/7] libsas: Use new workqueue to run sas event
Message-ID	<u1AZt-1Sl-39@gated-at.bofh.it>
In reply to	#1684001

Now all libsas works are queued to scsi host workqueue,
include sas event work post by LLDD and sas discovery
work, and a sas hotplug flow may be divided into several
works, e.g libsas receive a PORTE_BYTES_DMAED event,
now we process it as following steps:
sas_form_port  --- run in work in shot workq
	sas_discover_domain  --- run in another work in shost workq
		...
		sas_probe_devices  --- run in new work in shost workq
We found during hot-add a device, libsas may need run several
works in same workqueue to add device in system, the process is
not atomic, it may interrupt by other sas event works, like
PHYE_LOSS_OF_SIGNAL. Finally, we would found lots unexpected
errors. This patch is preparation of execute libsas sas event
in sync.

Signed-off-by: Yijing Wang <wangyijing@huawei.com>
CC: John Garry <john.garry@huawei.com>
CC: Johannes Thumshirn <jthumshirn@suse.de>
CC: Ewan Milne <emilne@redhat.com>
CC: Christoph Hellwig <hch@lst.de>
CC: Tomas Henzl <thenzl@redhat.com>
CC: Dan Williams <dan.j.williams@intel.com>
---
 drivers/scsi/libsas/sas_event.c | 4 ++--
 drivers/scsi/libsas/sas_init.c  | 7 +++++++
 include/scsi/libsas.h           | 1 +
 3 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/drivers/scsi/libsas/sas_event.c b/drivers/scsi/libsas/sas_event.c
index a1370bd..a72a089 100644
--- a/drivers/scsi/libsas/sas_event.c
+++ b/drivers/scsi/libsas/sas_event.c
@@ -46,7 +46,7 @@ int sas_queue_work(struct sas_ha_struct *ha, struct sas_work *sw)
 		if (list_empty(&sw->drain_node))
 			list_add(&sw->drain_node, &ha->defer_q);
 	} else
-		rc = scsi_queue_work(ha->core.shost, &sw->work);
+		rc = queue_work(ha->event_q, &sw->work);
 
 	return rc;
 }
@@ -69,7 +69,7 @@ void __sas_drain_work(struct sas_ha_struct *ha)
 {
 	int ret;
 	unsigned long flags;
-	struct workqueue_struct *wq = ha->core.shost->work_q;
+	struct workqueue_struct *wq = ha->event_q;
 	struct sas_work *sw, *_sw;
 
 	set_bit(SAS_HA_DRAINING, &ha->state);
diff --git a/drivers/scsi/libsas/sas_init.c b/drivers/scsi/libsas/sas_init.c
index c227a8b..2f3b736 100644
--- a/drivers/scsi/libsas/sas_init.c
+++ b/drivers/scsi/libsas/sas_init.c
@@ -115,6 +115,7 @@ void sas_hae_reset(struct work_struct *work)
 
 int sas_register_ha(struct sas_ha_struct *sas_ha)
 {
+	char name[64];
 	int error = 0;
 
 	mutex_init(&sas_ha->disco_mutex);
@@ -146,6 +147,11 @@ int sas_register_ha(struct sas_ha_struct *sas_ha)
 		goto Undo_ports;
 	}
 
+	snprintf(name, 64, "%s_event_q", dev_name(sas_ha->dev));
+	sas_ha->event_q = create_singlethread_workqueue(name);
+	if (!sas_ha->event_q)
+		goto Undo_ports;
+
 	INIT_LIST_HEAD(&sas_ha->eh_done_q);
 	INIT_LIST_HEAD(&sas_ha->eh_ata_q);
 
@@ -180,6 +186,7 @@ int sas_unregister_ha(struct sas_ha_struct *sas_ha)
 	mutex_lock(&sas_ha->drain_mutex);
 	__sas_drain_work(sas_ha);
 	mutex_unlock(&sas_ha->drain_mutex);
+	destroy_workqueue(sas_ha->event_q);
 
 	return 0;
 }
diff --git a/include/scsi/libsas.h b/include/scsi/libsas.h
index 628f48b..a01ca42 100644
--- a/include/scsi/libsas.h
+++ b/include/scsi/libsas.h
@@ -402,6 +402,7 @@ struct sas_ha_struct {
 	char *sas_ha_name;
 	struct device *dev;	  /* should be set */
 	struct module *lldd_module; /* should be set */
+	struct workqueue_struct *event_q;
 
 	u8 *sas_addr;		  /* must be set */
 	u8 hashed_sas_addr[HASHED_SAS_ADDR_SIZE];
-- 
2.5.0

[toc] | [prev] | [next] | [standalone]

#1687109 — Re: [PATCH v3 3/7] libsas: Use new workqueue to run sas event

From	Hannes Reinecke <hare@suse.de>
Date	2017-07-14 08:50 +0200
Subject	Re: [PATCH v3 3/7] libsas: Use new workqueue to run sas event
Message-ID	<u32Aj-84Q-45@gated-at.bofh.it>
In reply to	#1684004

On 07/10/2017 09:06 AM, Yijing Wang wrote:
> Now all libsas works are queued to scsi host workqueue,
> include sas event work post by LLDD and sas discovery
> work, and a sas hotplug flow may be divided into several
> works, e.g libsas receive a PORTE_BYTES_DMAED event,
> now we process it as following steps:
> sas_form_port  --- run in work in shot workq
> 	sas_discover_domain  --- run in another work in shost workq
> 		...
> 		sas_probe_devices  --- run in new work in shost workq
> We found during hot-add a device, libsas may need run several
> works in same workqueue to add device in system, the process is
> not atomic, it may interrupt by other sas event works, like
> PHYE_LOSS_OF_SIGNAL. Finally, we would found lots unexpected
> errors. This patch is preparation of execute libsas sas event
> in sync.
> 
> Signed-off-by: Yijing Wang <wangyijing@huawei.com>
> CC: John Garry <john.garry@huawei.com>
> CC: Johannes Thumshirn <jthumshirn@suse.de>
> CC: Ewan Milne <emilne@redhat.com>
> CC: Christoph Hellwig <hch@lst.de>
> CC: Tomas Henzl <thenzl@redhat.com>
> CC: Dan Williams <dan.j.williams@intel.com>
> ---
>  drivers/scsi/libsas/sas_event.c | 4 ++--
>  drivers/scsi/libsas/sas_init.c  | 7 +++++++
>  include/scsi/libsas.h           | 1 +
>  3 files changed, 10 insertions(+), 2 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare@suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

[toc] | [prev] | [next] | [standalone]

#1685674

From	John Garry <john.garry@huawei.com>
Date	2017-07-12 12:10 +0200
Message-ID	<u2mKK-6Tu-19@gated-at.bofh.it>
In reply to	#1684001

On 10/07/2017 08:06, Yijing Wang wrote:
> This patchset is based Johannes's patch
> "scsi: sas: scsi_queue_work can fail, so make callers aware"
>
> Now the libsas hotplug has some issues, Dan Williams report
> a similar bug here before
> https://www.mail-archive.com/linux-scsi@vger.kernel.org/msg39187.html
>
> The issues we have found
> 1. if LLDD burst reports lots of phy-up/phy-down sas events, some events
>    may lost because a same sas events is pending now, finally libsas topo
>    may different the hardware.
> 2. receive a phy down sas event, libsas call sas_deform_port to remove
>    devices, it would first delete the sas port, then put a destruction
>    discovery event in a new work, and queue it at the tail of workqueue,
>    once the sas port be deleted, its children device will be deleted too,
>    when the destruction work start, it will found the target device has
>    been removed, and report a sysfs warnning.
> 3. since a hotplug process will be devided into several works, if a phy up
>    sas event insert into phydown works, like
>    destruction work  ---> PORTE_BYTES_DMAED (sas_form_port) ---->PHYE_LOSS_OF_SIGNAL
>    the hot remove flow would broken by PORTE_BYTES_DMAED event, it's not
>    we expected, and issues would occur.
>
> The first patch fix the sas events lost, and the second one introudce wait-complete
> to fix the hotplug order issues.
>

I quickly tested this for basic hotplug.

Before:
root@(none)$ echo 0 > ./phy-0:6/sas_phy/phy-0:6/enable
root@(none)$ echo 0 > ./phy-0:5/sas_phy/phy-0:5/enable
root@(none)$ echo 0 > ./phy-0:4/sas_phy/phy-0:4/enable
root@(none)$ echo 0 > ./phy-0:3/sas_phy/phy-0:3/enable
root@(none)$ echo 0 > ./phy-0:3/sas_phy/phy-0:2/enable
root@(none)$ echo 0 > ./phy-0:2/sas_phy/phy-0:2/enable
root@(none)$ echo 0 > ./phy-0:1/sas_phy/phy-0:1/enable
root@(none)$ echo 0 > ./phy-0:0/sas_phy/phy-0:0/enable
root@(none)$ echo 0 > ./phy-0:7/sas_phy/phy-0:7/enable
root@(none)$ [  102.570694] sysfs group 'power' not found for kobject 
'0:0:7:0'
[  102.577250] ------------[ cut here ]------------
[  102.581861] WARNING: CPU: 3 PID: 1740 at fs/sysfs/group.c:237 
sysfs_remove_group+0x8c/0x94
[  102.590110] Modules linked in:
[  102.593154] CPU: 3 PID: 1740 Comm: kworker/u128:2 Not tainted 
4.12.0-rc1-00032-g3ab81fc #1907
[  102.601664] Hardware name: Huawei Taishan 2280 /D05, BIOS Hisilicon 
D05 UEFI Nemo 1.7 RC3 06/23/2017
[  102.610784] Workqueue: scsi_wq_0 sas_destruct_devices
[  102.615822] task: ffff8017d4793400 task.stack: ffff8017b7e70000
[  102.621728] PC is at sysfs_remove_group+0x8c/0x94
[  102.626419] LR is at sysfs_remove_group+0x8c/0x94
[  102.631109] pc : [<ffff000008267c44>] lr : [<ffff000008267c44>] 
pstate: 60000045
[  102.638490] sp : ffff8017b7e73b80
[  102.641791] x29: ffff8017b7e73b80 x28: ffff8017db010800
[  102.647091] x27: ffff000008e27000 x26: ffff8017d43e6600
[  102.652390] x25: ffff8017b8280000 x24: 0000000000000003
[  102.657689] x23: ffff8017b78864b0 x22: ffff8017b784c988
[  102.662988] x21: ffff8017b7886410 x20: ffff000008ee9dd0
[  102.668288] x19: 0000000000000000 x18: ffff000008a1b678
[  102.673587] x17: 000000000000000e x16: 0000000000000007
[  102.678886] x15: 0000000000000000 x14: 00000000000000a3
[  102.684185] x13: 0000000000000033 x12: 0000000000000028
[  102.689484] x11: ffff000008f3be58 x10: 0000000000000000
[  102.694783] x9 : 000000000000043c x8 : 6f6b20726f662064
[  102.700082] x7 : ffff000008e29e08 x6 : ffff8017fbe34c50
[  102.705382] x5 : 0000000000000000 x4 : 0000000000000000
[  102.710681] x3 : ffffffffffffffff x2 : ffff000008e427e0
[  102.715980] x1 : 0000000000000000 x0 : 0000000000000033
[  102.721279] ---[ end trace c216cc1451d5f7ec ]---
[  102.725882] Call trace:
[  102.728316] Exception stack(0xffff8017b7e739b0 to 0xffff8017b7e73ae0)
[  102.734742] 39a0:                                   0000000000000000 
0001000000000000
[  102.742557] 39c0: ffff8017b7e73b80 ffff000008267c44 ffff000008bfa050 
0000000000000000
[  102.750372] 39e0: ffff8017b78864b0 0000000000000003 ffff8017b8280000 
ffff8017d43e6600
[  102.758188] 3a00: ffff000008e27000 ffff8017db010800 ffff8017d4793400 
0000000000000000
[  102.766003] 3a20: ffff8017b7e73b80 ffff8017b7e73b80 ffff8017b7e73b40 
00000000ffffffc8
[  102.773818] 3a40: ffff8017b7e73a70 ffff00000810c12c 0000000000000033 
0000000000000000
[  102.781633] 3a60: ffff000008e427e0 ffffffffffffffff 0000000000000000 
0000000000000000
[  102.789449] 3a80: ffff8017fbe34c50 ffff000008e29e08 6f6b20726f662064 
000000000000043c
[  102.797264] 3aa0: 0000000000000000 ffff000008f3be58 0000000000000028 
0000000000000033
[  102.805079] 3ac0: 00000000000000a3 0000000000000000 0000000000000007 
000000000000000e
[  102.812895] [<ffff000008267c44>] sysfs_remove_group+0x8c/0x94
[  102.818628] [<ffff00000855b14c>] dpm_sysfs_remove+0x58/0x68
[  102.824188] [<ffff00000854e0e8>] device_del+0xf8/0x2d0
[  102.829312] [<ffff00000854e2d4>] device_unregister+0x14/0x2c
[  102.834959] [<ffff00000837e6e0>] bsg_unregister_queue+0x60/0x98
[  102.840866] [<ffff000008593cd4>] __scsi_remove_device+0xa0/0xbc

<snip>

[  151.331854] 3bc0: ffff0000081f21ac 0000ffff803370c0
[  151.336718] [<ffff000008267c44>] sysfs_remove_group+0x8c/0x94
[  151.342449] [<ffff00000855b14c>] dpm_sysfs_remove+0x58/0x68
[  151.348008] [<ffff00000854e0e8>] device_del+0xf8/0x2d0
[  151.353133] [<ffff000008597278>] sas_rphy_remove+0x54/0x80
[  151.358604] [<ffff0000085972b8>] sas_rphy_delete+0x14/0x28
[  151.364076] [<ffff00000859b304>] sas_destruct_devices+0x64/0x98
[  151.369982] [<ffff0000080d8194>] process_one_work+0x12c/0x28c
[  151.375714] [<ffff0000080d834c>] worker_thread+0x58/0x3b8
[  151.381100] [<ffff0000080ddee4>] kthread+0x100/0x12c
[  151.386050] [<ffff0000080836c0>] ret_from_fork+0x10/0x50
[  151.391360] hisi_sas_v2_hw HISI0162:01: found dev[0:2] is gone

root@(none)$

So the console locks for ~50 seconds with WARN garbage.

After:
...
root@(none)$ echo 0 > ./phy-0:7/sas_phy/phy-0:7/enable
root@(none)$ [  446.193336] hisi_sas_v2_hw HISI0162:01: found dev[8:1] 
is gone
[  446.249205] hisi_sas_v2_hw HISI0162:01: found dev[7:1] is gone
[  446.325201] hisi_sas_v2_hw HISI0162:01: found dev[6:1] is gone
[  446.373189] hisi_sas_v2_hw HISI0162:01: found dev[5:1] is gone
[  446.421187] hisi_sas_v2_hw HISI0162:01: found dev[4:1] is gone
[  446.457232] hisi_sas_v2_hw HISI0162:01: found dev[3:1] is gone
[  446.477151] sd 0:0:1:0: [sdb] Synchronizing SCSI cache
[  446.482373] sd 0:0:1:0: [sdb] Synchronize Cache(10) failed: Result: 
hostbyte=0x04 driverbyte=0x00
[  446.491238] sd 0:0:1:0: [sdb] Stopping disk
[  446.495419] sd 0:0:1:0: [sdb] Start/Stop Unit failed: Result: 
hostbyte=0x04 driverbyte=0x00
[  446.525227] hisi_sas_v2_hw HISI0162:01: found dev[2:5] is gone
[  446.569249] hisi_sas_v2_hw HISI0162:01: found dev[1:1] is gone
[  446.576872] hisi_sas_v2_hw HISI0162:01: found dev[0:2] is gone

root@(none)$

So much nicer. BTW, /dev/sdb is a SATA disk, the rest are SAS.

John

> v2->v3: some code improvements suggested by Johannes and John,
> 	    split v2 patch 2 into several small pathes.
> v1->v2: some code improvements suggested by John Garry
>
> Yijing Wang (7):
>   libsas: Use static sas event pool to appease sas event lost
>   libsas: remove unused port_gone_completion
>   libsas: Use new workqueue to run sas event
>   libsas: add sas event wait-complete support
>   libsas: add a new workqueue to run probe/destruct discovery event
>   libsas: add wait-complete support to sync discovery event
>   libsas: release disco mutex during waiting in sas_ex_discover_end_dev
>
>  drivers/scsi/libsas/sas_discover.c |  58 +++++++---
>  drivers/scsi/libsas/sas_event.c    | 212 ++++++++++++++++++++++++++++++++-----
>  drivers/scsi/libsas/sas_expander.c |  22 +++-
>  drivers/scsi/libsas/sas_init.c     |  21 ++--
>  drivers/scsi/libsas/sas_internal.h |  64 +++++++++++
>  drivers/scsi/libsas/sas_phy.c      |  48 +++------
>  drivers/scsi/libsas/sas_port.c     |  22 ++--
>  include/scsi/libsas.h              |  27 +++--
>  8 files changed, 373 insertions(+), 101 deletions(-)
>

[toc] | [prev] | [next] | [standalone]

#1685724

From	Johannes Thumshirn <jthumshirn@suse.de>
Date	2017-07-12 14:00 +0200
Message-ID	<u2otc-7MP-11@gated-at.bofh.it>
In reply to	#1685674

On Wed, Jul 12, 2017 at 10:59:27AM +0100, John Garry wrote:
> After:
> ...
> root@(none)$ echo 0 > ./phy-0:7/sas_phy/phy-0:7/enable
> root@(none)$ [  446.193336] hisi_sas_v2_hw HISI0162:01: found dev[8:1] is
> gone
> [  446.249205] hisi_sas_v2_hw HISI0162:01: found dev[7:1] is gone
> [  446.325201] hisi_sas_v2_hw HISI0162:01: found dev[6:1] is gone
> [  446.373189] hisi_sas_v2_hw HISI0162:01: found dev[5:1] is gone
> [  446.421187] hisi_sas_v2_hw HISI0162:01: found dev[4:1] is gone
> [  446.457232] hisi_sas_v2_hw HISI0162:01: found dev[3:1] is gone
> [  446.477151] sd 0:0:1:0: [sdb] Synchronizing SCSI cache
> [  446.482373] sd 0:0:1:0: [sdb] Synchronize Cache(10) failed: Result:
> hostbyte=0x04 driverbyte=0x00
> [  446.491238] sd 0:0:1:0: [sdb] Stopping disk
> [  446.495419] sd 0:0:1:0: [sdb] Start/Stop Unit failed: Result:
> hostbyte=0x04 driverbyte=0x00
> [  446.525227] hisi_sas_v2_hw HISI0162:01: found dev[2:5] is gone
> [  446.569249] hisi_sas_v2_hw HISI0162:01: found dev[1:1] is gone
> [  446.576872] hisi_sas_v2_hw HISI0162:01: found dev[0:2] is gone
> 
> root@(none)$
> 
> So much nicer. BTW, /dev/sdb is a SATA disk, the rest are SAS.

This is awesome. I hope I have some time reviewing the patches themselfes
soon.

	Johannes

-- 
Johannes Thumshirn                                          Storage
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

[toc] | [prev] | [next] | [standalone]

#1686167

From	wangyijing <wangyijing@huawei.com>
Date	2017-07-13 03:30 +0200
Message-ID	<u2B73-7qj-3@gated-at.bofh.it>
In reply to	#1685674


在 2017/7/12 17:59, John Garry 写道:
> On 10/07/2017 08:06, Yijing Wang wrote:
>> This patchset is based Johannes's patch
>> "scsi: sas: scsi_queue_work can fail, so make callers aware"
>>
>> Now the libsas hotplug has some issues, Dan Williams report
>> a similar bug here before
>> https://www.mail-archive.com/linux-scsi@vger.kernel.org/msg39187.html
>>
>> The issues we have found
>> 1. if LLDD burst reports lots of phy-up/phy-down sas events, some events
>>    may lost because a same sas events is pending now, finally libsas topo
>>    may different the hardware.
>> 2. receive a phy down sas event, libsas call sas_deform_port to remove
>>    devices, it would first delete the sas port, then put a destruction
>>    discovery event in a new work, and queue it at the tail of workqueue,
>>    once the sas port be deleted, its children device will be deleted too,
>>    when the destruction work start, it will found the target device has
>>    been removed, and report a sysfs warnning.
>> 3. since a hotplug process will be devided into several works, if a phy up
>>    sas event insert into phydown works, like
>>    destruction work  ---> PORTE_BYTES_DMAED (sas_form_port) ---->PHYE_LOSS_OF_SIGNAL
>>    the hot remove flow would broken by PORTE_BYTES_DMAED event, it's not
>>    we expected, and issues would occur.
>>
>> The first patch fix the sas events lost, and the second one introudce wait-complete
>> to fix the hotplug order issues.
>>
> 
> I quickly tested this for basic hotplug.
> 
> Before:
> root@(none)$ echo 0 > ./phy-0:6/sas_phy/phy-0:6/enable
> root@(none)$ echo 0 > ./phy-0:5/sas_phy/phy-0:5/enable
> root@(none)$ echo 0 > ./phy-0:4/sas_phy/phy-0:4/enable
> root@(none)$ echo 0 > ./phy-0:3/sas_phy/phy-0:3/enable
> root@(none)$ echo 0 > ./phy-0:3/sas_phy/phy-0:2/enable
> root@(none)$ echo 0 > ./phy-0:2/sas_phy/phy-0:2/enable
> root@(none)$ echo 0 > ./phy-0:1/sas_phy/phy-0:1/enable
> root@(none)$ echo 0 > ./phy-0:0/sas_phy/phy-0:0/enable
> root@(none)$ echo 0 > ./phy-0:7/sas_phy/phy-0:7/enable
> root@(none)$ [  102.570694] sysfs group 'power' not found for kobject '0:0:7:0'
> [  102.577250] ------------[ cut here ]------------
> [  102.581861] WARNING: CPU: 3 PID: 1740 at fs/sysfs/group.c:237 sysfs_remove_group+0x8c/0x94
> [  102.590110] Modules linked in:
> [  102.593154] CPU: 3 PID: 1740 Comm: kworker/u128:2 Not tainted 4.12.0-rc1-00032-g3ab81fc #1907
> [  102.601664] Hardware name: Huawei Taishan 2280 /D05, BIOS Hisilicon D05 UEFI Nemo 1.7 RC3 06/23/2017
> [  102.610784] Workqueue: scsi_wq_0 sas_destruct_devices
> [  102.615822] task: ffff8017d4793400 task.stack: ffff8017b7e70000
> [  102.621728] PC is at sysfs_remove_group+0x8c/0x94
> [  102.626419] LR is at sysfs_remove_group+0x8c/0x94
> [  102.631109] pc : [<ffff000008267c44>] lr : [<ffff000008267c44>] pstate: 60000045
> [  102.638490] sp : ffff8017b7e73b80
> [  102.641791] x29: ffff8017b7e73b80 x28: ffff8017db010800
> [  102.647091] x27: ffff000008e27000 x26: ffff8017d43e6600
> [  102.652390] x25: ffff8017b8280000 x24: 0000000000000003
> [  102.657689] x23: ffff8017b78864b0 x22: ffff8017b784c988
> [  102.662988] x21: ffff8017b7886410 x20: ffff000008ee9dd0
> [  102.668288] x19: 0000000000000000 x18: ffff000008a1b678
> [  102.673587] x17: 000000000000000e x16: 0000000000000007
> [  102.678886] x15: 0000000000000000 x14: 00000000000000a3
> [  102.684185] x13: 0000000000000033 x12: 0000000000000028
> [  102.689484] x11: ffff000008f3be58 x10: 0000000000000000
> [  102.694783] x9 : 000000000000043c x8 : 6f6b20726f662064
> [  102.700082] x7 : ffff000008e29e08 x6 : ffff8017fbe34c50
> [  102.705382] x5 : 0000000000000000 x4 : 0000000000000000
> [  102.710681] x3 : ffffffffffffffff x2 : ffff000008e427e0
> [  102.715980] x1 : 0000000000000000 x0 : 0000000000000033
> [  102.721279] ---[ end trace c216cc1451d5f7ec ]---
> [  102.725882] Call trace:
> [  102.728316] Exception stack(0xffff8017b7e739b0 to 0xffff8017b7e73ae0)
> [  102.734742] 39a0:                                   0000000000000000 0001000000000000
> [  102.742557] 39c0: ffff8017b7e73b80 ffff000008267c44 ffff000008bfa050 0000000000000000
> [  102.750372] 39e0: ffff8017b78864b0 0000000000000003 ffff8017b8280000 ffff8017d43e6600
> [  102.758188] 3a00: ffff000008e27000 ffff8017db010800 ffff8017d4793400 0000000000000000
> [  102.766003] 3a20: ffff8017b7e73b80 ffff8017b7e73b80 ffff8017b7e73b40 00000000ffffffc8
> [  102.773818] 3a40: ffff8017b7e73a70 ffff00000810c12c 0000000000000033 0000000000000000
> [  102.781633] 3a60: ffff000008e427e0 ffffffffffffffff 0000000000000000 0000000000000000
> [  102.789449] 3a80: ffff8017fbe34c50 ffff000008e29e08 6f6b20726f662064 000000000000043c
> [  102.797264] 3aa0: 0000000000000000 ffff000008f3be58 0000000000000028 0000000000000033
> [  102.805079] 3ac0: 00000000000000a3 0000000000000000 0000000000000007 000000000000000e
> [  102.812895] [<ffff000008267c44>] sysfs_remove_group+0x8c/0x94
> [  102.818628] [<ffff00000855b14c>] dpm_sysfs_remove+0x58/0x68
> [  102.824188] [<ffff00000854e0e8>] device_del+0xf8/0x2d0
> [  102.829312] [<ffff00000854e2d4>] device_unregister+0x14/0x2c
> [  102.834959] [<ffff00000837e6e0>] bsg_unregister_queue+0x60/0x98
> [  102.840866] [<ffff000008593cd4>] __scsi_remove_device+0xa0/0xbc
> 
> <snip>
> 
> [  151.331854] 3bc0: ffff0000081f21ac 0000ffff803370c0
> [  151.336718] [<ffff000008267c44>] sysfs_remove_group+0x8c/0x94
> [  151.342449] [<ffff00000855b14c>] dpm_sysfs_remove+0x58/0x68
> [  151.348008] [<ffff00000854e0e8>] device_del+0xf8/0x2d0
> [  151.353133] [<ffff000008597278>] sas_rphy_remove+0x54/0x80
> [  151.358604] [<ffff0000085972b8>] sas_rphy_delete+0x14/0x28
> [  151.364076] [<ffff00000859b304>] sas_destruct_devices+0x64/0x98
> [  151.369982] [<ffff0000080d8194>] process_one_work+0x12c/0x28c
> [  151.375714] [<ffff0000080d834c>] worker_thread+0x58/0x3b8
> [  151.381100] [<ffff0000080ddee4>] kthread+0x100/0x12c
> [  151.386050] [<ffff0000080836c0>] ret_from_fork+0x10/0x50
> [  151.391360] hisi_sas_v2_hw HISI0162:01: found dev[0:2] is gone
> 
> root@(none)$
> 
> So the console locks for ~50 seconds with WARN garbage.
> 
> After:
> ...
> root@(none)$ echo 0 > ./phy-0:7/sas_phy/phy-0:7/enable
> root@(none)$ [  446.193336] hisi_sas_v2_hw HISI0162:01: found dev[8:1] is gone
> [  446.249205] hisi_sas_v2_hw HISI0162:01: found dev[7:1] is gone
> [  446.325201] hisi_sas_v2_hw HISI0162:01: found dev[6:1] is gone
> [  446.373189] hisi_sas_v2_hw HISI0162:01: found dev[5:1] is gone
> [  446.421187] hisi_sas_v2_hw HISI0162:01: found dev[4:1] is gone
> [  446.457232] hisi_sas_v2_hw HISI0162:01: found dev[3:1] is gone
> [  446.477151] sd 0:0:1:0: [sdb] Synchronizing SCSI cache
> [  446.482373] sd 0:0:1:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=0x04 driverbyte=0x00
> [  446.491238] sd 0:0:1:0: [sdb] Stopping disk
> [  446.495419] sd 0:0:1:0: [sdb] Start/Stop Unit failed: Result: hostbyte=0x04 driverbyte=0x00
> [  446.525227] hisi_sas_v2_hw HISI0162:01: found dev[2:5] is gone
> [  446.569249] hisi_sas_v2_hw HISI0162:01: found dev[1:1] is gone
> [  446.576872] hisi_sas_v2_hw HISI0162:01: found dev[0:2] is gone
> 
> root@(none)$
> 
> So much nicer. BTW, /dev/sdb is a SATA disk, the rest are SAS.

I will check the calltrace, I tested in my local branch, the result is fine.

Thanks!
Yijing.


> 
> John
> 
>> v2->v3: some code improvements suggested by Johannes and John,
>>         split v2 patch 2 into several small pathes.
>> v1->v2: some code improvements suggested by John Garry
>>
>> Yijing Wang (7):
>>   libsas: Use static sas event pool to appease sas event lost
>>   libsas: remove unused port_gone_completion
>>   libsas: Use new workqueue to run sas event
>>   libsas: add sas event wait-complete support
>>   libsas: add a new workqueue to run probe/destruct discovery event
>>   libsas: add wait-complete support to sync discovery event
>>   libsas: release disco mutex during waiting in sas_ex_discover_end_dev
>>
>>  drivers/scsi/libsas/sas_discover.c |  58 +++++++---
>>  drivers/scsi/libsas/sas_event.c    | 212 ++++++++++++++++++++++++++++++++-----
>>  drivers/scsi/libsas/sas_expander.c |  22 +++-
>>  drivers/scsi/libsas/sas_init.c     |  21 ++--
>>  drivers/scsi/libsas/sas_internal.h |  64 +++++++++++
>>  drivers/scsi/libsas/sas_phy.c      |  48 +++------
>>  drivers/scsi/libsas/sas_port.c     |  22 ++--
>>  include/scsi/libsas.h              |  27 +++--
>>  8 files changed, 373 insertions(+), 101 deletions(-)
>>
> 
> 
> 
> .
>

[toc] | [prev] | [next] | [standalone]

#1686170

From	wangyijing <wangyijing@huawei.com>
Date	2017-07-13 03:40 +0200
Message-ID	<u2BgJ-7th-1@gated-at.bofh.it>
In reply to	#1685674


在 2017/7/12 17:59, John Garry 写道:
> On 10/07/2017 08:06, Yijing Wang wrote:
>> This patchset is based Johannes's patch
>> "scsi: sas: scsi_queue_work can fail, so make callers aware"
>>
>> Now the libsas hotplug has some issues, Dan Williams report
>> a similar bug here before
>> https://www.mail-archive.com/linux-scsi@vger.kernel.org/msg39187.html
>>
>> The issues we have found
>> 1. if LLDD burst reports lots of phy-up/phy-down sas events, some events
>>    may lost because a same sas events is pending now, finally libsas topo
>>    may different the hardware.
>> 2. receive a phy down sas event, libsas call sas_deform_port to remove
>>    devices, it would first delete the sas port, then put a destruction
>>    discovery event in a new work, and queue it at the tail of workqueue,
>>    once the sas port be deleted, its children device will be deleted too,
>>    when the destruction work start, it will found the target device has
>>    been removed, and report a sysfs warnning.
>> 3. since a hotplug process will be devided into several works, if a phy up
>>    sas event insert into phydown works, like
>>    destruction work  ---> PORTE_BYTES_DMAED (sas_form_port) ---->PHYE_LOSS_OF_SIGNAL
>>    the hot remove flow would broken by PORTE_BYTES_DMAED event, it's not
>>    we expected, and issues would occur.
>>
>> The first patch fix the sas events lost, and the second one introudce wait-complete
>> to fix the hotplug order issues.
>>
> 
> I quickly tested this for basic hotplug.
> 
> Before:
> root@(none)$ echo 0 > ./phy-0:6/sas_phy/phy-0:6/enable
> root@(none)$ echo 0 > ./phy-0:5/sas_phy/phy-0:5/enable
> root@(none)$ echo 0 > ./phy-0:4/sas_phy/phy-0:4/enable
> root@(none)$ echo 0 > ./phy-0:3/sas_phy/phy-0:3/enable
> root@(none)$ echo 0 > ./phy-0:3/sas_phy/phy-0:2/enable
> root@(none)$ echo 0 > ./phy-0:2/sas_phy/phy-0:2/enable
> root@(none)$ echo 0 > ./phy-0:1/sas_phy/phy-0:1/enable
> root@(none)$ echo 0 > ./phy-0:0/sas_phy/phy-0:0/enable
> root@(none)$ echo 0 > ./phy-0:7/sas_phy/phy-0:7/enable
> root@(none)$ [  102.570694] sysfs group 'power' not found for kobject '0:0:7:0'
> [  102.577250] ------------[ cut here ]------------
> [  102.581861] WARNING: CPU: 3 PID: 1740 at fs/sysfs/group.c:237 sysfs_remove_group+0x8c/0x94
> [  102.590110] Modules linked in:
> [  102.593154] CPU: 3 PID: 1740 Comm: kworker/u128:2 Not tainted 4.12.0-rc1-00032-g3ab81fc #1907
> [  102.601664] Hardware name: Huawei Taishan 2280 /D05, BIOS Hisilicon D05 UEFI Nemo 1.7 RC3 06/23/2017
> [  102.610784] Workqueue: scsi_wq_0 sas_destruct_devices
> [  102.615822] task: ffff8017d4793400 task.stack: ffff8017b7e70000
> [  102.621728] PC is at sysfs_remove_group+0x8c/0x94
> [  102.626419] LR is at sysfs_remove_group+0x8c/0x94
> [  102.631109] pc : [<ffff000008267c44>] lr : [<ffff000008267c44>] pstate: 60000045
> [  102.638490] sp : ffff8017b7e73b80
> [  102.641791] x29: ffff8017b7e73b80 x28: ffff8017db010800
> [  102.647091] x27: ffff000008e27000 x26: ffff8017d43e6600
> [  102.652390] x25: ffff8017b8280000 x24: 0000000000000003
> [  102.657689] x23: ffff8017b78864b0 x22: ffff8017b784c988
> [  102.662988] x21: ffff8017b7886410 x20: ffff000008ee9dd0
> [  102.668288] x19: 0000000000000000 x18: ffff000008a1b678
> [  102.673587] x17: 000000000000000e x16: 0000000000000007
> [  102.678886] x15: 0000000000000000 x14: 00000000000000a3
> [  102.684185] x13: 0000000000000033 x12: 0000000000000028
> [  102.689484] x11: ffff000008f3be58 x10: 0000000000000000
> [  102.694783] x9 : 000000000000043c x8 : 6f6b20726f662064
> [  102.700082] x7 : ffff000008e29e08 x6 : ffff8017fbe34c50
> [  102.705382] x5 : 0000000000000000 x4 : 0000000000000000
> [  102.710681] x3 : ffffffffffffffff x2 : ffff000008e427e0
> [  102.715980] x1 : 0000000000000000 x0 : 0000000000000033
> [  102.721279] ---[ end trace c216cc1451d5f7ec ]---
> [  102.725882] Call trace:
> [  102.728316] Exception stack(0xffff8017b7e739b0 to 0xffff8017b7e73ae0)
> [  102.734742] 39a0:                                   0000000000000000 0001000000000000
> [  102.742557] 39c0: ffff8017b7e73b80 ffff000008267c44 ffff000008bfa050 0000000000000000
> [  102.750372] 39e0: ffff8017b78864b0 0000000000000003 ffff8017b8280000 ffff8017d43e6600
> [  102.758188] 3a00: ffff000008e27000 ffff8017db010800 ffff8017d4793400 0000000000000000
> [  102.766003] 3a20: ffff8017b7e73b80 ffff8017b7e73b80 ffff8017b7e73b40 00000000ffffffc8
> [  102.773818] 3a40: ffff8017b7e73a70 ffff00000810c12c 0000000000000033 0000000000000000
> [  102.781633] 3a60: ffff000008e427e0 ffffffffffffffff 0000000000000000 0000000000000000
> [  102.789449] 3a80: ffff8017fbe34c50 ffff000008e29e08 6f6b20726f662064 000000000000043c
> [  102.797264] 3aa0: 0000000000000000 ffff000008f3be58 0000000000000028 0000000000000033
> [  102.805079] 3ac0: 00000000000000a3 0000000000000000 0000000000000007 000000000000000e
> [  102.812895] [<ffff000008267c44>] sysfs_remove_group+0x8c/0x94
> [  102.818628] [<ffff00000855b14c>] dpm_sysfs_remove+0x58/0x68
> [  102.824188] [<ffff00000854e0e8>] device_del+0xf8/0x2d0
> [  102.829312] [<ffff00000854e2d4>] device_unregister+0x14/0x2c
> [  102.834959] [<ffff00000837e6e0>] bsg_unregister_queue+0x60/0x98
> [  102.840866] [<ffff000008593cd4>] __scsi_remove_device+0xa0/0xbc
> 
> <snip>
> 
> [  151.331854] 3bc0: ffff0000081f21ac 0000ffff803370c0
> [  151.336718] [<ffff000008267c44>] sysfs_remove_group+0x8c/0x94
> [  151.342449] [<ffff00000855b14c>] dpm_sysfs_remove+0x58/0x68
> [  151.348008] [<ffff00000854e0e8>] device_del+0xf8/0x2d0
> [  151.353133] [<ffff000008597278>] sas_rphy_remove+0x54/0x80
> [  151.358604] [<ffff0000085972b8>] sas_rphy_delete+0x14/0x28
> [  151.364076] [<ffff00000859b304>] sas_destruct_devices+0x64/0x98
> [  151.369982] [<ffff0000080d8194>] process_one_work+0x12c/0x28c
> [  151.375714] [<ffff0000080d834c>] worker_thread+0x58/0x3b8
> [  151.381100] [<ffff0000080ddee4>] kthread+0x100/0x12c
> [  151.386050] [<ffff0000080836c0>] ret_from_fork+0x10/0x50
> [  151.391360] hisi_sas_v2_hw HISI0162:01: found dev[0:2] is gone
> 
> root@(none)$
> 
> So the console locks for ~50 seconds with WARN garbage.
> 
> After:
> ...
> root@(none)$ echo 0 > ./phy-0:7/sas_phy/phy-0:7/enable
> root@(none)$ [  446.193336] hisi_sas_v2_hw HISI0162:01: found dev[8:1] is gone
> [  446.249205] hisi_sas_v2_hw HISI0162:01: found dev[7:1] is gone
> [  446.325201] hisi_sas_v2_hw HISI0162:01: found dev[6:1] is gone
> [  446.373189] hisi_sas_v2_hw HISI0162:01: found dev[5:1] is gone
> [  446.421187] hisi_sas_v2_hw HISI0162:01: found dev[4:1] is gone
> [  446.457232] hisi_sas_v2_hw HISI0162:01: found dev[3:1] is gone
> [  446.477151] sd 0:0:1:0: [sdb] Synchronizing SCSI cache
> [  446.482373] sd 0:0:1:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=0x04 driverbyte=0x00
> [  446.491238] sd 0:0:1:0: [sdb] Stopping disk
> [  446.495419] sd 0:0:1:0: [sdb] Start/Stop Unit failed: Result: hostbyte=0x04 driverbyte=0x00
> [  446.525227] hisi_sas_v2_hw HISI0162:01: found dev[2:5] is gone
> [  446.569249] hisi_sas_v2_hw HISI0162:01: found dev[1:1] is gone
> [  446.576872] hisi_sas_v2_hw HISI0162:01: found dev[0:2] is gone
> 
> root@(none)$
> 
> So much nicer. BTW, /dev/sdb is a SATA disk, the rest are SAS.

Oh, I take a mistake ? The result you tested the hotplug which applied this patchset is fine ?

Thanks!
Yijing.


> 
> John
> 
>> v2->v3: some code improvements suggested by Johannes and John,
>>         split v2 patch 2 into several small pathes.
>> v1->v2: some code improvements suggested by John Garry
>>
>> Yijing Wang (7):
>>   libsas: Use static sas event pool to appease sas event lost
>>   libsas: remove unused port_gone_completion
>>   libsas: Use new workqueue to run sas event
>>   libsas: add sas event wait-complete support
>>   libsas: add a new workqueue to run probe/destruct discovery event
>>   libsas: add wait-complete support to sync discovery event
>>   libsas: release disco mutex during waiting in sas_ex_discover_end_dev
>>
>>  drivers/scsi/libsas/sas_discover.c |  58 +++++++---
>>  drivers/scsi/libsas/sas_event.c    | 212 ++++++++++++++++++++++++++++++++-----
>>  drivers/scsi/libsas/sas_expander.c |  22 +++-
>>  drivers/scsi/libsas/sas_init.c     |  21 ++--
>>  drivers/scsi/libsas/sas_internal.h |  64 +++++++++++
>>  drivers/scsi/libsas/sas_phy.c      |  48 +++------
>>  drivers/scsi/libsas/sas_port.c     |  22 ++--
>>  include/scsi/libsas.h              |  27 +++--
>>  8 files changed, 373 insertions(+), 101 deletions(-)
>>
> 
> 
> 
> .
>

[toc] | [prev] | [next] | [standalone]

#1686355

From	John Garry <john.garry@huawei.com>
Date	2017-07-13 10:10 +0200
Message-ID	<u2Hma-34F-19@gated-at.bofh.it>
In reply to	#1686170

On 13/07/2017 02:37, wangyijing wrote:
>> > So much nicer. BTW, /dev/sdb is a SATA disk, the rest are SAS.
> Oh, I take a mistake ? The result you tested the hotplug which applied this patchset is fine ?
>
> Thanks!
> Yijing.

Well basic hotplug is fine, as below. I did not do any robust testing.

root@(none)$ echo 0 > ./phy-0:7/sas_phy/phy-0:7/enable
root@(none)$ [  180.147676] hisi_sas_v2_hw HISI0162:01: found dev[8:1] 
is gone
[  180.216558] hisi_sas_v2_hw HISI0162:01: found dev[7:1] is gone
[  180.280548] hisi_sas_v2_hw HISI0162:01: found dev[6:1] is gone
[  180.352556] hisi_sas_v2_hw HISI0162:01: found dev[5:1] is gone
[  180.432495] hisi_sas_v2_hw HISI0162:01: found dev[4:1] is gone
[  180.508492] hisi_sas_v2_hw HISI0162:01: found dev[3:1] is gone
[  180.527577] sd 0:0:1:0: [sdb] Synchronizing SCSI cache
[  180.532728] sd 0:0:1:0: [sdb] Synchronize Cache(10) failed: Result: 
hostbyte=0x04 driverbyte=0x00
[  180.541591] sd 0:0:1:0: [sdb] Stopping disk
[  180.545767] sd 0:0:1:0: [sdb] Start/Stop Unit failed: Result: 
hostbyte=0x04 driverbyte=0x00
[  180.612491] hisi_sas_v2_hw HISI0162:01: found dev[2:5] is gone
[  180.696452] hisi_sas_v2_hw HISI0162:01: found dev[1:1] is gone
[  180.703221] hisi_sas_v2_hw HISI0162:01: found dev[0:2] is gone

root@(none)$ echo 1 > ./phy-0:7/sas_phy/phy-0:7/enable
root@(none)$ [  185.937831] hisi_sas_v2_hw HISI0162:01: phyup: phy7 
link_rate=11
[  185.996575] scsi 0:0:8:0: Direct-Access     SanDisk  LT0200MO 
P404 PQ: 0 ANSI: 6
[  187.059642] ata2.00: ATA-8: HGST HUS724040ALA640, MFAOA8B0, max UDMA/133
[  187.066341] ata2.00: 7814037168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[  187.073278] ata2.00: ATA Identify Device Log not supported
[  187.078755] ata2.00: Security Log not supported
[  187.085239] ata2.00: ATA Identify Device Log not supported
[  187.090715] ata2.00: Security Log not supported
[  187.095236] ata2.00: configured for UDMA/133
[  187.136917] scsi 0:0:9:0: Direct-Access     ATA      HGST HUS724040AL 
A8B0 PQ: 0 ANSI: 5
[  187.187612] sd 0:0:9:0: [sdb] 7814037168 512-byte logical blocks: 
(4.00 TB/3.64 TiB)
[  187.195365] sd 0:0:9:0: [sdb] Write Protect is off
[  187.200161] sd 0:0:9:0: [sdb] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[  187.223844] sd 0:0:9:0: [sdb] Attached SCSI disk
[  187.225498] scsi 0:0:10:0: Direct-Access     SanDisk  LT0200MO 
  P404 PQ: 0 ANSI: 6
[  187.243864] sd 0:0:8:0: [sda] 390721968 512-byte logical blocks: (200 
GB/186 GiB)
[  187.285879] sd 0:0:8:0: [sda] Write Protect is off
[  187.367898] sd 0:0:8:0: [sda] Write cache: disabled, read cache: 
disabled, supports DPO and FUA
[  187.524043] scsi 0:0:11:0: Direct-Access     SanDisk  LT0200MO 
  P404 PQ: 0 ANSI: 6
[  187.701505] sd 0:0:10:0: [sdc] 390721968 512-byte logical blocks: 
(200 GB/186 GiB)
[  187.743547] sd 0:0:10:0: [sdc] Write Protect is off
[  187.822546] scsi 0:0:12:0: Direct-Access     SanDisk  LT0200MO 
  P404 PQ: 0 ANSI: 6
[  187.825531] sd 0:0:10:0: [sdc] Write cache: disabled, read cache: 
disabled, supports DPO and FUA
[  188.000167] sd 0:0:11:0: [sdd] 390721968 512-byte logical blocks: 
(200 GB/186 GiB)
[  188.042205] sd 0:0:11:0: [sdd] Write Protect is off
[  188.121527] scsi 0:0:13:0: Direct-Access     SanDisk  LT0200MO 
  P404 PQ: 0 ANSI: 6
[  188.124274] sd 0:0:11:0: [sdd] Write cache: disabled, read cache: 
disabled, supports DPO and FUA
[  188.298942] sd 0:0:12:0: [sde] 390721968 512-byte logical blocks: 
(200 GB/186 GiB)
[  188.340960] sd 0:0:12:0: [sde] Write Protect is off
[  188.420023] scsi 0:0:14:0: Direct-Access     SanDisk  LT0200MO 
  P404 PQ: 0 ANSI: 6
[  188.422969] sd 0:0:12:0: [sde] Write cache: disabled, read cache: 
disabled, supports DPO and FUA
[  188.597501] sd 0:0:13:0: [sdf] 390721968 512-byte logical blocks: 
(200 GB/186 GiB)
[  188.605069] sd 0:0:8:0: [sda] Attached SCSI disk
[  188.639520] sd 0:0:13:0: [sdf] Write Protect is off
[  188.682445] scsi 0:0:15:0: Enclosure         12G SAS  Expander 
  RevB PQ: 0 ANSI: 6
[  188.721540] sd 0:0:13:0: [sdf] Write cache: disabled, read cache: 
disabled, supports DPO and FUA
[  188.896399] sd 0:0:14:0: [sdg] 390721968 512-byte logical blocks: 
(200 GB/186 GiB)
[  188.938445] sd 0:0:14:0: [sdg] Write Protect is off
[  189.020444] sd 0:0:14:0: [sdg] Write cache: disabled, read cache: 
disabled, supports DPO and FUA
[  189.060608] sd 0:0:10:0: [sdc] Attached SCSI disk
[  189.359073] sd 0:0:11:0: [sdd] Attached SCSI disk
[  189.657643] sd 0:0:12:0: [sde] Attached SCSI disk
[  189.956585] sd 0:0:13:0: [sdf] Attached SCSI disk
[  190.255148] sd 0:0:14:0: [sdg] Attached SCSI disk

root@(none)$ echo 0 > ./phy-0:7/sas_phy/phy-0:7/enable
root@(none)$ [  192.895718] hisi_sas_v2_hw HISI0162:01: found dev[8:1] 
is gone
[  192.964671] hisi_sas_v2_hw HISI0162:01: found dev[7:1] is gone
[  193.032744] hisi_sas_v2_hw HISI0162:01: found dev[6:1] is gone
[  193.096755] hisi_sas_v2_hw HISI0162:01: found dev[5:1] is gone
[  193.157072] hisi_sas_v2_hw HISI0162:01: found dev[4:1] is gone
[  193.221062] hisi_sas_v2_hw HISI0162:01: found dev[3:1] is gone
[  193.247684] sd 0:0:9:0: [sdb] Synchronizing SCSI cache
[  193.252834] sd 0:0:9:0: [sdb] Synchronize Cache(10) failed: Result: 
hostbyte=0x04 driverbyte=0x00
[  193.261701] sd 0:0:9:0: [sdb] Stopping disk
[  193.265879] sd 0:0:9:0: [sdb] Start/Stop Unit failed: Result: 
hostbyte=0x04 driverbyte=0x00
[  193.325165] hisi_sas_v2_hw HISI0162:01: found dev[2:5] is gone
[  193.381094] hisi_sas_v2_hw HISI0162:01: found dev[1:1] is gone
[  193.388719] hisi_sas_v2_hw HISI0162:01: found dev[0:2] is gone

root@(none)$ echo 1 > ./phy-0:7/sas_phy/phy-0:7/enable
root@(none)$ [  196.221879] hisi_sas_v2_hw HISI0162:01: phyup: phy7 
link_rate=11
[  196.281178] scsi 0:0:16:0: Direct-Access     SanDisk  LT0200MO 
  P404 PQ: 0 ANSI: 6
[  196.973390] ata3.00: ATA-8: HGST HUS724040ALA640, MFAOA8B0, max UDMA/133
[  196.980088] ata3.00: 7814037168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[  196.987022] ata3.00: ATA Identify Device Log not supported
[  196.992499] ata3.00: Security Log not supported
[  196.998953] ata3.00: ATA Identify Device Log not supported
[  197.004434] ata3.00: Security Log not supported
[  197.008954] ata3.00: configured for UDMA/133
[  197.050428] scsi 0:0:17:0: Direct-Access     ATA      HGST 
HUS724040AL A8B0 PQ: 0 ANSI: 5
[  197.091593] sd 0:0:17:0: [sdb] 7814037168 512-byte logical blocks: 
(4.00 TB/3.64 TiB)
[  197.099441] sd 0:0:17:0: [sdb] Write Protect is off
[  197.104326] sd 0:0:17:0: [sdb] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[  197.122982] sd 0:0:17:0: [sdb] Attached SCSI disk
[  197.129367] scsi 0:0:18:0: Direct-Access     SanDisk  LT0200MO 
  P404 PQ: 0 ANSI: 6
[  197.157605] sd 0:0:16:0: [sda] 390721968 512-byte logical blocks: 
(200 GB/186 GiB)
[  197.199622] sd 0:0:16:0: [sda] Write Protect is off
[  197.281604] sd 0:0:16:0: [sda] Write cache: disabled, read cache: 
disabled, supports DPO and FUA
[  197.427727] scsi 0:0:19:0: Direct-Access     SanDisk  LT0200MO 
  P404 PQ: 0 ANSI: 6
[  197.605362] sd 0:0:18:0: [sdc] 390721968 512-byte logical blocks: 
(200 GB/186 GiB)
[  197.647409] sd 0:0:18:0: [sdc] Write Protect is off
[  197.726403] scsi 0:0:20:0: Direct-Access     SanDisk  LT0200MO 
  P404 PQ: 0 ANSI: 6
[  197.729389] sd 0:0:18:0: [sdc] Write cache: disabled, read cache: 
disabled, supports DPO and FUA
[  197.903664] sd 0:0:19:0: [sdd] 390721968 512-byte logical blocks: 
(200 GB/186 GiB)
[  197.945699] sd 0:0:19:0: [sdd] Write Protect is off
[  198.024789] scsi 0:0:21:0: Direct-Access     SanDisk  LT0200MO 
  P404 PQ: 0 ANSI: 6
[  198.027685] sd 0:0:19:0: [sdd] Write cache: disabled, read cache: 
disabled, supports DPO and FUA
[  198.202410] sd 0:0:20:0: [sde] 390721968 512-byte logical blocks: 
(200 GB/186 GiB)
[  198.244458] sd 0:0:20:0: [sde] Write Protect is off
[  198.323137] scsi 0:0:22:0: Direct-Access     SanDisk  LT0200MO 
  P404 PQ: 0 ANSI: 6
[  198.326491] sd 0:0:20:0: [sde] Write cache: disabled, read cache: 
disabled, supports DPO and FUA
[  198.500754] sd 0:0:21:0: [sdf] 390721968 512-byte logical blocks: 
(200 GB/186 GiB)
[  198.516373] sd 0:0:16:0: [sda] Attached SCSI disk
[  198.542773] sd 0:0:21:0: [sdf] Write Protect is off
[  198.585184] scsi 0:0:23:0: Enclosure         12G SAS  Expander 
  RevB PQ: 0 ANSI: 6
[  198.624911] sd 0:0:21:0: [sdf] Write cache: disabled, read cache: 
disabled, supports DPO and FUA
[  198.799446] sd 0:0:22:0: [sdg] 390721968 512-byte logical blocks: 
(200 GB/186 GiB)
[  198.841484] sd 0:0:22:0: [sdg] Write Protect is off
[  198.923508] sd 0:0:22:0: [sdg] Write cache: disabled, read cache: 
disabled, supports DPO and FUA
[  198.964420] sd 0:0:18:0: [sdc] Attached SCSI disk
[  199.262903] sd 0:0:19:0: [sdd] Attached SCSI disk
[  199.561175] sd 0:0:20:0: [sde] Attached SCSI disk
[  199.859533] sd 0:0:21:0: [sdf] Attached SCSI disk

root@(none)$ [  200.158121] sd 0:0:22:0: [sdg] Attached SCSI disk

root@(none)$


>
>
>> >
>> > John

[toc] | [prev] | [next] | [standalone]

#1686385

From	wangyijing <wangyijing@huawei.com>
Date	2017-07-13 10:50 +0200
Message-ID	<u2HYR-3iC-3@gated-at.bofh.it>
In reply to	#1686355


在 2017/7/13 16:08, John Garry 写道:
> On 13/07/2017 02:37, wangyijing wrote:
>>> > So much nicer. BTW, /dev/sdb is a SATA disk, the rest are SAS.
>> Oh, I take a mistake ? The result you tested the hotplug which applied this patchset is fine ?
>>
>> Thanks!
>> Yijing.
> 
> Well basic hotplug is fine, as below. I did not do any robust testing.
> 

OK， thanks，I tested with and without fio running, the results are both fine.

Thanks!
Yijing.

> root@(none)$ echo 0 > ./phy-0:7/sas_phy/phy-0:7/enable
> root@(none)$ [  180.147676] hisi_sas_v2_hw HISI0162:01: found dev[8:1] is gone
> [  180.216558] hisi_sas_v2_hw HISI0162:01: found dev[7:1] is gone
> [  180.280548] hisi_sas_v2_hw HISI0162:01: found dev[6:1] is gone
> [  180.352556] hisi_sas_v2_hw HISI0162:01: found dev[5:1] is gone
> [  180.432495] hisi_sas_v2_hw HISI0162:01: found dev[4:1] is gone
> [  180.508492] hisi_sas_v2_hw HISI0162:01: found dev[3:1] is gone
> [  180.527577] sd 0:0:1:0: [sdb] Synchronizing SCSI cache
> [  180.532728] sd 0:0:1:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=0x04 driverbyte=0x00
> [  180.541591] sd 0:0:1:0: [sdb] Stopping disk
> [  180.545767] sd 0:0:1:0: [sdb] Start/Stop Unit failed: Result: hostbyte=0x04 driverbyte=0x00
> [  180.612491] hisi_sas_v2_hw HISI0162:01: found dev[2:5] is gone
> [  180.696452] hisi_sas_v2_hw HISI0162:01: found dev[1:1] is gone
> [  180.703221] hisi_sas_v2_hw HISI0162:01: found dev[0:2] is gone
> 
> root@(none)$ echo 1 > ./phy-0:7/sas_phy/phy-0:7/enable
> root@(none)$ [  185.937831] hisi_sas_v2_hw HISI0162:01: phyup: phy7 link_rate=11
> [  185.996575] scsi 0:0:8:0: Direct-Access     SanDisk  LT0200MO P404 PQ: 0 ANSI: 6
> [  187.059642] ata2.00: ATA-8: HGST HUS724040ALA640, MFAOA8B0, max UDMA/133
> [  187.066341] ata2.00: 7814037168 sectors, multi 0: LBA48 NCQ (depth 31/32)
> [  187.073278] ata2.00: ATA Identify Device Log not supported
> [  187.078755] ata2.00: Security Log not supported
> [  187.085239] ata2.00: ATA Identify Device Log not supported
> [  187.090715] ata2.00: Security Log not supported
> [  187.095236] ata2.00: configured for UDMA/133
> [  187.136917] scsi 0:0:9:0: Direct-Access     ATA      HGST HUS724040AL A8B0 PQ: 0 ANSI: 5
> [  187.187612] sd 0:0:9:0: [sdb] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
> [  187.195365] sd 0:0:9:0: [sdb] Write Protect is off
> [  187.200161] sd 0:0:9:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> [  187.223844] sd 0:0:9:0: [sdb] Attached SCSI disk
> [  187.225498] scsi 0:0:10:0: Direct-Access     SanDisk  LT0200MO  P404 PQ: 0 ANSI: 6
> [  187.243864] sd 0:0:8:0: [sda] 390721968 512-byte logical blocks: (200 GB/186 GiB)
> [  187.285879] sd 0:0:8:0: [sda] Write Protect is off
> [  187.367898] sd 0:0:8:0: [sda] Write cache: disabled, read cache: disabled, supports DPO and FUA
> [  187.524043] scsi 0:0:11:0: Direct-Access     SanDisk  LT0200MO  P404 PQ: 0 ANSI: 6
> [  187.701505] sd 0:0:10:0: [sdc] 390721968 512-byte logical blocks: (200 GB/186 GiB)
> [  187.743547] sd 0:0:10:0: [sdc] Write Protect is off
> [  187.822546] scsi 0:0:12:0: Direct-Access     SanDisk  LT0200MO  P404 PQ: 0 ANSI: 6
> [  187.825531] sd 0:0:10:0: [sdc] Write cache: disabled, read cache: disabled, supports DPO and FUA
> [  188.000167] sd 0:0:11:0: [sdd] 390721968 512-byte logical blocks: (200 GB/186 GiB)
> [  188.042205] sd 0:0:11:0: [sdd] Write Protect is off
> [  188.121527] scsi 0:0:13:0: Direct-Access     SanDisk  LT0200MO  P404 PQ: 0 ANSI: 6
> [  188.124274] sd 0:0:11:0: [sdd] Write cache: disabled, read cache: disabled, supports DPO and FUA
> [  188.298942] sd 0:0:12:0: [sde] 390721968 512-byte logical blocks: (200 GB/186 GiB)
> [  188.340960] sd 0:0:12:0: [sde] Write Protect is off
> [  188.420023] scsi 0:0:14:0: Direct-Access     SanDisk  LT0200MO  P404 PQ: 0 ANSI: 6
> [  188.422969] sd 0:0:12:0: [sde] Write cache: disabled, read cache: disabled, supports DPO and FUA
> [  188.597501] sd 0:0:13:0: [sdf] 390721968 512-byte logical blocks: (200 GB/186 GiB)
> [  188.605069] sd 0:0:8:0: [sda] Attached SCSI disk
> [  188.639520] sd 0:0:13:0: [sdf] Write Protect is off
> [  188.682445] scsi 0:0:15:0: Enclosure         12G SAS  Expander  RevB PQ: 0 ANSI: 6
> [  188.721540] sd 0:0:13:0: [sdf] Write cache: disabled, read cache: disabled, supports DPO and FUA
> [  188.896399] sd 0:0:14:0: [sdg] 390721968 512-byte logical blocks: (200 GB/186 GiB)
> [  188.938445] sd 0:0:14:0: [sdg] Write Protect is off
> [  189.020444] sd 0:0:14:0: [sdg] Write cache: disabled, read cache: disabled, supports DPO and FUA
> [  189.060608] sd 0:0:10:0: [sdc] Attached SCSI disk
> [  189.359073] sd 0:0:11:0: [sdd] Attached SCSI disk
> [  189.657643] sd 0:0:12:0: [sde] Attached SCSI disk
> [  189.956585] sd 0:0:13:0: [sdf] Attached SCSI disk
> [  190.255148] sd 0:0:14:0: [sdg] Attached SCSI disk
> 
> root@(none)$ echo 0 > ./phy-0:7/sas_phy/phy-0:7/enable
> root@(none)$ [  192.895718] hisi_sas_v2_hw HISI0162:01: found dev[8:1] is gone
> [  192.964671] hisi_sas_v2_hw HISI0162:01: found dev[7:1] is gone
> [  193.032744] hisi_sas_v2_hw HISI0162:01: found dev[6:1] is gone
> [  193.096755] hisi_sas_v2_hw HISI0162:01: found dev[5:1] is gone
> [  193.157072] hisi_sas_v2_hw HISI0162:01: found dev[4:1] is gone
> [  193.221062] hisi_sas_v2_hw HISI0162:01: found dev[3:1] is gone
> [  193.247684] sd 0:0:9:0: [sdb] Synchronizing SCSI cache
> [  193.252834] sd 0:0:9:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=0x04 driverbyte=0x00
> [  193.261701] sd 0:0:9:0: [sdb] Stopping disk
> [  193.265879] sd 0:0:9:0: [sdb] Start/Stop Unit failed: Result: hostbyte=0x04 driverbyte=0x00
> [  193.325165] hisi_sas_v2_hw HISI0162:01: found dev[2:5] is gone
> [  193.381094] hisi_sas_v2_hw HISI0162:01: found dev[1:1] is gone
> [  193.388719] hisi_sas_v2_hw HISI0162:01: found dev[0:2] is gone
> 
> root@(none)$ echo 1 > ./phy-0:7/sas_phy/phy-0:7/enable
> root@(none)$ [  196.221879] hisi_sas_v2_hw HISI0162:01: phyup: phy7 link_rate=11
> [  196.281178] scsi 0:0:16:0: Direct-Access     SanDisk  LT0200MO  P404 PQ: 0 ANSI: 6
> [  196.973390] ata3.00: ATA-8: HGST HUS724040ALA640, MFAOA8B0, max UDMA/133
> [  196.980088] ata3.00: 7814037168 sectors, multi 0: LBA48 NCQ (depth 31/32)
> [  196.987022] ata3.00: ATA Identify Device Log not supported
> [  196.992499] ata3.00: Security Log not supported
> [  196.998953] ata3.00: ATA Identify Device Log not supported
> [  197.004434] ata3.00: Security Log not supported
> [  197.008954] ata3.00: configured for UDMA/133
> [  197.050428] scsi 0:0:17:0: Direct-Access     ATA      HGST HUS724040AL A8B0 PQ: 0 ANSI: 5
> [  197.091593] sd 0:0:17:0: [sdb] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
> [  197.099441] sd 0:0:17:0: [sdb] Write Protect is off
> [  197.104326] sd 0:0:17:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> [  197.122982] sd 0:0:17:0: [sdb] Attached SCSI disk
> [  197.129367] scsi 0:0:18:0: Direct-Access     SanDisk  LT0200MO  P404 PQ: 0 ANSI: 6
> [  197.157605] sd 0:0:16:0: [sda] 390721968 512-byte logical blocks: (200 GB/186 GiB)
> [  197.199622] sd 0:0:16:0: [sda] Write Protect is off
> [  197.281604] sd 0:0:16:0: [sda] Write cache: disabled, read cache: disabled, supports DPO and FUA
> [  197.427727] scsi 0:0:19:0: Direct-Access     SanDisk  LT0200MO  P404 PQ: 0 ANSI: 6
> [  197.605362] sd 0:0:18:0: [sdc] 390721968 512-byte logical blocks: (200 GB/186 GiB)
> [  197.647409] sd 0:0:18:0: [sdc] Write Protect is off
> [  197.726403] scsi 0:0:20:0: Direct-Access     SanDisk  LT0200MO  P404 PQ: 0 ANSI: 6
> [  197.729389] sd 0:0:18:0: [sdc] Write cache: disabled, read cache: disabled, supports DPO and FUA
> [  197.903664] sd 0:0:19:0: [sdd] 390721968 512-byte logical blocks: (200 GB/186 GiB)
> [  197.945699] sd 0:0:19:0: [sdd] Write Protect is off
> [  198.024789] scsi 0:0:21:0: Direct-Access     SanDisk  LT0200MO  P404 PQ: 0 ANSI: 6
> [  198.027685] sd 0:0:19:0: [sdd] Write cache: disabled, read cache: disabled, supports DPO and FUA
> [  198.202410] sd 0:0:20:0: [sde] 390721968 512-byte logical blocks: (200 GB/186 GiB)
> [  198.244458] sd 0:0:20:0: [sde] Write Protect is off
> [  198.323137] scsi 0:0:22:0: Direct-Access     SanDisk  LT0200MO  P404 PQ: 0 ANSI: 6
> [  198.326491] sd 0:0:20:0: [sde] Write cache: disabled, read cache: disabled, supports DPO and FUA
> [  198.500754] sd 0:0:21:0: [sdf] 390721968 512-byte logical blocks: (200 GB/186 GiB)
> [  198.516373] sd 0:0:16:0: [sda] Attached SCSI disk
> [  198.542773] sd 0:0:21:0: [sdf] Write Protect is off
> [  198.585184] scsi 0:0:23:0: Enclosure         12G SAS  Expander  RevB PQ: 0 ANSI: 6
> [  198.624911] sd 0:0:21:0: [sdf] Write cache: disabled, read cache: disabled, supports DPO and FUA
> [  198.799446] sd 0:0:22:0: [sdg] 390721968 512-byte logical blocks: (200 GB/186 GiB)
> [  198.841484] sd 0:0:22:0: [sdg] Write Protect is off
> [  198.923508] sd 0:0:22:0: [sdg] Write cache: disabled, read cache: disabled, supports DPO and FUA
> [  198.964420] sd 0:0:18:0: [sdc] Attached SCSI disk
> [  199.262903] sd 0:0:19:0: [sdd] Attached SCSI disk
> [  199.561175] sd 0:0:20:0: [sde] Attached SCSI disk
> [  199.859533] sd 0:0:21:0: [sdf] Attached SCSI disk
> 
> root@(none)$ [  200.158121] sd 0:0:22:0: [sdg] Attached SCSI disk
> 
> root@(none)$
> 
> 
>>
>>
>>> >
>>> > John
> 
> 
> 
> .
>

[toc] | [prev] | [next] | [standalone]

#1687150

From	wangyijing <wangyijing@huawei.com>
Date	2017-07-14 10:30 +0200
Message-ID	<u3494-LP-17@gated-at.bofh.it>
In reply to	#1684001

Hi, I'm sorry to say that I have to stop the libsas hotplug improvement work, I will resign from
Huawei, so I have no time and hardware to continue to work at this issue. John is very familiar with
this work, and provide a lot of good suggestions. So if John like, I am glad he could join to work
at this issues, And my colleague Jason Yan could also provide helps.


Thanks!
Yijing.


在 2017/7/10 15:06, Yijing Wang 写道:
> This patchset is based Johannes's patch
> "scsi: sas: scsi_queue_work can fail, so make callers aware"
> 
> Now the libsas hotplug has some issues, Dan Williams report
> a similar bug here before
> https://www.mail-archive.com/linux-scsi@vger.kernel.org/msg39187.html
> 
> The issues we have found
> 1. if LLDD burst reports lots of phy-up/phy-down sas events, some events
>    may lost because a same sas events is pending now, finally libsas topo
>    may different the hardware.
> 2. receive a phy down sas event, libsas call sas_deform_port to remove
>    devices, it would first delete the sas port, then put a destruction
>    discovery event in a new work, and queue it at the tail of workqueue,
>    once the sas port be deleted, its children device will be deleted too,
>    when the destruction work start, it will found the target device has
>    been removed, and report a sysfs warnning.
> 3. since a hotplug process will be devided into several works, if a phy up
>    sas event insert into phydown works, like
>    destruction work  ---> PORTE_BYTES_DMAED (sas_form_port) ---->PHYE_LOSS_OF_SIGNAL
>    the hot remove flow would broken by PORTE_BYTES_DMAED event, it's not
>    we expected, and issues would occur.
> 
> The first patch fix the sas events lost, and the second one introudce wait-complete
> to fix the hotplug order issues.
> 
> v2->v3: some code improvements suggested by Johannes and John,
> 	    split v2 patch 2 into several small pathes.
> v1->v2: some code improvements suggested by John Garry
> 
> Yijing Wang (7):
>   libsas: Use static sas event pool to appease sas event lost
>   libsas: remove unused port_gone_completion
>   libsas: Use new workqueue to run sas event
>   libsas: add sas event wait-complete support
>   libsas: add a new workqueue to run probe/destruct discovery event
>   libsas: add wait-complete support to sync discovery event
>   libsas: release disco mutex during waiting in sas_ex_discover_end_dev
> 
>  drivers/scsi/libsas/sas_discover.c |  58 +++++++---
>  drivers/scsi/libsas/sas_event.c    | 212 ++++++++++++++++++++++++++++++++-----
>  drivers/scsi/libsas/sas_expander.c |  22 +++-
>  drivers/scsi/libsas/sas_init.c     |  21 ++--
>  drivers/scsi/libsas/sas_internal.h |  64 +++++++++++
>  drivers/scsi/libsas/sas_phy.c      |  48 +++------
>  drivers/scsi/libsas/sas_port.c     |  22 ++--
>  include/scsi/libsas.h              |  27 +++--
>  8 files changed, 373 insertions(+), 101 deletions(-)
>

[toc] | [prev] | [standalone]

csiph-web

[PATCH v3 0/7] Enhance libsas hotplug feature

Contents

#1684001 — [PATCH v3 0/7] Enhance libsas hotplug feature

#1684002 — [PATCH v3 4/7] libsas: add sas event wait-complete support

#1687121 — Re: [PATCH v3 4/7] libsas: add sas event wait-complete support

#1687129 — Re: [PATCH v3 4/7] libsas: add sas event wait-complete support

#1687162 — Re: [PATCH v3 4/7] libsas: add sas event wait-complete support

#1684003 — [PATCH v3 6/7] libsas: add wait-complete support to sync discovery event

#1685795 — Re: [PATCH v3 6/7] libsas: add wait-complete support to sync discovery event

#1686190 — Re: [PATCH v3 6/7] libsas: add wait-complete support to sync discovery event

#1687116 — Re: [PATCH v3 6/7] libsas: add wait-complete support to sync discovery event

#1684004 — [PATCH v3 3/7] libsas: Use new workqueue to run sas event

#1687109 — Re: [PATCH v3 3/7] libsas: Use new workqueue to run sas event

#1685674

#1685724

#1686167

#1686170

#1686355

#1686385

#1687150