Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1723604 > unrolled thread

Re: [PATCH kernel] PCI: Disable IOV before pcibios_sriov_disable()

Started byBjorn Helgaas <helgaas@kernel.org>
First post2017-08-30 21:10 +0200
Last post2017-08-31 19:10 +0200
Articles 3 — 2 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH kernel] PCI: Disable IOV before pcibios_sriov_disable() Bjorn Helgaas <helgaas@kernel.org> - 2017-08-30 21:10 +0200
    Re: [PATCH kernel] PCI: Disable IOV before pcibios_sriov_disable() Alexey Kardashevskiy <aik@ozlabs.ru> - 2017-08-31 05:10 +0200
    Re: [PATCH kernel] PCI: Disable IOV before pcibios_sriov_disable() Bjorn Helgaas <helgaas@kernel.org> - 2017-08-31 19:10 +0200

#1723604 — Re: [PATCH kernel] PCI: Disable IOV before pcibios_sriov_disable()

FromBjorn Helgaas <helgaas@kernel.org>
Date2017-08-30 21:10 +0200
SubjectRe: [PATCH kernel] PCI: Disable IOV before pcibios_sriov_disable()
Message-ID<ukgxc-3Pz-19@gated-at.bofh.it>
On Fri, Aug 11, 2017 at 06:19:33PM +1000, Alexey Kardashevskiy wrote:
> From: Gavin Shan <gwshan@linux.vnet.ibm.com>
> 
> The PowerNV platform is the only user of pcibios_sriov_disable().
> The IOV BAR could be shifted by pci_iov_update_resource(). The
> warning message in the function is printed if the IOV capability
> is in enabled (PCI_SRIOV_CTRL_VFE && PCI_SRIOV_CTRL_MSE) state.
> 
> This is the backtrace of what is happening:
>    pci_disable_sriov
>    sriov_disable
>    pnv_pci_sriov_disable
>    pnv_pci_vf_resource_shift
>    pci_update_resource
>    pci_iov_update_resource
> 
> This fixes the issue by disabling IOV capability before calling
> pcibios_sriov_disable(). With it, the disabling path matches
> the enabling path: pcibios_sriov_enable() is called before the
> IOV capability is enabled.
> 
> Cc: shan.gavin@gmail.com
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Paul Mackerras <paulus@samba.org>
> Reported-by: Carol L Soto <clsoto@us.ibm.com>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> Tested-by: Carol L Soto <clsoto@us.ibm.com>
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> ---
> 
> This is repost. Since Gavin left the team, I am trying to push it out.
> The previos converstion is here: https://patchwork.ozlabs.org/patch/732653/

I gave up on the previous issue.  I think this patch makes sense as-is
at least as far as the fact that we can't update a struct resource
while the device is still consuming it.  I reworked the changelog to
emphasize that.

I assume the fact that pci_iov_update_resource() dropped the resource
update caused some user-visible issue later on, and I might mention
that, too, if I knew what it was.

Here's what I would consider putting on pci/virtualization (the diff
is unchanged from your post):


commit 08132e7759b3929bea0ccdf8afe81ebf05351389
Author: Gavin Shan <gwshan@linux.vnet.ibm.com>
Date:   Fri Aug 11 18:19:33 2017 +1000

    PCI: Disable VF decoding before updating resources in pcibios_sriov_disable()
    
    A struct resource represents the address space consumed by a device.  We
    should not modify that resource while the device is actively using the
    address space.  For VFs, pci_iov_update_resource() enforces this by
    printing a warning and doing nothing if the VFE (VF Enable) and MSE (VF
    Memory Space Enable) bits are set.
    
    Previously, both sriov_enable() and sriov_disable() called the
    pcibios_sriov_disable() arch hook, which may update the struct resource,
    while VFE and MSE were enabled.  This effectively dropped the resource
    update pcibios_sriov_disable() intended to do.
    
    Disable VF memory decoding before calling pcibios_sriov_disable().
    
    Reported-by: Carol L Soto <clsoto@us.ibm.com>
    Tested-by: Carol L Soto <clsoto@us.ibm.com>
    Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
    Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
    [bhelgaas: changelog]
    Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
    Cc: shan.gavin@gmail.com
    Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
    Cc: Paul Mackerras <paulus@samba.org>

diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index 120485d6f352..ac41c8be9200 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -331,7 +331,6 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
 	while (i--)
 		pci_iov_remove_virtfn(dev, i, 0);
 
-	pcibios_sriov_disable(dev);
 err_pcibios:
 	iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE);
 	pci_cfg_access_lock(dev);
@@ -339,6 +338,8 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
 	ssleep(1);
 	pci_cfg_access_unlock(dev);
 
+	pcibios_sriov_disable(dev);
+
 	if (iov->link != dev->devfn)
 		sysfs_remove_link(&dev->dev.kobj, "dep_link");
 
@@ -357,14 +358,14 @@ static void sriov_disable(struct pci_dev *dev)
 	for (i = 0; i < iov->num_VFs; i++)
 		pci_iov_remove_virtfn(dev, i, 0);
 
-	pcibios_sriov_disable(dev);
-
 	iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE);
 	pci_cfg_access_lock(dev);
 	pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
 	ssleep(1);
 	pci_cfg_access_unlock(dev);
 
+	pcibios_sriov_disable(dev);
+
 	if (iov->link != dev->devfn)
 		sysfs_remove_link(&dev->dev.kobj, "dep_link");

[toc] | [next] | [standalone]

#1723784

FromAlexey Kardashevskiy <aik@ozlabs.ru>
Date2017-08-31 05:10 +0200
Message-ID<uko1I-dV-19@gated-at.bofh.it>
In reply to#1723604
On 31/08/17 05:02, Bjorn Helgaas wrote:
> On Fri, Aug 11, 2017 at 06:19:33PM +1000, Alexey Kardashevskiy wrote:
>> From: Gavin Shan <gwshan@linux.vnet.ibm.com>
>>
>> The PowerNV platform is the only user of pcibios_sriov_disable().
>> The IOV BAR could be shifted by pci_iov_update_resource(). The
>> warning message in the function is printed if the IOV capability
>> is in enabled (PCI_SRIOV_CTRL_VFE && PCI_SRIOV_CTRL_MSE) state.
>>
>> This is the backtrace of what is happening:
>>    pci_disable_sriov
>>    sriov_disable
>>    pnv_pci_sriov_disable
>>    pnv_pci_vf_resource_shift
>>    pci_update_resource
>>    pci_iov_update_resource
>>
>> This fixes the issue by disabling IOV capability before calling
>> pcibios_sriov_disable(). With it, the disabling path matches
>> the enabling path: pcibios_sriov_enable() is called before the
>> IOV capability is enabled.
>>
>> Cc: shan.gavin@gmail.com
>> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
>> Cc: Paul Mackerras <paulus@samba.org>
>> Reported-by: Carol L Soto <clsoto@us.ibm.com>
>> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>> Tested-by: Carol L Soto <clsoto@us.ibm.com>
>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>> ---
>>
>> This is repost. Since Gavin left the team, I am trying to push it out.
>> The previos converstion is here: https://patchwork.ozlabs.org/patch/732653/
> 
> I gave up on the previous issue.  I think this patch makes sense as-is
> at least as far as the fact that we can't update a struct resource
> while the device is still consuming it.  I reworked the changelog to
> emphasize that.
> 
> I assume the fact that pci_iov_update_resource() dropped the resource
> update caused some user-visible issue later on, and I might mention
> that, too, if I knew what it was.

I could not identify any issue so far in my test setup - I recreated VFs
several times, run some traffic through them on one of mellanox'es so the
message+backtrace seems to be the only issue for now.


> Here's what I would consider putting on pci/virtualization (the diff
> is unchanged from your post):


This sounds good to me, thanks for updating the commit log.

I'll still try and finish my homework with updating that comment about the
hole in arch/powerpc/platforms/powernv/pci-ioda.c.

Cheers.

> 
> 
> commit 08132e7759b3929bea0ccdf8afe81ebf05351389
> Author: Gavin Shan <gwshan@linux.vnet.ibm.com>
> Date:   Fri Aug 11 18:19:33 2017 +1000
> 
>     PCI: Disable VF decoding before updating resources in pcibios_sriov_disable()
>     
>     A struct resource represents the address space consumed by a device.  We
>     should not modify that resource while the device is actively using the
>     address space.  For VFs, pci_iov_update_resource() enforces this by
>     printing a warning and doing nothing if the VFE (VF Enable) and MSE (VF
>     Memory Space Enable) bits are set.
>     
>     Previously, both sriov_enable() and sriov_disable() called the
>     pcibios_sriov_disable() arch hook, which may update the struct resource,
>     while VFE and MSE were enabled.  This effectively dropped the resource
>     update pcibios_sriov_disable() intended to do.
>     
>     Disable VF memory decoding before calling pcibios_sriov_disable().
>     
>     Reported-by: Carol L Soto <clsoto@us.ibm.com>
>     Tested-by: Carol L Soto <clsoto@us.ibm.com>
>     Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>     Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>     [bhelgaas: changelog]
>     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>     Cc: shan.gavin@gmail.com
>     Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
>     Cc: Paul Mackerras <paulus@samba.org>

> 
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index 120485d6f352..ac41c8be9200 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -331,7 +331,6 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
>  	while (i--)
>  		pci_iov_remove_virtfn(dev, i, 0);
>  
> -	pcibios_sriov_disable(dev);
>  err_pcibios:
>  	iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE);
>  	pci_cfg_access_lock(dev);
> @@ -339,6 +338,8 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
>  	ssleep(1);
>  	pci_cfg_access_unlock(dev);
>  
> +	pcibios_sriov_disable(dev);
> +
>  	if (iov->link != dev->devfn)
>  		sysfs_remove_link(&dev->dev.kobj, "dep_link");
>  
> @@ -357,14 +358,14 @@ static void sriov_disable(struct pci_dev *dev)
>  	for (i = 0; i < iov->num_VFs; i++)
>  		pci_iov_remove_virtfn(dev, i, 0);
>  
> -	pcibios_sriov_disable(dev);
> -
>  	iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE);
>  	pci_cfg_access_lock(dev);
>  	pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
>  	ssleep(1);
>  	pci_cfg_access_unlock(dev);
>  
> +	pcibios_sriov_disable(dev);
> +
>  	if (iov->link != dev->devfn)
>  		sysfs_remove_link(&dev->dev.kobj, "dep_link");
>  
> 


-- 
Alexey

[toc] | [prev] | [next] | [standalone]

#1724424

FromBjorn Helgaas <helgaas@kernel.org>
Date2017-08-31 19:10 +0200
Message-ID<ukB8C-8op-7@gated-at.bofh.it>
In reply to#1723604
On Wed, Aug 30, 2017 at 02:02:06PM -0500, Bjorn Helgaas wrote:
> On Fri, Aug 11, 2017 at 06:19:33PM +1000, Alexey Kardashevskiy wrote:
> > From: Gavin Shan <gwshan@linux.vnet.ibm.com>
> > 
> > The PowerNV platform is the only user of pcibios_sriov_disable().
> > The IOV BAR could be shifted by pci_iov_update_resource(). The
> > warning message in the function is printed if the IOV capability
> > is in enabled (PCI_SRIOV_CTRL_VFE && PCI_SRIOV_CTRL_MSE) state.
> > 
> > This is the backtrace of what is happening:
> >    pci_disable_sriov
> >    sriov_disable
> >    pnv_pci_sriov_disable
> >    pnv_pci_vf_resource_shift
> >    pci_update_resource
> >    pci_iov_update_resource
> > 
> > This fixes the issue by disabling IOV capability before calling
> > pcibios_sriov_disable(). With it, the disabling path matches
> > the enabling path: pcibios_sriov_enable() is called before the
> > IOV capability is enabled.
> > 
> > Cc: shan.gavin@gmail.com
> > Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> > Cc: Paul Mackerras <paulus@samba.org>
> > Reported-by: Carol L Soto <clsoto@us.ibm.com>
> > Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> > Tested-by: Carol L Soto <clsoto@us.ibm.com>
> > Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> > ---
> > 
> > This is repost. Since Gavin left the team, I am trying to push it out.
> > The previos converstion is here: https://patchwork.ozlabs.org/patch/732653/
> 
> I gave up on the previous issue.  I think this patch makes sense as-is
> at least as far as the fact that we can't update a struct resource
> while the device is still consuming it.  I reworked the changelog to
> emphasize that.
> 
> I assume the fact that pci_iov_update_resource() dropped the resource
> update caused some user-visible issue later on, and I might mention
> that, too, if I knew what it was.
> 
> Here's what I would consider putting on pci/virtualization (the diff
> is unchanged from your post):

I applied the patch below on pci/virtualization for v4.14.

> commit 08132e7759b3929bea0ccdf8afe81ebf05351389
> Author: Gavin Shan <gwshan@linux.vnet.ibm.com>
> Date:   Fri Aug 11 18:19:33 2017 +1000
> 
>     PCI: Disable VF decoding before updating resources in pcibios_sriov_disable()
>     
>     A struct resource represents the address space consumed by a device.  We
>     should not modify that resource while the device is actively using the
>     address space.  For VFs, pci_iov_update_resource() enforces this by
>     printing a warning and doing nothing if the VFE (VF Enable) and MSE (VF
>     Memory Space Enable) bits are set.
>     
>     Previously, both sriov_enable() and sriov_disable() called the
>     pcibios_sriov_disable() arch hook, which may update the struct resource,
>     while VFE and MSE were enabled.  This effectively dropped the resource
>     update pcibios_sriov_disable() intended to do.
>     
>     Disable VF memory decoding before calling pcibios_sriov_disable().
>     
>     Reported-by: Carol L Soto <clsoto@us.ibm.com>
>     Tested-by: Carol L Soto <clsoto@us.ibm.com>
>     Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>     Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>     [bhelgaas: changelog]
>     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>     Cc: shan.gavin@gmail.com
>     Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
>     Cc: Paul Mackerras <paulus@samba.org>
> 
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index 120485d6f352..ac41c8be9200 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -331,7 +331,6 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
>  	while (i--)
>  		pci_iov_remove_virtfn(dev, i, 0);
>  
> -	pcibios_sriov_disable(dev);
>  err_pcibios:
>  	iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE);
>  	pci_cfg_access_lock(dev);
> @@ -339,6 +338,8 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
>  	ssleep(1);
>  	pci_cfg_access_unlock(dev);
>  
> +	pcibios_sriov_disable(dev);
> +
>  	if (iov->link != dev->devfn)
>  		sysfs_remove_link(&dev->dev.kobj, "dep_link");
>  
> @@ -357,14 +358,14 @@ static void sriov_disable(struct pci_dev *dev)
>  	for (i = 0; i < iov->num_VFs; i++)
>  		pci_iov_remove_virtfn(dev, i, 0);
>  
> -	pcibios_sriov_disable(dev);
> -
>  	iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE);
>  	pci_cfg_access_lock(dev);
>  	pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
>  	ssleep(1);
>  	pci_cfg_access_unlock(dev);
>  
> +	pcibios_sriov_disable(dev);
> +
>  	if (iov->link != dev->devfn)
>  		sysfs_remove_link(&dev->dev.kobj, "dep_link");
>

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web