Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1390901 > unrolled thread

[PATCH kernel v4 00/11] powerpc/powernv/npu: Enable PCI pass through for NVLink

Back to article view | Back to linux.kernel

Contents

  [PATCH kernel v4 00/11] powerpc/powernv/npu: Enable PCI pass through for NVLink Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-04-29 11:10 +0200
    [PATCH kernel v4 08/11] powerpc/powernv/ioda2: Export debug helper pe_level_printk() Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-04-29 11:10 +0200
      Re: [PATCH kernel v4 08/11] powerpc/powernv/ioda2: Export debug helper pe_level_printk() Alistair Popple <alistair@popple.id.au> - 2016-05-03 07:50 +0200
        Re: [PATCH kernel v4 08/11] powerpc/powernv/ioda2: Export debug helper pe_level_printk() Alistair Popple <alistair@popple.id.au> - 2016-05-03 08:00 +0200
    [PATCH kernel v4 02/11] vfio/spapr: Relax the IOMMU compatibility check Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-04-29 11:10 +0200
      Re: [PATCH kernel v4 02/11] vfio/spapr: Relax the IOMMU  compatibility check Alex Williamson <alex.williamson@redhat.com> - 2016-04-29 17:50 +0200
      Re: [kernel,v4,02/11] vfio/spapr: Relax the IOMMU compatibility check Michael Ellerman <mpe@ellerman.id.au> - 2016-05-10 23:50 +0200
    [PATCH kernel v4 07/11] powerpc/powernv/npu: Simplify DMA setup Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-04-29 11:10 +0200
    [PATCH kernel v4 01/11] vfio_pci: Test for extended capabilities if config space > 256 bytes Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-04-29 11:10 +0200
      Re: [PATCH kernel v4 01/11] vfio_pci: Test for extended  capabilities if config space > 256 bytes Alex Williamson <alex.williamson@redhat.com> - 2016-04-29 17:50 +0200
    [PATCH kernel v4 09/11] powerpc/powernv/npu: Add set/unset window helpers Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-04-29 11:20 +0200
      Re: [PATCH kernel v4 09/11] powerpc/powernv/npu: Add set/unset window helpers Alistair Popple <alistair@popple.id.au> - 2016-05-03 08:30 +0200
    [PATCH kernel v4 06/11] powerpc/powernv/npu: Use the correct IOMMU page size Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-04-29 11:20 +0200

#1390901 — [PATCH kernel v4 00/11] powerpc/powernv/npu: Enable PCI pass through for NVLink

IBM POWER8 NVlink systems contain usual Tesla K40-ish GPUs but also
contain a couple of really fast links between GPU and CPU. These links
are exposed to the userspace by the OPAL firmware as bridges.
In order to make these links work when GPU is passed to the guest,
these bridges need to be passed as well; otherwise performance will
degrade. More details are in 11/11.

This reworks the existing NPU support in the powernv platform and adds
VFIO support on top of that.

v4 has new patch "powerpc/powernv/npu: Add set/unset window" and bunch of
cleanups.

"vfio_pci: Test for extended capabilities if config space > 256 bytes" is
included here if anyone decides to test the patchset which will crash
without it.

This was tested on POWER8NVL platform; pvr=0x004c0100.


Please comment. Thanks.

Alex, I guess we will need your "acked-by" for
"vfio/spapr: Relax the IOMMU compatibility check" to proceed.


Alexey Kardashevskiy (11):
  vfio_pci: Test for extended capabilities if config space > 256 bytes
  vfio/spapr: Relax the IOMMU compatibility check
  powerpc/powernv: Rename pnv_pci_ioda2_tce_invalidate_entire
  powerpc/powernv: Define TCE Kill flags
  powerpc/powernv/npu: TCE Kill helpers cleanup
  powerpc/powernv/npu: Use the correct IOMMU page size
  powerpc/powernv/npu: Simplify DMA setup
  powerpc/powernv/ioda2: Export debug helper pe_level_printk()
  powerpc/powernv/npu: Add set/unset window helpers
  powerpc/powernv/npu: Rework TCE Kill handling
  powerpc/powernv/npu: Enable NVLink pass through

 arch/powerpc/platforms/powernv/npu-dma.c  | 287 ++++++++++++++++--------------
 arch/powerpc/platforms/powernv/pci-ioda.c | 224 +++++++++++++++--------
 arch/powerpc/platforms/powernv/pci.h      |  31 ++--
 drivers/vfio/pci/vfio_pci_config.c        |  17 +-
 drivers/vfio/vfio_iommu_spapr_tce.c       |   3 +-
 5 files changed, 327 insertions(+), 235 deletions(-)

-- 
2.5.0.rc3

[toc] | [next] | [standalone]

#1390902 — [PATCH kernel v4 08/11] powerpc/powernv/ioda2: Export debug helper pe_level_printk()

This exports debugging helper pe_level_printk() and corresponding macroses
so they can be used in npu-dma.c.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 9 +--------
 arch/powerpc/platforms/powernv/pci.h      | 9 +++++++++
 2 files changed, 10 insertions(+), 8 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 272521e..db7695f 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -56,7 +56,7 @@
 
 static void pnv_pci_ioda2_table_free_pages(struct iommu_table *tbl);
 
-static void pe_level_printk(const struct pnv_ioda_pe *pe, const char *level,
+void pe_level_printk(const struct pnv_ioda_pe *pe, const char *level,
 			    const char *fmt, ...)
 {
 	struct va_format vaf;
@@ -87,13 +87,6 @@ static void pe_level_printk(const struct pnv_ioda_pe *pe, const char *level,
 	va_end(args);
 }
 
-#define pe_err(pe, fmt, ...)					\
-	pe_level_printk(pe, KERN_ERR, fmt, ##__VA_ARGS__)
-#define pe_warn(pe, fmt, ...)					\
-	pe_level_printk(pe, KERN_WARNING, fmt, ##__VA_ARGS__)
-#define pe_info(pe, fmt, ...)					\
-	pe_level_printk(pe, KERN_INFO, fmt, ##__VA_ARGS__)
-
 static bool pnv_iommu_bypass_disabled __read_mostly;
 
 static int __init iommu_setup(char *str)
diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
index d574a9d..485e5b1 100644
--- a/arch/powerpc/platforms/powernv/pci.h
+++ b/arch/powerpc/platforms/powernv/pci.h
@@ -236,6 +236,15 @@ extern void pnv_pci_dma_bus_setup(struct pci_bus *bus);
 extern int pnv_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type);
 extern void pnv_teardown_msi_irqs(struct pci_dev *pdev);
 
+extern void pe_level_printk(const struct pnv_ioda_pe *pe, const char *level,
+			    const char *fmt, ...);
+#define pe_err(pe, fmt, ...)					\
+	pe_level_printk(pe, KERN_ERR, fmt, ##__VA_ARGS__)
+#define pe_warn(pe, fmt, ...)					\
+	pe_level_printk(pe, KERN_WARNING, fmt, ##__VA_ARGS__)
+#define pe_info(pe, fmt, ...)					\
+	pe_level_printk(pe, KERN_INFO, fmt, ##__VA_ARGS__)
+
 /* Nvlink functions */
 extern void pnv_npu_init_dma_pe(struct pnv_ioda_pe *npe);
 extern void pnv_npu_setup_dma_pe(struct pnv_ioda_pe *npe);
-- 
2.5.0.rc3

[toc] | [prev] | [next] | [standalone]

#1393065 — Re: [PATCH kernel v4 08/11] powerpc/powernv/ioda2: Export debug helper pe_level_printk()

There's one call to pr_warn() in pnv_npu_disable_bypass() that could arguably 
be converted to pe_warn(), but we can clean that up later as the patch looks 
fine and I'm assuming subsequent patches make use of these.

Reviewed-By: Alistair Popple <alistair@popple.id.au>

On Fri, 29 Apr 2016 18:55:21 Alexey Kardashevskiy wrote:
> This exports debugging helper pe_level_printk() and corresponding macroses
> so they can be used in npu-dma.c.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> ---
>  arch/powerpc/platforms/powernv/pci-ioda.c | 9 +--------
>  arch/powerpc/platforms/powernv/pci.h      | 9 +++++++++
>  2 files changed, 10 insertions(+), 8 deletions(-)
> 
> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c 
b/arch/powerpc/platforms/powernv/pci-ioda.c
> index 272521e..db7695f 100644
> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> @@ -56,7 +56,7 @@
>  
>  static void pnv_pci_ioda2_table_free_pages(struct iommu_table *tbl);
>  
> -static void pe_level_printk(const struct pnv_ioda_pe *pe, const char 
*level,
> +void pe_level_printk(const struct pnv_ioda_pe *pe, const char *level,
>  			    const char *fmt, ...)
>  {
>  	struct va_format vaf;
> @@ -87,13 +87,6 @@ static void pe_level_printk(const struct pnv_ioda_pe *pe, 
const char *level,
>  	va_end(args);
>  }
>  
> -#define pe_err(pe, fmt, ...)					\
> -	pe_level_printk(pe, KERN_ERR, fmt, ##__VA_ARGS__)
> -#define pe_warn(pe, fmt, ...)					\
> -	pe_level_printk(pe, KERN_WARNING, fmt, ##__VA_ARGS__)
> -#define pe_info(pe, fmt, ...)					\
> -	pe_level_printk(pe, KERN_INFO, fmt, ##__VA_ARGS__)
> -
>  static bool pnv_iommu_bypass_disabled __read_mostly;
>  
>  static int __init iommu_setup(char *str)
> diff --git a/arch/powerpc/platforms/powernv/pci.h 
b/arch/powerpc/platforms/powernv/pci.h
> index d574a9d..485e5b1 100644
> --- a/arch/powerpc/platforms/powernv/pci.h
> +++ b/arch/powerpc/platforms/powernv/pci.h
> @@ -236,6 +236,15 @@ extern void pnv_pci_dma_bus_setup(struct pci_bus *bus);
>  extern int pnv_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type);
>  extern void pnv_teardown_msi_irqs(struct pci_dev *pdev);
>  
> +extern void pe_level_printk(const struct pnv_ioda_pe *pe, const char 
*level,
> +			    const char *fmt, ...);
> +#define pe_err(pe, fmt, ...)					\
> +	pe_level_printk(pe, KERN_ERR, fmt, ##__VA_ARGS__)
> +#define pe_warn(pe, fmt, ...)					\
> +	pe_level_printk(pe, KERN_WARNING, fmt, ##__VA_ARGS__)
> +#define pe_info(pe, fmt, ...)					\
> +	pe_level_printk(pe, KERN_INFO, fmt, ##__VA_ARGS__)
> +
>  /* Nvlink functions */
>  extern void pnv_npu_init_dma_pe(struct pnv_ioda_pe *npe);
>  extern void pnv_npu_setup_dma_pe(struct pnv_ioda_pe *npe);
> 

[toc] | [prev] | [next] | [standalone]

#1393071 — Re: [PATCH kernel v4 08/11] powerpc/powernv/ioda2: Export debug helper pe_level_printk()

On Tue, 3 May 2016 15:46:33 Alistair Popple wrote:
> There's one call to pr_warn() in pnv_npu_disable_bypass() that could 
arguably 
> be converted to pe_warn(), but we can clean that up later as the patch looks 
> fine and I'm assuming subsequent patches make use of these.

And inevitably the next patch in the series cleans that up anyway. Feel free 
to ignore the noise above :-)

> Reviewed-By: Alistair Popple <alistair@popple.id.au>
> 
> On Fri, 29 Apr 2016 18:55:21 Alexey Kardashevskiy wrote:
> > This exports debugging helper pe_level_printk() and corresponding macroses
> > so they can be used in npu-dma.c.
> > 
> > Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> > ---
> >  arch/powerpc/platforms/powernv/pci-ioda.c | 9 +--------
> >  arch/powerpc/platforms/powernv/pci.h      | 9 +++++++++
> >  2 files changed, 10 insertions(+), 8 deletions(-)
> > 
> > diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c 
> b/arch/powerpc/platforms/powernv/pci-ioda.c
> > index 272521e..db7695f 100644
> > --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> > +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> > @@ -56,7 +56,7 @@
> >  
> >  static void pnv_pci_ioda2_table_free_pages(struct iommu_table *tbl);
> >  
> > -static void pe_level_printk(const struct pnv_ioda_pe *pe, const char 
> *level,
> > +void pe_level_printk(const struct pnv_ioda_pe *pe, const char *level,
> >  			    const char *fmt, ...)
> >  {
> >  	struct va_format vaf;
> > @@ -87,13 +87,6 @@ static void pe_level_printk(const struct pnv_ioda_pe 
*pe, 
> const char *level,
> >  	va_end(args);
> >  }
> >  
> > -#define pe_err(pe, fmt, ...)					\
> > -	pe_level_printk(pe, KERN_ERR, fmt, ##__VA_ARGS__)
> > -#define pe_warn(pe, fmt, ...)					\
> > -	pe_level_printk(pe, KERN_WARNING, fmt, ##__VA_ARGS__)
> > -#define pe_info(pe, fmt, ...)					\
> > -	pe_level_printk(pe, KERN_INFO, fmt, ##__VA_ARGS__)
> > -
> >  static bool pnv_iommu_bypass_disabled __read_mostly;
> >  
> >  static int __init iommu_setup(char *str)
> > diff --git a/arch/powerpc/platforms/powernv/pci.h 
> b/arch/powerpc/platforms/powernv/pci.h
> > index d574a9d..485e5b1 100644
> > --- a/arch/powerpc/platforms/powernv/pci.h
> > +++ b/arch/powerpc/platforms/powernv/pci.h
> > @@ -236,6 +236,15 @@ extern void pnv_pci_dma_bus_setup(struct pci_bus 
*bus);
> >  extern int pnv_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type);
> >  extern void pnv_teardown_msi_irqs(struct pci_dev *pdev);
> >  
> > +extern void pe_level_printk(const struct pnv_ioda_pe *pe, const char 
> *level,
> > +			    const char *fmt, ...);
> > +#define pe_err(pe, fmt, ...)					\
> > +	pe_level_printk(pe, KERN_ERR, fmt, ##__VA_ARGS__)
> > +#define pe_warn(pe, fmt, ...)					\
> > +	pe_level_printk(pe, KERN_WARNING, fmt, ##__VA_ARGS__)
> > +#define pe_info(pe, fmt, ...)					\
> > +	pe_level_printk(pe, KERN_INFO, fmt, ##__VA_ARGS__)
> > +
> >  /* Nvlink functions */
> >  extern void pnv_npu_init_dma_pe(struct pnv_ioda_pe *npe);
> >  extern void pnv_npu_setup_dma_pe(struct pnv_ioda_pe *npe);
> > 

[toc] | [prev] | [next] | [standalone]

#1390903 — [PATCH kernel v4 02/11] vfio/spapr: Relax the IOMMU compatibility check

We are going to have multiple different types of PHB on the same system
with POWER8 + NVLink and PHBs will have different IOMMU ops. However
we only really care about one callback - create_table - so we can
relax the compatibility check here.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
 drivers/vfio/vfio_iommu_spapr_tce.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
index 0582b72..3054e3f 100644
--- a/drivers/vfio/vfio_iommu_spapr_tce.c
+++ b/drivers/vfio/vfio_iommu_spapr_tce.c
@@ -1188,7 +1188,8 @@ static int tce_iommu_attach_group(void *iommu_data,
 			goto unlock_exit;
 		}
 		table_group_tmp = iommu_group_get_iommudata(tcegrp->grp);
-		if (table_group_tmp->ops != table_group->ops) {
+		if (table_group_tmp->ops->create_table !=
+				table_group->ops->create_table) {
 			pr_warn("tce_vfio: Group %d is incompatible with group %d\n",
 					iommu_group_id(iommu_group),
 					iommu_group_id(tcegrp->grp));
-- 
2.5.0.rc3

[toc] | [prev] | [next] | [standalone]

#1391264 — Re: [PATCH kernel v4 02/11] vfio/spapr: Relax the IOMMU compatibility check

On Fri, 29 Apr 2016 18:55:15 +1000
Alexey Kardashevskiy <aik@ozlabs.ru> wrote:

> We are going to have multiple different types of PHB on the same system
> with POWER8 + NVLink and PHBs will have different IOMMU ops. However
> we only really care about one callback - create_table - so we can
> relax the compatibility check here.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
>  drivers/vfio/vfio_iommu_spapr_tce.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
> index 0582b72..3054e3f 100644
> --- a/drivers/vfio/vfio_iommu_spapr_tce.c
> +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
> @@ -1188,7 +1188,8 @@ static int tce_iommu_attach_group(void *iommu_data,
>  			goto unlock_exit;
>  		}
>  		table_group_tmp = iommu_group_get_iommudata(tcegrp->grp);
> -		if (table_group_tmp->ops != table_group->ops) {
> +		if (table_group_tmp->ops->create_table !=
> +				table_group->ops->create_table) {
>  			pr_warn("tce_vfio: Group %d is incompatible with group %d\n",
>  					iommu_group_id(iommu_group),
>  					iommu_group_id(tcegrp->grp));

Acked-by: Alex Williamson <alex.williamson@redhat.com>

[toc] | [prev] | [next] | [standalone]

#1398543 — Re: [kernel,v4,02/11] vfio/spapr: Relax the IOMMU compatibility check

On Fri, 2016-29-04 at 08:55:15 UTC, Alexey Kardashevskiy wrote:
> We are going to have multiple different types of PHB on the same system
> with POWER8 + NVLink and PHBs will have different IOMMU ops. However
> we only really care about one callback - create_table - so we can
> relax the compatibility check here.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> Acked-by: Alex Williamson <alex.williamson@redhat.com>

Series applied to powerpc next, thanks.

https://git.kernel.org/powerpc/c/45295687a90a31135ab575803e

cheers

[toc] | [prev] | [next] | [standalone]

#1390906 — [PATCH kernel v4 07/11] powerpc/powernv/npu: Simplify DMA setup

NPU devices are emulated in firmware and mainly used for NPU NVLink
training; one NPU device is per a hardware link. Their DMA/TCE setup
must match the GPU which is connected via PCIe and NVLink so any changes
to the DMA/TCE setup on the GPU PCIe device need to be propagated to
the NVLink device as this is what device drivers expect and it doesn't
make much sense to do anything else.

This makes NPU DMA setup explicit.
pnv_npu_ioda_controller_ops::pnv_npu_dma_set_mask is moved to pci-ioda,
made static and prints warning as dma_set_mask() should never be called
on this function as in any case it will not configure GPU; so we make
this explicit.

Instead of using PNV_IODA_PE_PEER and peers[] (which the next patch will
remove), we test every PCI device if there are corresponding NVLink
devices. If there are any, we propagate bypass mode to just found NPU
devices by calling the setup helper directly (which takes @bypass) and
avoid guessing (i.e. calculating from DMA mask) whether we need bypass
or not on NPU devices. Since DMA setup happens in very rare occasion,
this will not slow down booting or VFIO start/stop much.

This renames pnv_npu_disable_bypass to pnv_npu_dma_set_32 to make it
more clear what the function really does which is programming 32bit
table address to the TVT ("disabling bypass" means writing zeroes to
the TVT).

This removes pnv_npu_dma_set_bypass() from pnv_npu_ioda_fixup() as
the DMA configuration on NPU does not matter until dma_set_mask() is
called on GPU and that will do the NPU DMA configuration.

This removes phb->dma_dev_setup initialization for NPU as
pnv_pci_ioda_dma_dev_setup is no-op for it anyway.

This stops using npe->tce_bypass_base as it never changes and values
other than zero are not supported.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
Reviewed-by: Alistair Popple <alistair@popple.id.au>
---
Changes:
v2:
* changed first paragraph of the commit log from Alistair comment
* removed npe->tce_bypass_base
---
 arch/powerpc/platforms/powernv/npu-dma.c  | 89 ++++++++++++++-----------------
 arch/powerpc/platforms/powernv/pci-ioda.c | 30 +++++------
 arch/powerpc/platforms/powernv/pci.h      |  3 +-
 3 files changed, 53 insertions(+), 69 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/npu-dma.c b/arch/powerpc/platforms/powernv/npu-dma.c
index 5bd5fee..bec9267 100644
--- a/arch/powerpc/platforms/powernv/npu-dma.c
+++ b/arch/powerpc/platforms/powernv/npu-dma.c
@@ -196,10 +196,9 @@ void pnv_npu_init_dma_pe(struct pnv_ioda_pe *npe)
 }
 
 /*
- * For the NPU we want to point the TCE table at the same table as the
- * real PCI device.
+ * Enables 32 bit DMA on NPU.
  */
-static void pnv_npu_disable_bypass(struct pnv_ioda_pe *npe)
+static void pnv_npu_dma_set_32(struct pnv_ioda_pe *npe)
 {
 	struct pnv_phb *phb = npe->phb;
 	struct pci_dev *gpdev;
@@ -235,72 +234,62 @@ static void pnv_npu_disable_bypass(struct pnv_ioda_pe *npe)
 }
 
 /*
- * Enable/disable bypass mode on the NPU. The NPU only supports one
+ * Enables bypass mode on the NPU. The NPU only supports one
  * window per link, so bypass needs to be explicitly enabled or
  * disabled. Unlike for a PHB3 bypass and non-bypass modes can't be
  * active at the same time.
  */
-int pnv_npu_dma_set_bypass(struct pnv_ioda_pe *npe, bool enable)
+static int pnv_npu_dma_set_bypass(struct pnv_ioda_pe *npe)
 {
 	struct pnv_phb *phb = npe->phb;
 	int64_t rc = 0;
+	phys_addr_t top = memblock_end_of_DRAM();
 
 	if (phb->type != PNV_PHB_NPU || !npe->pdev)
 		return -EINVAL;
 
-	if (enable) {
-		/* Enable the bypass window */
-		phys_addr_t top = memblock_end_of_DRAM();
+	/* Enable the bypass window */
 
-		npe->tce_bypass_base = 0;
-		top = roundup_pow_of_two(top);
-		dev_info(&npe->pdev->dev, "Enabling bypass for PE %d\n",
-			 npe->pe_number);
-		rc = opal_pci_map_pe_dma_window_real(phb->opal_id,
-					npe->pe_number, npe->pe_number,
-					npe->tce_bypass_base, top);
-	} else {
-		/*
-		 * Disable the bypass window by replacing it with the
-		 * TCE32 window.
-		 */
-		pnv_npu_disable_bypass(npe);
-	}
+	top = roundup_pow_of_two(top);
+	dev_info(&npe->pdev->dev, "Enabling bypass for PE %d\n",
+			npe->pe_number);
+	rc = opal_pci_map_pe_dma_window_real(phb->opal_id,
+			npe->pe_number, npe->pe_number,
+			0 /* bypass base */, top);
 
 	return rc;
 }
 
-int pnv_npu_dma_set_mask(struct pci_dev *npdev, u64 dma_mask)
+void pnv_npu_try_dma_set_bypass(struct pci_dev *gpdev, bool bypass)
 {
-	struct pci_controller *hose = pci_bus_to_host(npdev->bus);
-	struct pnv_phb *phb = hose->private_data;
-	struct pci_dn *pdn = pci_get_pdn(npdev);
-	struct pnv_ioda_pe *npe, *gpe;
-	struct pci_dev *gpdev;
-	uint64_t top;
-	bool bypass = false;
+	int i;
+	struct pnv_phb *phb;
+	struct pci_dn *pdn;
+	struct pnv_ioda_pe *npe;
+	struct pci_dev *npdev;
 
-	if (WARN_ON(!pdn || pdn->pe_number == IODA_INVALID_PE))
-		return -ENXIO;
+	for (i = 0; ; ++i) {
+		npdev = pnv_pci_get_npu_dev(gpdev, i);
 
-	/* We only do bypass if it's enabled on the linked device */
-	npe = &phb->ioda.pe_array[pdn->pe_number];
-	gpe = get_gpu_pci_dev_and_pe(npe, &gpdev);
-	if (!gpe)
-		return -ENODEV;
+		if (!npdev)
+			break;
 
-	if (gpe->tce_bypass_enabled) {
-		top = gpe->tce_bypass_base + memblock_end_of_DRAM() - 1;
-		bypass = (dma_mask >= top);
+		pdn = pci_get_pdn(npdev);
+		if (WARN_ON(!pdn || pdn->pe_number == IODA_INVALID_PE))
+			return;
+
+		phb = pci_bus_to_host(npdev->bus)->private_data;
+
+		/* We only do bypass if it's enabled on the linked device */
+		npe = &phb->ioda.pe_array[pdn->pe_number];
+
+		if (bypass) {
+			dev_info(&npdev->dev,
+					"Using 64-bit DMA iommu bypass\n");
+			pnv_npu_dma_set_bypass(npe);
+		} else {
+			dev_info(&npdev->dev, "Using 32-bit DMA via iommu\n");
+			pnv_npu_dma_set_32(npe);
+		}
 	}
-
-	if (bypass)
-		dev_info(&npdev->dev, "Using 64-bit DMA iommu bypass\n");
-	else
-		dev_info(&npdev->dev, "Using 32-bit DMA via iommu\n");
-
-	pnv_npu_dma_set_bypass(npe, bypass);
-	*npdev->dev.dma_mask = dma_mask;
-
-	return 0;
 }
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index a67d51e..272521e 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -1640,8 +1640,6 @@ static int pnv_pci_ioda_dma_set_mask(struct pci_dev *pdev, u64 dma_mask)
 	struct pnv_ioda_pe *pe;
 	uint64_t top;
 	bool bypass = false;
-	struct pci_dev *linked_npu_dev;
-	int i;
 
 	if (WARN_ON(!pdn || pdn->pe_number == IODA_INVALID_PE))
 		return -ENODEV;;
@@ -1662,15 +1660,7 @@ static int pnv_pci_ioda_dma_set_mask(struct pci_dev *pdev, u64 dma_mask)
 	*pdev->dev.dma_mask = dma_mask;
 
 	/* Update peer npu devices */
-	if (pe->flags & PNV_IODA_PE_PEER)
-		for (i = 0; i < PNV_IODA_MAX_PEER_PES; i++) {
-			if (!pe->peers[i])
-				continue;
-
-			linked_npu_dev = pe->peers[i]->pdev;
-			if (dma_get_mask(&linked_npu_dev->dev) != dma_mask)
-				dma_set_mask(&linked_npu_dev->dev, dma_mask);
-		}
+	pnv_npu_try_dma_set_bypass(pdev, bypass);
 
 	return 0;
 }
@@ -3094,7 +3084,6 @@ static void pnv_npu_ioda_fixup(void)
 			enable_bypass = dma_get_mask(&pe->pdev->dev) ==
 				DMA_BIT_MASK(64);
 			pnv_npu_init_dma_pe(pe);
-			pnv_npu_dma_set_bypass(pe, enable_bypass);
 		}
 	}
 }
@@ -3246,6 +3235,14 @@ static const struct pci_controller_ops pnv_pci_ioda_controller_ops = {
        .shutdown = pnv_pci_ioda_shutdown,
 };
 
+static int pnv_npu_dma_set_mask(struct pci_dev *npdev, u64 dma_mask)
+{
+	dev_err_once(&npdev->dev,
+			"%s operation unsupported for NVLink devices\n",
+			__func__);
+	return -EPERM;
+}
+
 static const struct pci_controller_ops pnv_npu_ioda_controller_ops = {
 	.dma_dev_setup = pnv_pci_dma_dev_setup,
 #ifdef CONFIG_PCI_MSI
@@ -3402,9 +3399,6 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	/* Setup RID -> PE mapping function */
 	phb->bdfn_to_pe = pnv_ioda_bdfn_to_pe;
 
-	/* Setup TCEs */
-	phb->dma_dev_setup = pnv_pci_ioda_dma_dev_setup;
-
 	/* Setup MSI support */
 	pnv_pci_init_ioda_msis(phb);
 
@@ -3417,10 +3411,12 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	 */
 	ppc_md.pcibios_fixup = pnv_pci_ioda_fixup;
 
-	if (phb->type == PNV_PHB_NPU)
+	if (phb->type == PNV_PHB_NPU) {
 		hose->controller_ops = pnv_npu_ioda_controller_ops;
-	else
+	} else {
+		phb->dma_dev_setup = pnv_pci_ioda_dma_dev_setup;
 		hose->controller_ops = pnv_pci_ioda_controller_ops;
+	}
 
 #ifdef CONFIG_PCI_IOV
 	ppc_md.pcibios_fixup_sriov = pnv_pci_ioda_fixup_iov_resources;
diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
index 0b89a4c..d574a9d 100644
--- a/arch/powerpc/platforms/powernv/pci.h
+++ b/arch/powerpc/platforms/powernv/pci.h
@@ -239,8 +239,7 @@ extern void pnv_teardown_msi_irqs(struct pci_dev *pdev);
 /* Nvlink functions */
 extern void pnv_npu_init_dma_pe(struct pnv_ioda_pe *npe);
 extern void pnv_npu_setup_dma_pe(struct pnv_ioda_pe *npe);
-extern int pnv_npu_dma_set_bypass(struct pnv_ioda_pe *npe, bool enabled);
-extern int pnv_npu_dma_set_mask(struct pci_dev *npdev, u64 dma_mask);
+extern void pnv_npu_try_dma_set_bypass(struct pci_dev *gpdev, bool bypass);
 extern void pnv_pci_ioda2_tce_invalidate_entire(struct pnv_phb *phb, bool rm);
 
 #endif /* __POWERNV_PCI_H */
-- 
2.5.0.rc3

[toc] | [prev] | [next] | [standalone]

#1390909 — [PATCH kernel v4 01/11] vfio_pci: Test for extended capabilities if config space > 256 bytes

PCI-Express spec says that reading 4 bytes at offset 100h should return
zero if there is no extended capability so VFIO reads this dword to
know if there are extended capabilities.

However it is not always possible to access the extended space so
generic PCI code in pci_cfg_space_size_ext() checks if
pci_read_config_dword() can read beyond 100h and if the check fails,
it sets the config space size to 100h.

VFIO does its own extended capabilities check by reading at offset 100h
which may produce 0xffffffff which VFIO treats as the extended config
space presense and calls vfio_ecap_init() which fails to parse
capabilities (which is expected) but right before the exit, it writes
zero at offset 100h which is beyond the buffer allocated for
vdev->vconfig (which is 256 bytes) which leads to random memory
corruption.

This makes VFIO only check for the extended capabilities if
the discovered config size is more than 256 bytes.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
Changes:
v2:
* instead of checking for 0xffffffff, this only does the check if
device's config size is big enough
---
 drivers/vfio/pci/vfio_pci_config.c | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c
index 142c533..d0c4358 100644
--- a/drivers/vfio/pci/vfio_pci_config.c
+++ b/drivers/vfio/pci/vfio_pci_config.c
@@ -1124,9 +1124,12 @@ static int vfio_cap_len(struct vfio_pci_device *vdev, u8 cap, u8 pos)
 			return pcibios_err_to_errno(ret);
 
 		if (PCI_X_CMD_VERSION(word)) {
-			/* Test for extended capabilities */
-			pci_read_config_dword(pdev, PCI_CFG_SPACE_SIZE, &dword);
-			vdev->extended_caps = (dword != 0);
+			if (pdev->cfg_size > PCI_CFG_SPACE_SIZE) {
+				/* Test for extended capabilities */
+				pci_read_config_dword(pdev, PCI_CFG_SPACE_SIZE,
+						&dword);
+				vdev->extended_caps = (dword != 0);
+			}
 			return PCI_CAP_PCIX_SIZEOF_V2;
 		} else
 			return PCI_CAP_PCIX_SIZEOF_V0;
@@ -1138,9 +1141,11 @@ static int vfio_cap_len(struct vfio_pci_device *vdev, u8 cap, u8 pos)
 
 		return byte;
 	case PCI_CAP_ID_EXP:
-		/* Test for extended capabilities */
-		pci_read_config_dword(pdev, PCI_CFG_SPACE_SIZE, &dword);
-		vdev->extended_caps = (dword != 0);
+		if (pdev->cfg_size > PCI_CFG_SPACE_SIZE) {
+			/* Test for extended capabilities */
+			pci_read_config_dword(pdev, PCI_CFG_SPACE_SIZE, &dword);
+			vdev->extended_caps = dword != 0;
+		}
 
 		/* length based on version */
 		if ((pcie_caps_reg(pdev) & PCI_EXP_FLAGS_VERS) == 1)
-- 
2.5.0.rc3

[toc] | [prev] | [next] | [standalone]

#1391263 — Re: [PATCH kernel v4 01/11] vfio_pci: Test for extended capabilities if config space > 256 bytes

On Fri, 29 Apr 2016 18:55:14 +1000
Alexey Kardashevskiy <aik@ozlabs.ru> wrote:

> PCI-Express spec says that reading 4 bytes at offset 100h should return
> zero if there is no extended capability so VFIO reads this dword to
> know if there are extended capabilities.
> 
> However it is not always possible to access the extended space so
> generic PCI code in pci_cfg_space_size_ext() checks if
> pci_read_config_dword() can read beyond 100h and if the check fails,
> it sets the config space size to 100h.
> 
> VFIO does its own extended capabilities check by reading at offset 100h
> which may produce 0xffffffff which VFIO treats as the extended config
> space presense and calls vfio_ecap_init() which fails to parse
> capabilities (which is expected) but right before the exit, it writes
> zero at offset 100h which is beyond the buffer allocated for
> vdev->vconfig (which is 256 bytes) which leads to random memory
> corruption.
> 
> This makes VFIO only check for the extended capabilities if
> the discovered config size is more than 256 bytes.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> ---
> Changes:
> v2:
> * instead of checking for 0xffffffff, this only does the check if
> device's config size is big enough
> ---

I'll take this patch separately, please drop from this series.  Thanks,

Alex

>  drivers/vfio/pci/vfio_pci_config.c | 17 +++++++++++------
>  1 file changed, 11 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c
> index 142c533..d0c4358 100644
> --- a/drivers/vfio/pci/vfio_pci_config.c
> +++ b/drivers/vfio/pci/vfio_pci_config.c
> @@ -1124,9 +1124,12 @@ static int vfio_cap_len(struct vfio_pci_device *vdev, u8 cap, u8 pos)
>  			return pcibios_err_to_errno(ret);
>  
>  		if (PCI_X_CMD_VERSION(word)) {
> -			/* Test for extended capabilities */
> -			pci_read_config_dword(pdev, PCI_CFG_SPACE_SIZE, &dword);
> -			vdev->extended_caps = (dword != 0);
> +			if (pdev->cfg_size > PCI_CFG_SPACE_SIZE) {
> +				/* Test for extended capabilities */
> +				pci_read_config_dword(pdev, PCI_CFG_SPACE_SIZE,
> +						&dword);
> +				vdev->extended_caps = (dword != 0);
> +			}
>  			return PCI_CAP_PCIX_SIZEOF_V2;
>  		} else
>  			return PCI_CAP_PCIX_SIZEOF_V0;
> @@ -1138,9 +1141,11 @@ static int vfio_cap_len(struct vfio_pci_device *vdev, u8 cap, u8 pos)
>  
>  		return byte;
>  	case PCI_CAP_ID_EXP:
> -		/* Test for extended capabilities */
> -		pci_read_config_dword(pdev, PCI_CFG_SPACE_SIZE, &dword);
> -		vdev->extended_caps = (dword != 0);
> +		if (pdev->cfg_size > PCI_CFG_SPACE_SIZE) {
> +			/* Test for extended capabilities */
> +			pci_read_config_dword(pdev, PCI_CFG_SPACE_SIZE, &dword);
> +			vdev->extended_caps = dword != 0;
> +		}
>  
>  		/* length based on version */
>  		if ((pcie_caps_reg(pdev) & PCI_EXP_FLAGS_VERS) == 1)

[toc] | [prev] | [next] | [standalone]

#1390915 — [PATCH kernel v4 09/11] powerpc/powernv/npu: Add set/unset window helpers

The upcoming NVLink passthrough support will require NPU code to cope
with two DMA windows.

This adds a pnv_npu_set_window() helper which programs 32bit window to
the hardware. This also adds multilevel TCE support.

This adds a pnv_npu_unset_window() helper which removes the DMA window
from the hardware. This does not make difference now as the caller -
pnv_npu_dma_set_bypass() - enables bypass in the hardware but the next
patch will use it to manage TCE table lists for TCE Kill handling.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/platforms/powernv/npu-dma.c | 65 +++++++++++++++++++++++++++-----
 1 file changed, 55 insertions(+), 10 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/npu-dma.c b/arch/powerpc/platforms/powernv/npu-dma.c
index bec9267..800d70f 100644
--- a/arch/powerpc/platforms/powernv/npu-dma.c
+++ b/arch/powerpc/platforms/powernv/npu-dma.c
@@ -159,6 +159,56 @@ static struct pnv_ioda_pe *get_gpu_pci_dev_and_pe(struct pnv_ioda_pe *npe,
 	return pe;
 }
 
+static long pnv_npu_set_window(struct pnv_ioda_pe *npe,
+		struct iommu_table *tbl)
+{
+	struct pnv_phb *phb = npe->phb;
+	int64_t rc;
+	const unsigned long size = tbl->it_indirect_levels ?
+		tbl->it_level_size : tbl->it_size;
+	const __u64 start_addr = tbl->it_offset << tbl->it_page_shift;
+	const __u64 win_size = tbl->it_size << tbl->it_page_shift;
+
+	pe_info(npe, "Setting up window %llx..%llx pg=%lx\n",
+			start_addr, start_addr + win_size - 1,
+			IOMMU_PAGE_SIZE(tbl));
+
+	rc = opal_pci_map_pe_dma_window(phb->opal_id,
+			npe->pe_number,
+			npe->pe_number,
+			tbl->it_indirect_levels + 1,
+			__pa(tbl->it_base),
+			size << 3,
+			IOMMU_PAGE_SIZE(tbl));
+	if (rc) {
+		pe_err(npe, "Failed to configure TCE table, err %lld\n", rc);
+		return rc;
+	}
+	pnv_pci_ioda2_tce_invalidate_entire(phb, false);
+
+	return 0;
+}
+
+static long pnv_npu_unset_window(struct pnv_ioda_pe *npe)
+{
+	struct pnv_phb *phb = npe->phb;
+	int64_t rc;
+
+	pe_info(npe, "Removing DMA window\n");
+
+	rc = opal_pci_map_pe_dma_window(phb->opal_id, npe->pe_number,
+			npe->pe_number,
+			0/* levels */, 0/* table address */,
+			0/* table size */, 0/* page size */);
+	if (rc) {
+		pe_err(npe, "Unmapping failed, ret = %lld\n", rc);
+		return rc;
+	}
+	pnv_pci_ioda2_tce_invalidate_entire(phb, false);
+
+	return 0;
+}
+
 void pnv_npu_init_dma_pe(struct pnv_ioda_pe *npe)
 {
 	struct pnv_ioda_pe *gpe;
@@ -200,10 +250,8 @@ void pnv_npu_init_dma_pe(struct pnv_ioda_pe *npe)
  */
 static void pnv_npu_dma_set_32(struct pnv_ioda_pe *npe)
 {
-	struct pnv_phb *phb = npe->phb;
 	struct pci_dev *gpdev;
 	struct pnv_ioda_pe *gpe;
-	struct iommu_table *tbl;
 	int64_t rc;
 
 	/*
@@ -217,14 +265,7 @@ static void pnv_npu_dma_set_32(struct pnv_ioda_pe *npe)
 	if (!gpe)
 		return;
 
-	tbl = gpe->table_group.tables[0];
-	rc = opal_pci_map_pe_dma_window(phb->opal_id, npe->pe_number,
-					npe->pe_number, 1, __pa(tbl->it_base),
-					tbl->it_size << 3,
-					IOMMU_PAGE_SIZE(tbl));
-	if (rc != OPAL_SUCCESS)
-		pr_warn("%s: Error %lld setting DMA window on PHB#%d-PE#%d\n",
-			__func__, rc, phb->hose->global_number, npe->pe_number);
+	rc = pnv_npu_set_window(npe, gpe->table_group.tables[0]);
 
 	/*
 	 * We don't initialise npu_pe->tce32_table as we always use
@@ -248,6 +289,10 @@ static int pnv_npu_dma_set_bypass(struct pnv_ioda_pe *npe)
 	if (phb->type != PNV_PHB_NPU || !npe->pdev)
 		return -EINVAL;
 
+	rc = pnv_npu_unset_window(npe);
+	if (rc != OPAL_SUCCESS)
+		return rc;
+
 	/* Enable the bypass window */
 
 	top = roundup_pow_of_two(top);
-- 
2.5.0.rc3

[toc] | [prev] | [next] | [standalone]

#1393079 — Re: [PATCH kernel v4 09/11] powerpc/powernv/npu: Add set/unset window helpers

On Fri, 29 Apr 2016 18:55:22 Alexey Kardashevskiy wrote:
> The upcoming NVLink passthrough support will require NPU code to cope
> with two DMA windows.
> 
> This adds a pnv_npu_set_window() helper which programs 32bit window to
> the hardware. This also adds multilevel TCE support.
> 
> This adds a pnv_npu_unset_window() helper which removes the DMA window
> from the hardware. This does not make difference now as the caller -
> pnv_npu_dma_set_bypass() - enables bypass in the hardware but the next
> patch will use it to manage TCE table lists for TCE Kill handling.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> ---
>  arch/powerpc/platforms/powernv/npu-dma.c | 65 
+++++++++++++++++++++++++++-----
>  1 file changed, 55 insertions(+), 10 deletions(-)
> 
> diff --git a/arch/powerpc/platforms/powernv/npu-dma.c 
b/arch/powerpc/platforms/powernv/npu-dma.c
> index bec9267..800d70f 100644
> --- a/arch/powerpc/platforms/powernv/npu-dma.c
> +++ b/arch/powerpc/platforms/powernv/npu-dma.c
> @@ -159,6 +159,56 @@ static struct pnv_ioda_pe 
*get_gpu_pci_dev_and_pe(struct pnv_ioda_pe *npe,
>  	return pe;
>  }
>  
> +static long pnv_npu_set_window(struct pnv_ioda_pe *npe,
> +		struct iommu_table *tbl)
> +{
> +	struct pnv_phb *phb = npe->phb;
> +	int64_t rc;
> +	const unsigned long size = tbl->it_indirect_levels ?
> +		tbl->it_level_size : tbl->it_size;
> +	const __u64 start_addr = tbl->it_offset << tbl->it_page_shift;
> +	const __u64 win_size = tbl->it_size << tbl->it_page_shift;
> +
> +	pe_info(npe, "Setting up window %llx..%llx pg=%lx\n",
> +			start_addr, start_addr + win_size - 1,
> +			IOMMU_PAGE_SIZE(tbl));
> +
> +	rc = opal_pci_map_pe_dma_window(phb->opal_id,
> +			npe->pe_number,
> +			npe->pe_number,
> +			tbl->it_indirect_levels + 1,
> +			__pa(tbl->it_base),
> +			size << 3,
> +			IOMMU_PAGE_SIZE(tbl));
> +	if (rc) {
> +		pe_err(npe, "Failed to configure TCE table, err %lld\n", rc);
> +		return rc;
> +	}
> +	pnv_pci_ioda2_tce_invalidate_entire(phb, false);
> +
> +	return 0;
> +}
> +
> +static long pnv_npu_unset_window(struct pnv_ioda_pe *npe)
> +{
> +	struct pnv_phb *phb = npe->phb;
> +	int64_t rc;
> +
> +	pe_info(npe, "Removing DMA window\n");
> +
> +	rc = opal_pci_map_pe_dma_window(phb->opal_id, npe->pe_number,
> +			npe->pe_number,
> +			0/* levels */, 0/* table address */,
> +			0/* table size */, 0/* page size */);
> +	if (rc) {
> +		pe_err(npe, "Unmapping failed, ret = %lld\n", rc);
> +		return rc;
> +	}
> +	pnv_pci_ioda2_tce_invalidate_entire(phb, false);
> +
> +	return 0;
> +}
> +
>  void pnv_npu_init_dma_pe(struct pnv_ioda_pe *npe)
>  {
>  	struct pnv_ioda_pe *gpe;
> @@ -200,10 +250,8 @@ void pnv_npu_init_dma_pe(struct pnv_ioda_pe *npe)
>   */
>  static void pnv_npu_dma_set_32(struct pnv_ioda_pe *npe)
>  {
> -	struct pnv_phb *phb = npe->phb;
>  	struct pci_dev *gpdev;
>  	struct pnv_ioda_pe *gpe;
> -	struct iommu_table *tbl;
>  	int64_t rc;
>  
>  	/*
> @@ -217,14 +265,7 @@ static void pnv_npu_dma_set_32(struct pnv_ioda_pe *npe)
>  	if (!gpe)
>  		return;
>  
> -	tbl = gpe->table_group.tables[0];
> -	rc = opal_pci_map_pe_dma_window(phb->opal_id, npe->pe_number,
> -					npe->pe_number, 1, __pa(tbl->it_base),
> -					tbl->it_size << 3,
> -					IOMMU_PAGE_SIZE(tbl));
> -	if (rc != OPAL_SUCCESS)
> -		pr_warn("%s: Error %lld setting DMA window on PHB#%d-PE#%d\n",
> -			__func__, rc, phb->hose->global_number, npe-
>pe_number);
> +	rc = pnv_npu_set_window(npe, gpe->table_group.tables[0]);
>  
>  	/*
>  	 * We don't initialise npu_pe->tce32_table as we always use
> @@ -248,6 +289,10 @@ static int pnv_npu_dma_set_bypass(struct pnv_ioda_pe 
*npe)
>  	if (phb->type != PNV_PHB_NPU || !npe->pdev)
>  		return -EINVAL;
>  
> +	rc = pnv_npu_unset_window(npe);

As noted in the commit message you technically don't need to do this as the 
subsequent OPAL call to enable the bypass window below will just overwrite the 
same TVE. However I like the logic of clearing the existing TVE before writing 
a new one anyway as it also ensures the TCE cache gets properly invalidated.

Other than that this patch looks like it doesn't change any existing 
behaviour.

Reviewed-By: Alistair Popple <alistair@popple.id.au>

> +	if (rc != OPAL_SUCCESS)
> +		return rc;
> +
>  	/* Enable the bypass window */
>  
>  	top = roundup_pow_of_two(top);
> 

[toc] | [prev] | [next] | [standalone]

#1390918 — [PATCH kernel v4 06/11] powerpc/powernv/npu: Use the correct IOMMU page size

This uses the page size from iommu_table instead of hard-coded 4K.
This should cause no change in behavior.

While we are here, move bits around to prepare for further rework
which will define and use iommu_table_group_ops.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
Reviewed-by: Alistair Popple <alistair@popple.id.au>
---
 arch/powerpc/platforms/powernv/npu-dma.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/npu-dma.c b/arch/powerpc/platforms/powernv/npu-dma.c
index 778570c..5bd5fee 100644
--- a/arch/powerpc/platforms/powernv/npu-dma.c
+++ b/arch/powerpc/platforms/powernv/npu-dma.c
@@ -204,8 +204,7 @@ static void pnv_npu_disable_bypass(struct pnv_ioda_pe *npe)
 	struct pnv_phb *phb = npe->phb;
 	struct pci_dev *gpdev;
 	struct pnv_ioda_pe *gpe;
-	void *addr;
-	unsigned int size;
+	struct iommu_table *tbl;
 	int64_t rc;
 
 	/*
@@ -219,11 +218,11 @@ static void pnv_npu_disable_bypass(struct pnv_ioda_pe *npe)
 	if (!gpe)
 		return;
 
-	addr = (void *)gpe->table_group.tables[0]->it_base;
-	size = gpe->table_group.tables[0]->it_size << 3;
+	tbl = gpe->table_group.tables[0];
 	rc = opal_pci_map_pe_dma_window(phb->opal_id, npe->pe_number,
-					npe->pe_number, 1, __pa(addr),
-					size, 0x1000);
+					npe->pe_number, 1, __pa(tbl->it_base),
+					tbl->it_size << 3,
+					IOMMU_PAGE_SIZE(tbl));
 	if (rc != OPAL_SUCCESS)
 		pr_warn("%s: Error %lld setting DMA window on PHB#%d-PE#%d\n",
 			__func__, rc, phb->hose->global_number, npe->pe_number);
-- 
2.5.0.rc3

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web