Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1268033

RE: PCIe host controller behind IOMMU on ARM

Show all headers | View raw

Hi Arnd,

On 12 November 2015 09:49, Arnd Bergmann wrote:
> On Thursday 12 November 2015 09:26:33 Phil Edworthy wrote:
> > On 11 November 2015 18:25, LIviu wrote:
> > > On Mon, Nov 09, 2015 at 12:32:13PM +0000, Phil Edworthy wrote:
> 
> > > I think you're mixing things a bit or not explaining them very well. Having the
> > > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot
> > > carry 64-bit addresses. It depends on how they get translated by the host
> bridge
> > > or its associated ATS block. I can't see why you can't have a setup where
> > > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit.
> > > You just have to be careful on how you setup your mem64 ranges so that
> they
> > > don't
> > > overlap with the 32-bit ranges when translated.
> > From a HW point of view I agree that we can setup the PCI host bridge such that
> > it uses 64-bit PCI address, with 32-bit cpu addresses. Though in practice doesn't
> > this mean that the dma ops used by card drivers has to be provided by our PCI
> > host bridge driver so we can apply the translation to those PCI addresses?
> > This comes back to my point below about how to do this. Adding a bus notifier
> > to do this may be too late, and arm64 doesn't implement set_dma_ops().
> >
> > > And no, you should not limit at the card driver the DMA_BIT_MASK() unless
> the
> > > card is not capable of supporting more than 32-bit addresses.
> > If there was infrastructure that checked all parents dma-ranges when the
> > dma_set_mask() function is called as Arnd pointed out, this would nicely solve
> > the problem.
> 
> of_dma_configure calls of_dma_get_range to do all this for the PCIe host,
> and then calls arch_setup_dma_ops() so the architecture specific code can
> enforce the limits in dma_set_mask and pick an appropriate set of dma
> operations. The missing part is in the implementation of arch_setup_dma_ops,
> which currently happily ignores the base and limit.
I don't think it's as simple as that, though I could be wrong!

First off, of_dma_configure() sets a default coherent_dma_mask to 4GiB.
This default is set for the 'platform soc' device. For my own testing I increased
this to DMA_BIT_MASK(63). Note that setting it to DMA_BIT_MASK(64) causes
boot failure that I haven't looked into.

Then pci_device_add() sets the devices coherent_dma_mask to 4GiB before
calling of_pci_dma_configure(). I assume it does this on the basis that this is a
good default for PCI drivers that don't call dma_set_mask().
So if arch_setup_dma_ops() walks up the parents to limit the mask, you'll hit
this mask.

Finally, dma_set_mask_and_coherent() is called from the PCI card driver
but it doesn't check the parents dma masks either.

Thanks
Phil

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

PCIe host controller behind IOMMU on ARM Phil Edworthy <phil.edworthy@renesas.com> - 2015-11-04 15:00 +0100
  Re: PCIe host controller behind IOMMU on ARM "Liviu.Dudau@arm.com" <Liviu.Dudau@arm.com> - 2015-11-04 15:30 +0100
    RE: PCIe host controller behind IOMMU on ARM Phil Edworthy <phil.edworthy@renesas.com> - 2015-11-04 15:50 +0100
      Re: PCIe host controller behind IOMMU on ARM "Liviu.Dudau@arm.com" <Liviu.Dudau@arm.com> - 2015-11-04 16:10 +0100
        RE: PCIe host controller behind IOMMU on ARM Phil Edworthy <phil.edworthy@renesas.com> - 2015-11-04 16:20 +0100
          Re: PCIe host controller behind IOMMU on ARM Will Deacon <will.deacon@arm.com> - 2015-11-04 16:40 +0100
            RE: PCIe host controller behind IOMMU on ARM Phil Edworthy <phil.edworthy@renesas.com> - 2015-11-04 19:10 +0100
        RE: PCIe host controller behind IOMMU on ARM Phil Edworthy <phil.edworthy@renesas.com> - 2015-11-09 13:40 +0100
          Re: PCIe host controller behind IOMMU on ARM "Liviu.Dudau@arm.com" <Liviu.Dudau@arm.com> - 2015-11-11 19:30 +0100
            Re: PCIe host controller behind IOMMU on ARM Arnd Bergmann <arnd@arndb.de> - 2015-11-11 21:30 +0100
            RE: PCIe host controller behind IOMMU on ARM Phil Edworthy <phil.edworthy@renesas.com> - 2015-11-12 10:30 +0100
              Re: PCIe host controller behind IOMMU on ARM Arnd Bergmann <arnd@arndb.de> - 2015-11-12 11:00 +0100
                RE: PCIe host controller behind IOMMU on ARM Phil Edworthy <phil.edworthy@renesas.com> - 2015-11-12 16:40 +0100
                Re: PCIe host controller behind IOMMU on ARM Arnd Bergmann <arnd@arndb.de> - 2015-11-12 17:20 +0100
                RE: PCIe host controller behind IOMMU on ARM Phil Edworthy <phil.edworthy@renesas.com> - 2015-11-13 14:10 +0100
                Re: PCIe host controller behind IOMMU on ARM Arnd Bergmann <arnd@arndb.de> - 2015-11-13 15:10 +0100
                RE: PCIe host controller behind IOMMU on ARM Phil Edworthy <phil.edworthy@renesas.com> - 2015-11-13 15:20 +0100
              Re: PCIe host controller behind IOMMU on ARM "Liviu.Dudau@arm.com" <Liviu.Dudau@arm.com> - 2015-11-12 11:40 +0100

csiph-web