Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1680116 > unrolled thread

Re: [PATCH v9 2/3] PCI: Add tango PCIe host bridge support

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH v9 2/3] PCI: Add tango PCIe host bridge support Bjorn Helgaas <helgaas@kernel.org> - 2017-07-03 15:50 +0200
    Re: [PATCH v9 2/3] PCI: Add tango PCIe host bridge support Marc Gonzalez <marc_gonzalez@sigmadesigns.com> - 2017-07-03 16:40 +0200
      Re: [PATCH v9 2/3] PCI: Add tango PCIe host bridge support Bjorn Helgaas <helgaas@kernel.org> - 2017-07-04 18:00 +0200
        Re: [PATCH v9 2/3] PCI: Add tango PCIe host bridge support Mason <slash.tmp@free.fr> - 2017-07-05 01:50 +0200
    Re: [PATCH v9 2/3] PCI: Add tango PCIe host bridge support Russell King - ARM Linux <linux@armlinux.org.uk> - 2017-07-03 20:20 +0200
      Re: [PATCH v9 2/3] PCI: Add tango PCIe host bridge support Ard Biesheuvel <ard.biesheuvel@linaro.org> - 2017-07-03 20:50 +0200
      Re: [PATCH v9 2/3] PCI: Add tango PCIe host bridge support Bjorn Helgaas <helgaas@kernel.org> - 2017-07-04 17:20 +0200
        Re: [PATCH v9 2/3] PCI: Add tango PCIe host bridge support Russell King - ARM Linux <linux@armlinux.org.uk> - 2017-07-04 20:20 +0200
      Re: [PATCH v9 2/3] PCI: Add tango PCIe host bridge support Mason <slash.tmp@free.fr> - 2017-07-05 02:10 +0200
        Re: [PATCH v9 2/3] PCI: Add tango PCIe host bridge support Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-05 07:30 +0200
        Re: [PATCH v9 2/3] PCI: Add tango PCIe host bridge support Mark Brown <broonie@kernel.org> - 2017-07-05 14:40 +0200

#1680116 — Re: [PATCH v9 2/3] PCI: Add tango PCIe host bridge support

On Mon, Jul 03, 2017 at 11:54:20AM +0200, Marc Gonzalez wrote:
> Hello Bjorn,
> 
> On 03/07/2017 01:18, Bjorn Helgaas wrote:
> 
> > On Tue, Jun 20, 2017 at 10:17:40AM +0200, Marc Gonzalez wrote:
> > 
> >> This driver is required to work around several hardware bugs
> >> in the PCIe controller.
> >
> > [ Snip style issues ]
> 
> I wish you had pointed these out in your May 23 review, or in
> my March 29 version (or even just alluded to them). I would
> have fixed them in time for 2.13.

They're trivial, and if they were the only issues I would have just
done them myself while merging.

> >> +	/*
> >> +	 * QUIRK #2
> >> +	 * Unfortunately, config and mem spaces are muxed.
> >> +	 * Linux does not support such a setting, since drivers are free
> >> +	 * to access mem space directly, at any time.
> >> +	 * Therefore, we can only PRAY that config and mem space accesses
> >> +	 * NEVER occur concurrently.
> >> +	 */
> >> +	writel_relaxed(1, pcie->mux);
> >> +	ret = pci_generic_config_read(bus, devfn, where, size, val);
> >> +	writel_relaxed(0, pcie->mux);
> > 
> > I'm very hesitant about this.  When people stress this, we're going to
> > get reports of data corruption.  Even with the disclaimer below, I
> > don't feel good about this.  Adding the driver is an implicit claim
> > that we support the device, but we know it can't be made reliable.
> 
> I can't say I didn't see this coming (I had taken your long silence
> as a sign of your reluctance) but back in May, I thought you implied
> that a warning + tainting the kernel would be sufficient.
> 
> Mark Rutland points out stop_machine. I will test this solution.
> Would you find that acceptable?

Sounds possible, though I can't say for sure without seeing the code.

The problem is serializing vs. memory accesses, since they don't use
any wrappers.  However, they are ioremapped(), so it's at least
conceivable that another solution would be to use VM to trap those
accesses.  I'm not a VM person, so I don't know whether that's
feasible in Linux.

> > What is the benefit of adding this driver?  How many units are in the
> > field?  Are you hoping to have support in distros like RHEL?  Are
> > these running self-built kernels straight from kernel.org?  Is it
> > feasible for you to distribute this driver separately from the
> > upstream kernel?
> 
> The benefit of upstreaming (for me) is making kernel maintainers
> aware that one is using specific internal APIs. Then one may be
> notified when these APIs are about to change.
> 
> I'm told we have sold ~100k units. Though I don't know how many
> are in the field and using PCIe.
> 
> There are no plans to use "full-blown" distros, we use embedded
> oriented distros, such as buildroot.
> 
> Maintaining out-of-tree drivers is what we've been doing for
> ~15 years, and there are many pain-points involved. Ask Greg
> what he thinks of OOT drivers.
> 
> 
> >> +static int tango_check_pcie_link(void __iomem *test_out)
> > 
> > I think this is checking for link up.  Rename to tango_pcie_link_up()
> > to follow the convention of other drivers.  Take a struct tango_pcie *
> > instead of an address, if possible.
> 
> Anything's possible. NB: if I pass the struct, then I have to store
> the address in the struct, which isn't the case now, since I never
> need the address later. If you don't mind adding an unnecessary
> field to the struct, I can do it. What do you say?

The benefit of following the same formula as other drivers is pretty
large.  Most drivers save the equivalent of "base" in the struct.  If
you did that, you wouldn't need an extra pointer; you would just use
"base + SMP8759_MUX" in the config accessors and "base +
SMP8759_TEST_OUT" in tango_pcie_link_up().

> >> +static struct platform_driver tango_pcie_driver = {
> >> +	.probe	= tango_pcie_probe,
> >> +	.driver	= {
> >> +		.name = KBUILD_MODNAME,
> >> +		.of_match_table = tango_pcie_ids,
> > 
> > I think you need ".suppress_bind_attrs = true" here to prevent issues
> > when unbinding driver.  See
> > http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=a5f40e8098fe
> 
> Will do. In that case, I can drop tango_pcie_remove() right?

I think so; I don't think there would be any way to remove the driver.

Bjorn

[toc] | [next] | [standalone]

#1680303

Bjorn Helgaas wrote:

> Marc Gonzalez wrote:
>
>> On 03/07/2017 01:18, Bjorn Helgaas wrote:
>>
>>> On Tue, Jun 20, 2017 at 10:17:40AM +0200, Marc Gonzalez wrote:
>>>
>>>> +static int tango_check_pcie_link(void __iomem *test_out)
>>>
>>> I think this is checking for link up.  Rename to tango_pcie_link_up()
>>> to follow the convention of other drivers.  Take a struct tango_pcie *
>>> instead of an address, if possible.
>>
>> Anything's possible. NB: if I pass the struct, then I have to store
>> the address in the struct, which isn't the case now, since I never
>> need the address later. If you don't mind adding an unnecessary
>> field to the struct, I can do it. What do you say?
> 
> The benefit of following the same formula as other drivers is pretty
> large.  Most drivers save the equivalent of "base" in the struct.  If
> you did that, you wouldn't need an extra pointer; you would just use
> "base + SMP8759_MUX" in the config accessors and "base + SMP8759_TEST_OUT"
> in tango_pcie_link_up().

The problem is that TEST_OUT is at 0x74 on SMP8759, but at 0x138
on my other chip. In fact, all registers have been "reshuffled",
and none have the same offsets on the two chips.

My solution was to define specific registers in the struct.

In my [RFC PATCH v0.2] posted March 23, I tried illustrating
the issue:

+static const struct of_device_id tango_pcie_ids[] = {
+	{ .compatible = "sigma,smp8759-pcie" },
+	{ .compatible = "sigma,rev2-pcie" },
+	{ /* sentinel */ },
+};
+
+static void smp8759_init(struct tango_pcie *pcie, void __iomem *base)
+{
+	pcie->mux		= base + 0x48;
+	pcie->msi_status	= base + 0x80;
+	pcie->msi_mask		= base + 0xa0;
+	pcie->msi_doorbell	= 0xa0000000 + 0x2e07c;
+}
+
+static void rev2_init(struct tango_pcie *pcie, void __iomem *base)
+{
+	void __iomem *misc_irq	= base + 0x40;
+	void __iomem *doorbell	= base + 0x8c;
+
+	pcie->mux		= base + 0x2c;
+	pcie->msi_status	= base + 0x4c;
+	pcie->msi_mask		= base + 0x6c;
+	pcie->msi_doorbell	= 0x80000000;
+
+	writel(lower_32_bits(pcie->msi_doorbell), doorbell + 0);
+	writel(upper_32_bits(pcie->msi_doorbell), doorbell + 4);
+
+	/* Enable legacy PCI interrupts */
+	writel(BIT(15), misc_irq);
+	writel(0xf << 4, misc_irq + 4);
+}


Do you agree that the 'base + OFFSET' idiom does not work in
this specific situation? Would you handle it differently?

Regards.

[toc] | [prev] | [next] | [standalone]

#1681072

On Mon, Jul 03, 2017 at 04:34:29PM +0200, Marc Gonzalez wrote:
> Bjorn Helgaas wrote:
> 
> > Marc Gonzalez wrote:
> >
> >> On 03/07/2017 01:18, Bjorn Helgaas wrote:
> >>
> >>> On Tue, Jun 20, 2017 at 10:17:40AM +0200, Marc Gonzalez wrote:
> >>>
> >>>> +static int tango_check_pcie_link(void __iomem *test_out)
> >>>
> >>> I think this is checking for link up.  Rename to tango_pcie_link_up()
> >>> to follow the convention of other drivers.  Take a struct tango_pcie *
> >>> instead of an address, if possible.
> >>
> >> Anything's possible. NB: if I pass the struct, then I have to store
> >> the address in the struct, which isn't the case now, since I never
> >> need the address later. If you don't mind adding an unnecessary
> >> field to the struct, I can do it. What do you say?
> > 
> > The benefit of following the same formula as other drivers is pretty
> > large.  Most drivers save the equivalent of "base" in the struct.  If
> > you did that, you wouldn't need an extra pointer; you would just use
> > "base + SMP8759_MUX" in the config accessors and "base + SMP8759_TEST_OUT"
> > in tango_pcie_link_up().
> 
> The problem is that TEST_OUT is at 0x74 on SMP8759, but at 0x138
> on my other chip. In fact, all registers have been "reshuffled",
> and none have the same offsets on the two chips.
> 
> My solution was to define specific registers in the struct.
> 
> In my [RFC PATCH v0.2] posted March 23, I tried illustrating
> the issue:
> 
> +static const struct of_device_id tango_pcie_ids[] = {
> +	{ .compatible = "sigma,smp8759-pcie" },
> +	{ .compatible = "sigma,rev2-pcie" },
> +	{ /* sentinel */ },
> +};
> +
> +static void smp8759_init(struct tango_pcie *pcie, void __iomem *base)
> +{
> +	pcie->mux		= base + 0x48;
> +	pcie->msi_status	= base + 0x80;
> +	pcie->msi_mask		= base + 0xa0;
> +	pcie->msi_doorbell	= 0xa0000000 + 0x2e07c;
> +}
> +
> +static void rev2_init(struct tango_pcie *pcie, void __iomem *base)
> +{
> +	void __iomem *misc_irq	= base + 0x40;
> +	void __iomem *doorbell	= base + 0x8c;
> +
> +	pcie->mux		= base + 0x2c;
> +	pcie->msi_status	= base + 0x4c;
> +	pcie->msi_mask		= base + 0x6c;
> +	pcie->msi_doorbell	= 0x80000000;
> +
> +	writel(lower_32_bits(pcie->msi_doorbell), doorbell + 0);
> +	writel(upper_32_bits(pcie->msi_doorbell), doorbell + 4);
> +
> +	/* Enable legacy PCI interrupts */
> +	writel(BIT(15), misc_irq);
> +	writel(0xf << 4, misc_irq + 4);
> +}
> 
> 
> Do you agree that the 'base + OFFSET' idiom does not work in
> this specific situation? Would you handle it differently?

It's definitely a hassle to support chips with different register
layouts.  Your hardware guys are really making your life hard :)

drivers/pci/host/pcie-iproc.c is one strategy.  It has
iproc_pcie_reg_paxb[] and iproc_pcie_reg_paxb_v2[] with register
offsets for different chip versions.

It saves a table of register offsets per controller, which is similar
to saving a set of pointers per controller.  Saving the pointers as
you suggest above is marginally more storage but probably easier to
read.

If the chips are fundamentally different, i.e., if they *operate*
differently in addition to having a different register layout, you
could make two separate drivers.

Bjorn

[toc] | [prev] | [next] | [standalone]

#1681227

On 04/07/2017 17:58, Bjorn Helgaas wrote:

> It's definitely a hassle to support chips with different register
> layouts.  Your hardware guys are really making your life hard :)

Now where did I put my foam bat...

> If the chips are fundamentally different, i.e., if they *operate*
> differently in addition to having a different register layout, you
> could make two separate drivers.

It's the exact same underlying IP. Revision 2 is only a
bug fix rev. IIUC, some of the fixes lead to adding a
register here, removing a register there... and I don't
think the HW dev ever considered the pain of supporting
both revs within a single driver.

This dual support explains some of the peculiarities
you noted in my submission.

Regards.

[toc] | [prev] | [next] | [standalone]

#1680520

On Mon, Jul 03, 2017 at 08:40:31AM -0500, Bjorn Helgaas wrote:
> The problem is serializing vs. memory accesses, since they don't use
> any wrappers.  However, they are ioremapped(), so it's at least
> conceivable that another solution would be to use VM to trap those
> accesses.  I'm not a VM person, so I don't know whether that's
> feasible in Linux.

Bjorn,

You're forgetting that MMIO (iow, memory returned by ioremap()) must
be accessed through the appropriate accessors, and must not be
directly dereferenced in C.  (We do have buggy drivers that do that
but they are buggy, and in many cases are getting attention to fix
that.)

However, adding a spinlock into them is really not nice, because it
adds extra overhead that's only necessary for rare cases like Sigma
Designs - especially when you consider that these accessors are used
for all MMIO accesses, not just PCI.  It would effectively mean that
we end up serialising all MMIO accesses throughout the kernel when
Sigma Designs SoCs are enabled, destroying some of the SMP benefit.

I don't think we can sanely use the MMU to trap those accesses either,
that would mean sending IPIs to tell other CPUs to do something, and
waiting for them to respond - which can deadlock if we're already in
an IRQ-protected region (iirc, config accesses are made with IRQs
off.)

I don't think there's an easy solution to this problem - and I'm not
sure that stop_machine() can be made to work in this path (which
needs a process context).  I have a suspicion that the Sigma Designs
PCI implementation is just soo insane that it's never going to work
reliably in a multi-SoC kernel without introducing severe performance
issues for everyone else.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[toc] | [prev] | [next] | [standalone]

#1680535

On 3 July 2017 at 19:11, Russell King - ARM Linux <linux@armlinux.org.uk> wrote:
> On Mon, Jul 03, 2017 at 08:40:31AM -0500, Bjorn Helgaas wrote:
>> The problem is serializing vs. memory accesses, since they don't use
>> any wrappers.  However, they are ioremapped(), so it's at least
>> conceivable that another solution would be to use VM to trap those
>> accesses.  I'm not a VM person, so I don't know whether that's
>> feasible in Linux.
>
> Bjorn,
>
> You're forgetting that MMIO (iow, memory returned by ioremap()) must
> be accessed through the appropriate accessors, and must not be
> directly dereferenced in C.  (We do have buggy drivers that do that
> but they are buggy, and in many cases are getting attention to fix
> that.)
>
> However, adding a spinlock into them is really not nice, because it
> adds extra overhead that's only necessary for rare cases like Sigma
> Designs - especially when you consider that these accessors are used
> for all MMIO accesses, not just PCI.  It would effectively mean that
> we end up serialising all MMIO accesses throughout the kernel when
> Sigma Designs SoCs are enabled, destroying some of the SMP benefit.
>
> I don't think we can sanely use the MMU to trap those accesses either,
> that would mean sending IPIs to tell other CPUs to do something, and
> waiting for them to respond - which can deadlock if we're already in
> an IRQ-protected region (iirc, config accesses are made with IRQs
> off.)
>
> I don't think there's an easy solution to this problem - and I'm not
> sure that stop_machine() can be made to work in this path (which
> needs a process context).  I have a suspicion that the Sigma Designs
> PCI implementation is just soo insane that it's never going to work
> reliably in a multi-SoC kernel without introducing severe performance
> issues for everyone else.
>

I suppose we could perhaps use per-cpu spinlocks? That would put the
complexity in the Sigma config space accessors, i.e., to take each
lock before proceeding with reprogramming the outbound window, and
other implementations wouldn't have to care. However, I do agree with
Russell that having this complexity in the first place is hard to
justify if the only implementation that requires it is a wacky design
that needs lots of other quirks to operate somewhat sanely to begin
with.

[toc] | [prev] | [next] | [standalone]

#1681038

On Mon, Jul 03, 2017 at 07:11:28PM +0100, Russell King - ARM Linux wrote:
> On Mon, Jul 03, 2017 at 08:40:31AM -0500, Bjorn Helgaas wrote:
> > The problem is serializing vs. memory accesses, since they don't use
> > any wrappers.  However, they are ioremapped(), so it's at least
> > conceivable that another solution would be to use VM to trap those
> > accesses.  I'm not a VM person, so I don't know whether that's
> > feasible in Linux.
> 
> Bjorn,
> 
> You're forgetting that MMIO (iow, memory returned by ioremap()) must
> be accessed through the appropriate accessors, and must not be
> directly dereferenced in C.  (We do have buggy drivers that do that
> but they are buggy, and in many cases are getting attention to fix
> that.)

Oh, you're right, thank you!  I guess you're referring to readb()
and friends.  I haven't found an actual prohibition on directly
dereferencing addresses returned from ioremap(), but
Documentation/driver-api/device-io.rst is clear that they're
suitable for passing to readb(), etc.

I recently told someone else my mistaken idea that ioremap() must
return a valid virtual address.  I wish I remembered who it was, so I
could correct that.  Documentation/DMA-API-HOWTO.txt also suggests
that ioremap() returns a virtual address -- I think I wrote that, and
maybe that virtual address reference should be tweaked a bit.

Another wrinkle is that the pci_mmap_resource() interface is exposed
via sysfs and allows direct userspace mmap of PCI MMIO resources.  In
that case, there is no accessor available.  I wonder if we need some
way to disable this mmap when readb() is non-trivial.

Bjorn

[toc] | [prev] | [next] | [standalone]

#1681116

On Tue, Jul 04, 2017 at 10:15:02AM -0500, Bjorn Helgaas wrote:
> On Mon, Jul 03, 2017 at 07:11:28PM +0100, Russell King - ARM Linux wrote:
> > On Mon, Jul 03, 2017 at 08:40:31AM -0500, Bjorn Helgaas wrote:
> > > The problem is serializing vs. memory accesses, since they don't use
> > > any wrappers.  However, they are ioremapped(), so it's at least
> > > conceivable that another solution would be to use VM to trap those
> > > accesses.  I'm not a VM person, so I don't know whether that's
> > > feasible in Linux.
> > 
> > Bjorn,
> > 
> > You're forgetting that MMIO (iow, memory returned by ioremap()) must
> > be accessed through the appropriate accessors, and must not be
> > directly dereferenced in C.  (We do have buggy drivers that do that
> > but they are buggy, and in many cases are getting attention to fix
> > that.)
> 
> Oh, you're right, thank you!  I guess you're referring to readb()
> and friends.  I haven't found an actual prohibition on directly
> dereferencing addresses returned from ioremap(), but
> Documentation/driver-api/device-io.rst is clear that they're
> suitable for passing to readb(), etc.

There was a strong suggestion years ago that what is returned from
ioremap() is a cookie that must not be dereferenced by drivers, and
that there was a suggestion that having ioremap() return the virtual
address with an offset (which read*() and friends would undo) would
be a good idea.  However, even back then, we had some cases where
drivers would directly dereference the pointer.  We have sparse today
which helps point these places out (provided drivers stay away from
__force, but unfortunately, I think we've ended up with people who
think that silencing sparse warnings with __force is more preferable
than leaving them there to point out where things are actually wrong.)

So, imho, unfortunately sparse has lost its usefulness in this regard.

> I recently told someone else my mistaken idea that ioremap() must
> return a valid virtual address.  I wish I remembered who it was, so I
> could correct that.  Documentation/DMA-API-HOWTO.txt also suggests
> that ioremap() returns a virtual address -- I think I wrote that, and
> maybe that virtual address reference should be tweaked a bit.

For most implementations, ioremap() does indeed return a virtual address,
but that was never how the API was defined in the first place - it was
always referred to as returning a cookie.

> Another wrinkle is that the pci_mmap_resource() interface is exposed
> via sysfs and allows direct userspace mmap of PCI MMIO resources.  In
> that case, there is no accessor available.  I wonder if we need some
> way to disable this mmap when readb() is non-trivial.

Hmm, no comment, except that while the PCI MMIO space is available to
userspace, and userspace is capable of running that thread on any CPU,
PCI MMIO space can't be switched to config space.

That's another nail in this coffin...

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[toc] | [prev] | [next] | [standalone]

#1681231

On 03/07/2017 20:11, Russell King - ARM Linux wrote:

> I don't think there's an easy solution to this problem - and I'm not
> sure that stop_machine() can be made to work in this path (which
> needs a process context).  I have a suspicion that the Sigma Designs
> PCI implementation is just soo insane that it's never going to work
> reliably in a multi-SoC kernel without introducing severe performance
> issues for everyone else.

If I remember correctly, this is the second HW block from
tango that has been deemed "too insane for Linux".

The first one was the DMA engine, which doesn't interrupt
when a transfer is done, but when a new transfer may be
programmed. (Though there is a simple work-around for
this one, if we give up command pipelining.)

Do larger SoC vendors have HW devs working closely with
Linux devs, to avoid these design bloopers?

Regards.

[toc] | [prev] | [next] | [standalone]

#1681281

On Wed, Jul 05, 2017 at 01:59:55AM +0200, Mason wrote:
> Do larger SoC vendors have HW devs working closely with
> Linux devs, to avoid these design bloopers?

Yes they generally do, as they have learned from their prior mistakes :)

good luck!

greg k-h

[toc] | [prev] | [next] | [standalone]

#1681511

[Multipart message — attachments visible in raw view] — view raw

On Wed, Jul 05, 2017 at 01:59:55AM +0200, Mason wrote:

> Do larger SoC vendors have HW devs working closely with
> Linux devs, to avoid these design bloopers?

Yes, or just internal software teams in general - hardware designs tend
to be a lot more usable if there's good dialogue with users about what's
useful and what isn't.  It's going to happen at some point anyway when
people can't get the chip working well.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web