Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1677347 > unrolled thread
| Started by | Logan Gunthorpe <logang@deltatee.com> |
|---|---|
| First post | 2017-06-29 05:30 +0200 |
| Last post | 2017-06-29 17:40 +0200 |
| Articles | 16 — 3 participants |
Back to article view | Back to linux.kernel
[PATCH 00/16] Logan Gunthorpe <logang@deltatee.com> - 2017-06-29 05:30 +0200
[PATCH 14/16] switchtec_ntb: implement scratchpad registers Logan Gunthorpe <logang@deltatee.com> - 2017-06-29 05:30 +0200
RE: [PATCH 14/16] switchtec_ntb: implement scratchpad registers "Allen Hubbe" <Allen.Hubbe@dell.com> - 2017-06-29 20:20 +0200
Re: [PATCH 14/16] switchtec_ntb: implement scratchpad registers Logan Gunthorpe <logang@deltatee.com> - 2017-06-29 20:40 +0200
[PATCH 09/16] switchtec_ntb: initialize hardware for memory windows Logan Gunthorpe <logang@deltatee.com> - 2017-06-29 05:30 +0200
[PATCH 01/16] switchtec: move structure definitions into a common header Logan Gunthorpe <logang@deltatee.com> - 2017-06-29 05:30 +0200
[PATCH 04/16] switchtec: add link event notifier callback Logan Gunthorpe <logang@deltatee.com> - 2017-06-29 05:40 +0200
[PATCH 05/16] ntb: ensure ntb_mw_get_align is only called when the link is up Logan Gunthorpe <logang@deltatee.com> - 2017-06-29 05:40 +0200
RE: [PATCH 05/16] ntb: ensure ntb_mw_get_align is only called when the link is up "Allen Hubbe" <Allen.Hubbe@dell.com> - 2017-06-29 20:20 +0200
[PATCH 12/16] switchtec_ntb: add link management Logan Gunthorpe <logang@deltatee.com> - 2017-06-29 05:40 +0200
RE: [PATCH 12/16] switchtec_ntb: add link management "Allen Hubbe" <Allen.Hubbe@dell.com> - 2017-06-29 20:20 +0200
Re: [PATCH 12/16] switchtec_ntb: add link management Logan Gunthorpe <logang@deltatee.com> - 2017-06-29 20:40 +0200
[PATCH 11/16] switchtec_ntb: add skeleton ntb driver Logan Gunthorpe <logang@deltatee.com> - 2017-06-29 05:40 +0200
[PATCH 10/16] switchtec_ntb: initialize hardware for doorbells and messages Logan Gunthorpe <logang@deltatee.com> - 2017-06-29 05:40 +0200
Re: [PATCH 00/16] Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-06-29 17:20 +0200
Re: [PATCH 00/16] Logan Gunthorpe <logang@deltatee.com> - 2017-06-29 17:40 +0200
| From | Logan Gunthorpe <logang@deltatee.com> |
|---|---|
| Date | 2017-06-29 05:30 +0200 |
| Subject | [PATCH 00/16] |
| Message-ID | <tXyjw-2hu-3@gated-at.bofh.it> |
Changes since the rfc:
- Rebased on ntb-next
- Switched ntb_part_op to use sleep instead of delay
- Dropped a number of useless dbg __func__ prints
- Went back to the dynamic instead of the static class
- Swapped the notifier block for a simple callback
- Modified the new ntb api so that a couple functions with pidx
now must be called after link up. Per our discussion on the list.
Also, please note, the first patch in the series will conflict with
switchtec patches that are queued up for 4.13. So this shouldn't
be merged until I can rebase it onto a later kernel. However, it
is suitable for review.
@Bjorn: per the discussion on the RFC, I think it's best if
this series goes through the NTB tree. Therefore, we just need an ack
from you when you have time.
Thanks,
Logan
--
This patchset implements Non-Transparent Bridge (NTB) support for the
Microsemi Switchtec series of switches. We're looking for some
review from the community at this point but hope to get it upstreamed
for v4.14.
Switchtec NTB support is configured over the same function and bar
as the management endpoint. Thus, the new driver hooks into the
management driver which we had merged in v4.12. We use the class
interface API to register an NTB device for every switchtec device
which supports NTB (not all do).
The Switchtec hardware supports doorbells, memory windows and messages.
Seeing there is no native scratchpad support, 128 spads are emulated
through the use of a pre-setup memory window. The switch has 64
doorbells which are shared between the two partitions and a
configurable set of memory windows. While the hardware supports more
than 2 partitions, this driver only supports the first two seeing
the current NTB API only supports two hosts.
The driver has been tested with ntb_netdev and fully passes the
ntb_test script.
This patchset is based off of ntb-next and can be found in this
git repo:
https://github.com/sbates130272/linux-p2pmem.git switchtec_ntb_next
Logan Gunthorpe (16):
switchtec: move structure definitions into a common header
switchtec: export class symbol for use in upper layer driver
switchtec: add ntb hardware register definitions
switchtec: add link event notifier callback
ntb: ensure ntb_mw_get_align is only called when the link is up
ntb: add check and comment for link up to mw_count and mw_get_align
ntb: ntb_test: ensure the link is up before trying to configure the
mws
switchtec_ntb: introduce initial ntb driver
switchtec_ntb: initialize hardware for memory windows
switchtec_ntb: initialize hardware for doorbells and messages
switchtec_ntb: add skeleton ntb driver
switchtec_ntb: add link management
switchtec_ntb: implement doorbell registers
switchtec_ntb: implement scratchpad registers
switchtec_ntb: add memory window support
switchtec_ntb: update switchtec documentation with notes for ntb
Documentation/switchtec.txt | 12 +
MAINTAINERS | 2 +
drivers/ntb/hw/Kconfig | 1 +
drivers/ntb/hw/Makefile | 1 +
drivers/ntb/hw/mscc/Kconfig | 9 +
drivers/ntb/hw/mscc/Makefile | 1 +
drivers/ntb/hw/mscc/switchtec_ntb.c | 1188 +++++++++++++++++++++++++++++++
drivers/ntb/ntb_transport.c | 20 +-
drivers/ntb/test/ntb_perf.c | 18 +-
drivers/ntb/test/ntb_tool.c | 6 +-
drivers/pci/switch/switchtec.c | 304 ++------
include/linux/ntb.h | 14 +-
include/linux/switchtec.h | 364 ++++++++++
tools/testing/selftests/ntb/ntb_test.sh | 4 +
14 files changed, 1670 insertions(+), 274 deletions(-)
create mode 100644 drivers/ntb/hw/mscc/Kconfig
create mode 100644 drivers/ntb/hw/mscc/Makefile
create mode 100644 drivers/ntb/hw/mscc/switchtec_ntb.c
create mode 100644 include/linux/switchtec.h
--
2.11.0
[toc] | [next] | [standalone]
| From | Logan Gunthorpe <logang@deltatee.com> |
|---|---|
| Date | 2017-06-29 05:30 +0200 |
| Subject | [PATCH 14/16] switchtec_ntb: implement scratchpad registers |
| Message-ID | <tXyjx-2hu-21@gated-at.bofh.it> |
| In reply to | #1677347 |
Seeing there is no dedicated hardware for this, we simply add
these as entries in the shared memory window. Thus, we could support
any number of them but 128 seems like enough, for now.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Stephen Bates <sbates@raithlin.com>
Reviewed-by: Kurt Schwemmer <kurt.schwemmer@microsemi.com>
---
drivers/ntb/hw/mscc/switchtec_ntb.c | 75 ++++++++++++++++++++++++++++++++++++-
1 file changed, 73 insertions(+), 2 deletions(-)
diff --git a/drivers/ntb/hw/mscc/switchtec_ntb.c b/drivers/ntb/hw/mscc/switchtec_ntb.c
index 980acf2e5b42..a42e80742b52 100644
--- a/drivers/ntb/hw/mscc/switchtec_ntb.c
+++ b/drivers/ntb/hw/mscc/switchtec_ntb.c
@@ -69,6 +69,7 @@ struct shared_mw {
u16 nr_direct_mw;
u16 nr_lut_mw;
u64 mw_sizes[MAX_MWS];
+ u32 spad[128];
};
#define MAX_DIRECT_MW ARRAY_SIZE(((struct ntb_ctrl_regs *)(0))->bar_entry)
@@ -453,22 +454,90 @@ static int switchtec_ntb_peer_db_set(struct ntb_dev *ntb, u64 db_bits)
static int switchtec_ntb_spad_count(struct ntb_dev *ntb)
{
- return 0;
+ struct switchtec_ntb *sndev = ntb_sndev(ntb);
+
+ return ARRAY_SIZE(sndev->self_shared->spad);
}
static u32 switchtec_ntb_spad_read(struct ntb_dev *ntb, int idx)
{
- return 0;
+ struct switchtec_ntb *sndev = ntb_sndev(ntb);
+
+ if (idx < 0 || idx >= ARRAY_SIZE(sndev->self_shared->spad))
+ return 0;
+
+ if (!sndev->self_shared)
+ return 0;
+
+ return sndev->self_shared->spad[idx];
}
static int switchtec_ntb_spad_write(struct ntb_dev *ntb, int idx, u32 val)
{
+ struct switchtec_ntb *sndev = ntb_sndev(ntb);
+
+ if (idx < 0 || idx >= ARRAY_SIZE(sndev->self_shared->spad))
+ return -EINVAL;
+
+ if (!sndev->self_shared)
+ return -EIO;
+
+ sndev->self_shared->spad[idx] = val;
+
return 0;
}
+static u32 switchtec_ntb_peer_spad_read(struct ntb_dev *ntb, int pidx,
+ int sidx)
+{
+ struct switchtec_ntb *sndev = ntb_sndev(ntb);
+
+ if (pidx != NTB_DEF_PEER_IDX)
+ return -EINVAL;
+
+ if (sidx < 0 || sidx >= ARRAY_SIZE(sndev->peer_shared->spad))
+ return 0;
+
+ if (!sndev->peer_shared)
+ return 0;
+
+ return ioread32(&sndev->peer_shared->spad[sidx]);
+}
+
static int switchtec_ntb_peer_spad_write(struct ntb_dev *ntb, int pidx,
int sidx, u32 val)
{
+ struct switchtec_ntb *sndev = ntb_sndev(ntb);
+
+ if (pidx != NTB_DEF_PEER_IDX)
+ return -EINVAL;
+
+ if (sidx < 0 || sidx >= ARRAY_SIZE(sndev->peer_shared->spad))
+ return -EINVAL;
+
+ if (!sndev->peer_shared)
+ return -EIO;
+
+ iowrite32(val, &sndev->peer_shared->spad[sidx]);
+
+ return 0;
+}
+
+static int switchtec_ntb_peer_spad_addr(struct ntb_dev *ntb, int pidx,
+ int sidx, phys_addr_t *spad_addr)
+{
+ struct switchtec_ntb *sndev = ntb_sndev(ntb);
+ unsigned long offset;
+
+ if (pidx != NTB_DEF_PEER_IDX)
+ return -EINVAL;
+
+ offset = (unsigned long)&sndev->peer_shared->spad[sidx] -
+ (unsigned long)sndev->stdev->mmio;
+
+ if (spad_addr)
+ *spad_addr = pci_resource_start(ntb->pdev, 0) + offset;
+
return 0;
}
@@ -494,7 +563,9 @@ static const struct ntb_dev_ops switchtec_ntb_ops = {
.spad_count = switchtec_ntb_spad_count,
.spad_read = switchtec_ntb_spad_read,
.spad_write = switchtec_ntb_spad_write,
+ .peer_spad_read = switchtec_ntb_peer_spad_read,
.peer_spad_write = switchtec_ntb_peer_spad_write,
+ .peer_spad_addr = switchtec_ntb_peer_spad_addr,
};
static void switchtec_ntb_init_sndev(struct switchtec_ntb *sndev)
--
2.11.0
[toc] | [prev] | [next] | [standalone]
| From | "Allen Hubbe" <Allen.Hubbe@dell.com> |
|---|---|
| Date | 2017-06-29 20:20 +0200 |
| Subject | RE: [PATCH 14/16] switchtec_ntb: implement scratchpad registers |
| Message-ID | <tXMcN-2zD-11@gated-at.bofh.it> |
| In reply to | #1677348 |
From: Logan Gunthorpe
> Seeing there is no dedicated hardware for this, we simply add
> these as entries in the shared memory window. Thus, we could support
> any number of them but 128 seems like enough, for now.
Scratchpads are not natively supported by this hardware,
- but the hardware does natively support more than two ntb ports,
- but this software substitute for scratchpads looks like it only works with two ntb ports:
> + if (pidx != NTB_DEF_PEER_IDX)
> + return -EINVAL;
This could get in the way of letting the driver support more than two ports later on. Is there already a plan to change this to support more than two ports?
This is also not the only hardware to lack scratchpads, but does have memory windows. Wouldn't it be better for a software substitute like this to be done in a way that it is not tied to a specific hardware driver?
>
> Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
> Reviewed-by: Stephen Bates <sbates@raithlin.com>
> Reviewed-by: Kurt Schwemmer <kurt.schwemmer@microsemi.com>
> ---
> drivers/ntb/hw/mscc/switchtec_ntb.c | 75 ++++++++++++++++++++++++++++++++++++-
> 1 file changed, 73 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/ntb/hw/mscc/switchtec_ntb.c b/drivers/ntb/hw/mscc/switchtec_ntb.c
> index 980acf2e5b42..a42e80742b52 100644
> --- a/drivers/ntb/hw/mscc/switchtec_ntb.c
> +++ b/drivers/ntb/hw/mscc/switchtec_ntb.c
> @@ -69,6 +69,7 @@ struct shared_mw {
> u16 nr_direct_mw;
> u16 nr_lut_mw;
> u64 mw_sizes[MAX_MWS];
> + u32 spad[128];
> };
>
> #define MAX_DIRECT_MW ARRAY_SIZE(((struct ntb_ctrl_regs *)(0))->bar_entry)
> @@ -453,22 +454,90 @@ static int switchtec_ntb_peer_db_set(struct ntb_dev *ntb, u64 db_bits)
>
> static int switchtec_ntb_spad_count(struct ntb_dev *ntb)
> {
> - return 0;
> + struct switchtec_ntb *sndev = ntb_sndev(ntb);
> +
> + return ARRAY_SIZE(sndev->self_shared->spad);
> }
>
> static u32 switchtec_ntb_spad_read(struct ntb_dev *ntb, int idx)
> {
> - return 0;
> + struct switchtec_ntb *sndev = ntb_sndev(ntb);
> +
> + if (idx < 0 || idx >= ARRAY_SIZE(sndev->self_shared->spad))
> + return 0;
> +
> + if (!sndev->self_shared)
> + return 0;
> +
> + return sndev->self_shared->spad[idx];
> }
>
> static int switchtec_ntb_spad_write(struct ntb_dev *ntb, int idx, u32 val)
> {
> + struct switchtec_ntb *sndev = ntb_sndev(ntb);
> +
> + if (idx < 0 || idx >= ARRAY_SIZE(sndev->self_shared->spad))
> + return -EINVAL;
> +
> + if (!sndev->self_shared)
> + return -EIO;
> +
> + sndev->self_shared->spad[idx] = val;
> +
> return 0;
> }
>
> +static u32 switchtec_ntb_peer_spad_read(struct ntb_dev *ntb, int pidx,
> + int sidx)
> +{
> + struct switchtec_ntb *sndev = ntb_sndev(ntb);
> +
> + if (pidx != NTB_DEF_PEER_IDX)
> + return -EINVAL;
> +
> + if (sidx < 0 || sidx >= ARRAY_SIZE(sndev->peer_shared->spad))
> + return 0;
> +
> + if (!sndev->peer_shared)
> + return 0;
> +
> + return ioread32(&sndev->peer_shared->spad[sidx]);
> +}
> +
> static int switchtec_ntb_peer_spad_write(struct ntb_dev *ntb, int pidx,
> int sidx, u32 val)
> {
> + struct switchtec_ntb *sndev = ntb_sndev(ntb);
> +
> + if (pidx != NTB_DEF_PEER_IDX)
> + return -EINVAL;
> +
> + if (sidx < 0 || sidx >= ARRAY_SIZE(sndev->peer_shared->spad))
> + return -EINVAL;
> +
> + if (!sndev->peer_shared)
> + return -EIO;
> +
> + iowrite32(val, &sndev->peer_shared->spad[sidx]);
> +
> + return 0;
> +}
> +
> +static int switchtec_ntb_peer_spad_addr(struct ntb_dev *ntb, int pidx,
> + int sidx, phys_addr_t *spad_addr)
> +{
> + struct switchtec_ntb *sndev = ntb_sndev(ntb);
> + unsigned long offset;
> +
> + if (pidx != NTB_DEF_PEER_IDX)
> + return -EINVAL;
> +
> + offset = (unsigned long)&sndev->peer_shared->spad[sidx] -
> + (unsigned long)sndev->stdev->mmio;
> +
> + if (spad_addr)
> + *spad_addr = pci_resource_start(ntb->pdev, 0) + offset;
> +
> return 0;
> }
>
> @@ -494,7 +563,9 @@ static const struct ntb_dev_ops switchtec_ntb_ops = {
> .spad_count = switchtec_ntb_spad_count,
> .spad_read = switchtec_ntb_spad_read,
> .spad_write = switchtec_ntb_spad_write,
> + .peer_spad_read = switchtec_ntb_peer_spad_read,
> .peer_spad_write = switchtec_ntb_peer_spad_write,
> + .peer_spad_addr = switchtec_ntb_peer_spad_addr,
> };
>
> static void switchtec_ntb_init_sndev(struct switchtec_ntb *sndev)
> --
> 2.11.0
[toc] | [prev] | [next] | [standalone]
| From | Logan Gunthorpe <logang@deltatee.com> |
|---|---|
| Date | 2017-06-29 20:40 +0200 |
| Subject | Re: [PATCH 14/16] switchtec_ntb: implement scratchpad registers |
| Message-ID | <tXMwa-2FO-9@gated-at.bofh.it> |
| In reply to | #1678037 |
On 6/29/2017 12:11 PM, Allen Hubbe wrote: > This could get in the way of letting the driver support more than two ports later on. Is there already a plan to change this to support more than two ports? Well, as I mentioned this patchset is only to support 2 ports. Future work will expand this and remove the restriction. The core concept of the emulated spads doesn't get in the way but will need some further consideration. But multi-port support will need a lot more work in the ntb clients anyway. There's no sense having support in my driver until I can actually test things. And I'll probably be fine doing some of the client work once we get there. > This is also not the only hardware to lack scratchpads, but does have memory windows. Wouldn't it be better for a software substitute like this to be done in a way that it is not tied to a specific hardware driver? Yeah, we discussed this previously. Some of this is kind of hardware dependent though. In this case we are using a small fixed size LUT window. If the other hardware doesn't have similar support, using up one of the BAR memory windows (which are a very limited resource) for this might not make sense. The switchtec driver also uses this special memory window to negotiate link status. Anyway, I'm happy to collaborate to making what common parts we can possible with the IDT driver but Serge seemed against the whole concept of emulation. Logan
[toc] | [prev] | [next] | [standalone]
| From | Logan Gunthorpe <logang@deltatee.com> |
|---|---|
| Date | 2017-06-29 05:30 +0200 |
| Subject | [PATCH 09/16] switchtec_ntb: initialize hardware for memory windows |
| Message-ID | <tXyjx-2hu-23@gated-at.bofh.it> |
| In reply to | #1677347 |
This commit adds the code to initialize the memory windows in the
hardware. This includes setting up the requester ID table, and
figuring out which bar corresponds to which memory window. (Seeing
the switch can be configured with any number of bars.)
Also, seeing the device doesn't have hardware for scratchpads or
determining the link status, we create a shared memory window that has
these features. A magic number with a version copmonent will be used
to determine if the otherside's driver is actually up.
The shared memory window also informs the other side of the
size and count of the local memory windows.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Stephen Bates <sbates@raithlin.com>
Reviewed-by: Kurt Schwemmer <kurt.schwemmer@microsemi.com>
---
drivers/ntb/hw/mscc/switchtec_ntb.c | 338 ++++++++++++++++++++++++++++++++++++
1 file changed, 338 insertions(+)
diff --git a/drivers/ntb/hw/mscc/switchtec_ntb.c b/drivers/ntb/hw/mscc/switchtec_ntb.c
index 253efba72275..ff03424153d9 100644
--- a/drivers/ntb/hw/mscc/switchtec_ntb.c
+++ b/drivers/ntb/hw/mscc/switchtec_ntb.c
@@ -15,37 +15,374 @@
#include <linux/switchtec.h>
#include <linux/module.h>
+#include <linux/delay.h>
+#include <linux/kthread.h>
MODULE_DESCRIPTION("Microsemi Switchtec(tm) NTB Driver");
MODULE_VERSION("0.1");
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Microsemi Corporation");
+static bool use_lut_mws;
+module_param(use_lut_mws, bool, 0644);
+MODULE_PARM_DESC(use_lut_mws,
+ "Enable the use of the LUT based memory windows");
+
+#ifndef ioread64
+#ifdef readq
+#define ioread64 readq
+#else
+#define ioread64 _ioread64
+static inline u64 _ioread64(void __iomem *mmio)
+{
+ u64 low, high;
+
+ low = ioread32(mmio);
+ high = ioread32(mmio + sizeof(u32));
+ return low | (high << 32);
+}
+#endif
+#endif
+
+#ifndef iowrite64
+#ifdef writeq
+#define iowrite64 writeq
+#else
+#define iowrite64 _iowrite64
+static inline void _iowrite64(u64 val, void __iomem *mmio)
+{
+ iowrite32(val, mmio);
+ iowrite32(val >> 32, mmio + sizeof(u32));
+}
+#endif
+#endif
+
+#define SWITCHTEC_NTB_MAGIC 0x45CC0001
+#define MAX_MWS 128
+
+struct shared_mw {
+ u32 magic;
+ u32 partition_id;
+ u16 nr_direct_mw;
+ u16 nr_lut_mw;
+ u64 mw_sizes[MAX_MWS];
+};
+
+#define MAX_DIRECT_MW ARRAY_SIZE(((struct ntb_ctrl_regs *)(0))->bar_entry)
+#define LUT_SIZE SZ_64K
+
struct switchtec_ntb {
struct switchtec_dev *stdev;
+
+ int self_partition;
+ int peer_partition;
+
+ struct ntb_info_regs __iomem *mmio_ntb;
+ struct ntb_ctrl_regs __iomem *mmio_ctrl;
+ struct ntb_dbmsg_regs __iomem *mmio_dbmsg;
+ struct ntb_ctrl_regs __iomem *mmio_self_ctrl;
+ struct ntb_ctrl_regs __iomem *mmio_peer_ctrl;
+ struct ntb_dbmsg_regs __iomem *mmio_self_dbmsg;
+
+ struct shared_mw *self_shared;
+ struct shared_mw __iomem *peer_shared;
+ dma_addr_t self_shared_dma;
+
+ int nr_direct_mw;
+ int nr_lut_mw;
+ int direct_mw_to_bar[MAX_DIRECT_MW];
};
+static int switchtec_ntb_part_op(struct switchtec_ntb *sndev,
+ struct ntb_ctrl_regs __iomem *ctl,
+ u32 op, int wait_status)
+{
+ static const char * const op_text[] = {
+ [NTB_CTRL_PART_OP_LOCK] = "lock",
+ [NTB_CTRL_PART_OP_CFG] = "configure",
+ [NTB_CTRL_PART_OP_RESET] = "reset",
+ };
+
+ int i;
+ u32 ps;
+ int status;
+
+ switch (op) {
+ case NTB_CTRL_PART_OP_LOCK:
+ status = NTB_CTRL_PART_STATUS_LOCKING;
+ break;
+ case NTB_CTRL_PART_OP_CFG:
+ status = NTB_CTRL_PART_STATUS_CONFIGURING;
+ break;
+ case NTB_CTRL_PART_OP_RESET:
+ status = NTB_CTRL_PART_STATUS_RESETTING;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ iowrite32(op, &ctl->partition_op);
+
+ for (i = 0; i < 1000; i++) {
+ if (msleep_interruptible(50) != 0) {
+ iowrite32(NTB_CTRL_PART_OP_RESET, &ctl->partition_op);
+ return -EINTR;
+ }
+
+ ps = ioread32(&ctl->partition_status) & 0xFFFF;
+
+ if (ps != status)
+ break;
+ }
+
+ if (ps == wait_status)
+ return 0;
+
+ if (ps == status) {
+ dev_err(&sndev->stdev->dev,
+ "Timed out while peforming %s (%d). (%08x)",
+ op_text[op], op,
+ ioread32(&ctl->partition_status));
+
+ return -ETIMEDOUT;
+ }
+
+ return -EIO;
+}
+
+static void switchtec_ntb_init_sndev(struct switchtec_ntb *sndev)
+{
+ u64 part_map;
+
+ sndev->self_partition = sndev->stdev->partition;
+
+ sndev->mmio_ntb = sndev->stdev->mmio_ntb;
+ part_map = ioread64(&sndev->mmio_ntb->ep_map);
+ part_map &= ~(1 << sndev->self_partition);
+ sndev->peer_partition = ffs(part_map) - 1;
+
+ dev_dbg(&sndev->stdev->dev, "Partition ID %d of %d (%llx)",
+ sndev->self_partition, sndev->stdev->partition_count,
+ part_map);
+
+ sndev->mmio_ctrl = (void * __iomem)sndev->mmio_ntb +
+ SWITCHTEC_NTB_REG_CTRL_OFFSET;
+ sndev->mmio_dbmsg = (void * __iomem)sndev->mmio_ntb +
+ SWITCHTEC_NTB_REG_DBMSG_OFFSET;
+
+ sndev->mmio_self_ctrl = &sndev->mmio_ctrl[sndev->self_partition];
+ sndev->mmio_peer_ctrl = &sndev->mmio_ctrl[sndev->peer_partition];
+ sndev->mmio_self_dbmsg = &sndev->mmio_dbmsg[sndev->self_partition];
+}
+
+static void switchtec_ntb_init_mw(struct switchtec_ntb *sndev)
+{
+ int i;
+ int cnt = 0;
+
+ for (i = 0; i < ARRAY_SIZE(sndev->mmio_self_ctrl->bar_entry); i++) {
+ u32 r = ioread32(&sndev->mmio_self_ctrl->bar_entry[i].ctl);
+
+ if (r & NTB_CTRL_BAR_VALID)
+ sndev->direct_mw_to_bar[cnt++] = i;
+ }
+
+ sndev->nr_direct_mw = cnt;
+
+ sndev->nr_lut_mw = ioread16(&sndev->mmio_self_ctrl->lut_table_entries);
+ sndev->nr_lut_mw = rounddown_pow_of_two(sndev->nr_lut_mw);
+
+ dev_dbg(&sndev->stdev->dev, "MWs: %d direct, %d lut",
+ sndev->nr_direct_mw, cnt - sndev->nr_direct_mw);
+}
+
+static int switchtec_ntb_init_req_id_table(struct switchtec_ntb *sndev)
+{
+ int rc = 0;
+ u16 req_id = ioread16(&sndev->mmio_ntb->requester_id);
+ u32 error;
+
+ if (ioread32(&sndev->mmio_self_ctrl->req_id_table_size) < 2) {
+ dev_err(&sndev->stdev->dev,
+ "Not enough requester IDs available.");
+ return -EFAULT;
+ }
+
+ rc = switchtec_ntb_part_op(sndev, sndev->mmio_self_ctrl,
+ NTB_CTRL_PART_OP_LOCK,
+ NTB_CTRL_PART_STATUS_LOCKED);
+ if (rc)
+ return rc;
+
+ iowrite32(NTB_PART_CTRL_ID_PROT_DIS,
+ &sndev->mmio_self_ctrl->partition_ctrl);
+
+ // Root Complex Requester ID
+ iowrite32(0 << 16 | NTB_CTRL_REQ_ID_EN,
+ &sndev->mmio_self_ctrl->req_id_table[0]);
+
+ // Host Bridge Requester ID
+ iowrite32(req_id << 16 | NTB_CTRL_REQ_ID_EN,
+ &sndev->mmio_self_ctrl->req_id_table[1]);
+
+ rc = switchtec_ntb_part_op(sndev, sndev->mmio_self_ctrl,
+ NTB_CTRL_PART_OP_CFG,
+ NTB_CTRL_PART_STATUS_NORMAL);
+
+ if (rc == -EIO) {
+ error = ioread32(&sndev->mmio_self_ctrl->req_id_error);
+ dev_err(&sndev->stdev->dev,
+ "Error setting up the requester ID table: %08x",
+ error);
+ }
+
+ return rc;
+}
+
+static void switchtec_ntb_init_shared(struct switchtec_ntb *sndev)
+{
+ int i;
+
+ memset(sndev->self_shared, 0, LUT_SIZE);
+ sndev->self_shared->magic = SWITCHTEC_NTB_MAGIC;
+ sndev->self_shared->partition_id = sndev->stdev->partition;
+ sndev->self_shared->nr_direct_mw = sndev->nr_direct_mw;
+ sndev->self_shared->nr_lut_mw = use_lut_mws ? sndev->nr_lut_mw - 1 : 0;
+
+ for (i = 0; i < sndev->nr_direct_mw; i++) {
+ int bar = sndev->direct_mw_to_bar[i];
+ resource_size_t sz = pci_resource_len(sndev->stdev->pdev, bar);
+
+ if (i == 0)
+ sz = min_t(resource_size_t, sz,
+ LUT_SIZE * sndev->nr_lut_mw);
+
+ sndev->self_shared->mw_sizes[i] = sz;
+ }
+
+ for (i = 0; i < sndev->self_shared->nr_lut_mw; i++) {
+ int idx = sndev->nr_direct_mw + i;
+
+ sndev->self_shared->mw_sizes[idx] = LUT_SIZE;
+ }
+}
+
+static int switchtec_ntb_init_shared_mw(struct switchtec_ntb *sndev)
+{
+ struct ntb_ctrl_regs __iomem *ctl = sndev->mmio_peer_ctrl;
+ int bar = sndev->direct_mw_to_bar[0];
+ u32 ctl_val;
+ int rc;
+
+ sndev->self_shared = dma_zalloc_coherent(&sndev->stdev->pdev->dev,
+ LUT_SIZE,
+ &sndev->self_shared_dma,
+ GFP_KERNEL);
+ if (!sndev->self_shared) {
+ dev_err(&sndev->stdev->dev,
+ "unable to allocate memory for shared mw");
+ return -ENOMEM;
+ }
+
+ switchtec_ntb_init_shared(sndev);
+
+ rc = switchtec_ntb_part_op(sndev, ctl, NTB_CTRL_PART_OP_LOCK,
+ NTB_CTRL_PART_STATUS_LOCKED);
+ if (rc)
+ goto unalloc_and_exit;
+
+ ctl_val = ioread32(&ctl->bar_entry[bar].ctl);
+ ctl_val |= NTB_CTRL_BAR_LUT_WIN_EN;
+ ctl_val &= 0xFF;
+ ctl_val |= ilog2(LUT_SIZE) << 8;
+ ctl_val |= (sndev->nr_lut_mw - 1) << 14;
+ iowrite32(ctl_val, &ctl->bar_entry[bar].ctl);
+
+ iowrite64((NTB_CTRL_LUT_EN | (sndev->self_partition << 1) |
+ sndev->self_shared_dma),
+ &ctl->lut_entry[0]);
+
+ rc = switchtec_ntb_part_op(sndev, ctl, NTB_CTRL_PART_OP_CFG,
+ NTB_CTRL_PART_STATUS_NORMAL);
+ if (rc) {
+ u32 bar_error, lut_error;
+
+ bar_error = ioread32(&ctl->bar_error);
+ lut_error = ioread32(&ctl->lut_error);
+ dev_err(&sndev->stdev->dev,
+ "Error setting up shared MW: %08x / %08x",
+ bar_error, lut_error);
+ goto unalloc_and_exit;
+ }
+
+ sndev->peer_shared = pci_iomap(sndev->stdev->pdev, bar, LUT_SIZE);
+ if (!sndev->peer_shared) {
+ rc = -ENOMEM;
+ goto unalloc_and_exit;
+ }
+
+ dev_dbg(&sndev->stdev->dev, "Shared MW Ready");
+ return 0;
+
+unalloc_and_exit:
+ dma_free_coherent(&sndev->stdev->pdev->dev, LUT_SIZE,
+ sndev->self_shared, sndev->self_shared_dma);
+
+ return rc;
+}
+
+static void switchtec_ntb_deinit_shared_mw(struct switchtec_ntb *sndev)
+{
+ if (sndev->peer_shared)
+ pci_iounmap(sndev->stdev->pdev, sndev->peer_shared);
+
+ if (sndev->self_shared)
+ dma_free_coherent(&sndev->stdev->pdev->dev, LUT_SIZE,
+ sndev->self_shared,
+ sndev->self_shared_dma);
+}
+
static int switchtec_ntb_add(struct device *dev,
struct class_interface *class_intf)
{
struct switchtec_dev *stdev = to_stdev(dev);
struct switchtec_ntb *sndev;
+ int rc;
stdev->sndev = NULL;
if (stdev->pdev->class != MICROSEMI_NTB_CLASSCODE)
return -ENODEV;
+ if (stdev->partition_count != 2)
+ dev_warn(dev, "ntb driver only supports 2 partitions");
+
sndev = kzalloc_node(sizeof(*sndev), GFP_KERNEL, dev_to_node(dev));
if (!sndev)
return -ENOMEM;
sndev->stdev = stdev;
+ switchtec_ntb_init_sndev(sndev);
+ switchtec_ntb_init_mw(sndev);
+
+ rc = switchtec_ntb_init_req_id_table(sndev);
+ if (rc)
+ goto free_and_exit;
+
+ rc = switchtec_ntb_init_shared_mw(sndev);
+ if (rc)
+ goto free_and_exit;
+
stdev->sndev = sndev;
dev_info(dev, "NTB device registered");
return 0;
+
+free_and_exit:
+ kfree(sndev);
+ dev_err(dev, "failed to register ntb device: %d", rc);
+ return rc;
}
void switchtec_ntb_remove(struct device *dev,
@@ -58,6 +395,7 @@ void switchtec_ntb_remove(struct device *dev,
return;
stdev->sndev = NULL;
+ switchtec_ntb_deinit_shared_mw(sndev);
kfree(sndev);
dev_info(dev, "ntb device unregistered");
}
--
2.11.0
[toc] | [prev] | [next] | [standalone]
| From | Logan Gunthorpe <logang@deltatee.com> |
|---|---|
| Date | 2017-06-29 05:30 +0200 |
| Subject | [PATCH 01/16] switchtec: move structure definitions into a common header |
| Message-ID | <tXyjx-2hu-25@gated-at.bofh.it> |
| In reply to | #1677347 |
Create the switchtec.h header in include/linux with hardware defines
and the switchtec_dev structure moved directly from switchtec.c.
This is a prep patch for created an NTB driver for switchtec.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Stephen Bates <sbates@raithlin.com>
Reviewed-by: Kurt Schwemmer <kurt.schwemmer@microsemi.com>
Acked-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
MAINTAINERS | 1 +
drivers/pci/switch/switchtec.c | 251 +-------------------------------------
include/linux/switchtec.h | 270 +++++++++++++++++++++++++++++++++++++++++
3 files changed, 272 insertions(+), 250 deletions(-)
create mode 100644 include/linux/switchtec.h
diff --git a/MAINTAINERS b/MAINTAINERS
index b0b3b625e4ad..de6924290f73 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -9847,6 +9847,7 @@ F: Documentation/switchtec.txt
F: Documentation/ABI/testing/sysfs-class-switchtec
F: drivers/pci/switch/switchtec*
F: include/uapi/linux/switchtec_ioctl.h
+F: include/linux/switchtec.h
PCI DRIVER FOR NVIDIA TEGRA
M: Thierry Reding <thierry.reding@gmail.com>
diff --git a/drivers/pci/switch/switchtec.c b/drivers/pci/switch/switchtec.c
index f6a63406c76e..a27b2bacad5a 100644
--- a/drivers/pci/switch/switchtec.c
+++ b/drivers/pci/switch/switchtec.c
@@ -13,6 +13,7 @@
*
*/
+#include <linux/switchtec.h>
#include <linux/switchtec_ioctl.h>
#include <linux/interrupt.h>
@@ -20,8 +21,6 @@
#include <linux/fs.h>
#include <linux/uaccess.h>
#include <linux/poll.h>
-#include <linux/pci.h>
-#include <linux/cdev.h>
#include <linux/wait.h>
MODULE_DESCRIPTION("Microsemi Switchtec(tm) PCIe Management Driver");
@@ -37,254 +36,6 @@ static dev_t switchtec_devt;
static struct class *switchtec_class;
static DEFINE_IDA(switchtec_minor_ida);
-#define MICROSEMI_VENDOR_ID 0x11f8
-#define MICROSEMI_NTB_CLASSCODE 0x068000
-#define MICROSEMI_MGMT_CLASSCODE 0x058000
-
-#define SWITCHTEC_MRPC_PAYLOAD_SIZE 1024
-#define SWITCHTEC_MAX_PFF_CSR 48
-
-#define SWITCHTEC_EVENT_OCCURRED BIT(0)
-#define SWITCHTEC_EVENT_CLEAR BIT(0)
-#define SWITCHTEC_EVENT_EN_LOG BIT(1)
-#define SWITCHTEC_EVENT_EN_CLI BIT(2)
-#define SWITCHTEC_EVENT_EN_IRQ BIT(3)
-#define SWITCHTEC_EVENT_FATAL BIT(4)
-
-enum {
- SWITCHTEC_GAS_MRPC_OFFSET = 0x0000,
- SWITCHTEC_GAS_TOP_CFG_OFFSET = 0x1000,
- SWITCHTEC_GAS_SW_EVENT_OFFSET = 0x1800,
- SWITCHTEC_GAS_SYS_INFO_OFFSET = 0x2000,
- SWITCHTEC_GAS_FLASH_INFO_OFFSET = 0x2200,
- SWITCHTEC_GAS_PART_CFG_OFFSET = 0x4000,
- SWITCHTEC_GAS_NTB_OFFSET = 0x10000,
- SWITCHTEC_GAS_PFF_CSR_OFFSET = 0x134000,
-};
-
-struct mrpc_regs {
- u8 input_data[SWITCHTEC_MRPC_PAYLOAD_SIZE];
- u8 output_data[SWITCHTEC_MRPC_PAYLOAD_SIZE];
- u32 cmd;
- u32 status;
- u32 ret_value;
-} __packed;
-
-enum mrpc_status {
- SWITCHTEC_MRPC_STATUS_INPROGRESS = 1,
- SWITCHTEC_MRPC_STATUS_DONE = 2,
- SWITCHTEC_MRPC_STATUS_ERROR = 0xFF,
- SWITCHTEC_MRPC_STATUS_INTERRUPTED = 0x100,
-};
-
-struct sw_event_regs {
- u64 event_report_ctrl;
- u64 reserved1;
- u64 part_event_bitmap;
- u64 reserved2;
- u32 global_summary;
- u32 reserved3[3];
- u32 stack_error_event_hdr;
- u32 stack_error_event_data;
- u32 reserved4[4];
- u32 ppu_error_event_hdr;
- u32 ppu_error_event_data;
- u32 reserved5[4];
- u32 isp_error_event_hdr;
- u32 isp_error_event_data;
- u32 reserved6[4];
- u32 sys_reset_event_hdr;
- u32 reserved7[5];
- u32 fw_exception_hdr;
- u32 reserved8[5];
- u32 fw_nmi_hdr;
- u32 reserved9[5];
- u32 fw_non_fatal_hdr;
- u32 reserved10[5];
- u32 fw_fatal_hdr;
- u32 reserved11[5];
- u32 twi_mrpc_comp_hdr;
- u32 twi_mrpc_comp_data;
- u32 reserved12[4];
- u32 twi_mrpc_comp_async_hdr;
- u32 twi_mrpc_comp_async_data;
- u32 reserved13[4];
- u32 cli_mrpc_comp_hdr;
- u32 cli_mrpc_comp_data;
- u32 reserved14[4];
- u32 cli_mrpc_comp_async_hdr;
- u32 cli_mrpc_comp_async_data;
- u32 reserved15[4];
- u32 gpio_interrupt_hdr;
- u32 gpio_interrupt_data;
- u32 reserved16[4];
-} __packed;
-
-struct sys_info_regs {
- u32 device_id;
- u32 device_version;
- u32 firmware_version;
- u32 reserved1;
- u32 vendor_table_revision;
- u32 table_format_version;
- u32 partition_id;
- u32 cfg_file_fmt_version;
- u32 reserved2[58];
- char vendor_id[8];
- char product_id[16];
- char product_revision[4];
- char component_vendor[8];
- u16 component_id;
- u8 component_revision;
-} __packed;
-
-struct flash_info_regs {
- u32 flash_part_map_upd_idx;
-
- struct active_partition_info {
- u32 address;
- u32 build_version;
- u32 build_string;
- } active_img;
-
- struct active_partition_info active_cfg;
- struct active_partition_info inactive_img;
- struct active_partition_info inactive_cfg;
-
- u32 flash_length;
-
- struct partition_info {
- u32 address;
- u32 length;
- } cfg0;
-
- struct partition_info cfg1;
- struct partition_info img0;
- struct partition_info img1;
- struct partition_info nvlog;
- struct partition_info vendor[8];
-};
-
-struct ntb_info_regs {
- u8 partition_count;
- u8 partition_id;
- u16 reserved1;
- u64 ep_map;
- u16 requester_id;
-} __packed;
-
-struct part_cfg_regs {
- u32 status;
- u32 state;
- u32 port_cnt;
- u32 usp_port_mode;
- u32 usp_pff_inst_id;
- u32 vep_pff_inst_id;
- u32 dsp_pff_inst_id[47];
- u32 reserved1[11];
- u16 vep_vector_number;
- u16 usp_vector_number;
- u32 port_event_bitmap;
- u32 reserved2[3];
- u32 part_event_summary;
- u32 reserved3[3];
- u32 part_reset_hdr;
- u32 part_reset_data[5];
- u32 mrpc_comp_hdr;
- u32 mrpc_comp_data[5];
- u32 mrpc_comp_async_hdr;
- u32 mrpc_comp_async_data[5];
- u32 dyn_binding_hdr;
- u32 dyn_binding_data[5];
- u32 reserved4[159];
-} __packed;
-
-enum {
- SWITCHTEC_PART_CFG_EVENT_RESET = 1 << 0,
- SWITCHTEC_PART_CFG_EVENT_MRPC_CMP = 1 << 1,
- SWITCHTEC_PART_CFG_EVENT_MRPC_ASYNC_CMP = 1 << 2,
- SWITCHTEC_PART_CFG_EVENT_DYN_PART_CMP = 1 << 3,
-};
-
-struct pff_csr_regs {
- u16 vendor_id;
- u16 device_id;
- u32 pci_cfg_header[15];
- u32 pci_cap_region[48];
- u32 pcie_cap_region[448];
- u32 indirect_gas_window[128];
- u32 indirect_gas_window_off;
- u32 reserved[127];
- u32 pff_event_summary;
- u32 reserved2[3];
- u32 aer_in_p2p_hdr;
- u32 aer_in_p2p_data[5];
- u32 aer_in_vep_hdr;
- u32 aer_in_vep_data[5];
- u32 dpc_hdr;
- u32 dpc_data[5];
- u32 cts_hdr;
- u32 cts_data[5];
- u32 reserved3[6];
- u32 hotplug_hdr;
- u32 hotplug_data[5];
- u32 ier_hdr;
- u32 ier_data[5];
- u32 threshold_hdr;
- u32 threshold_data[5];
- u32 power_mgmt_hdr;
- u32 power_mgmt_data[5];
- u32 tlp_throttling_hdr;
- u32 tlp_throttling_data[5];
- u32 force_speed_hdr;
- u32 force_speed_data[5];
- u32 credit_timeout_hdr;
- u32 credit_timeout_data[5];
- u32 link_state_hdr;
- u32 link_state_data[5];
- u32 reserved4[174];
-} __packed;
-
-struct switchtec_dev {
- struct pci_dev *pdev;
- struct device dev;
- struct cdev cdev;
-
- int partition;
- int partition_count;
- int pff_csr_count;
- char pff_local[SWITCHTEC_MAX_PFF_CSR];
-
- void __iomem *mmio;
- struct mrpc_regs __iomem *mmio_mrpc;
- struct sw_event_regs __iomem *mmio_sw_event;
- struct sys_info_regs __iomem *mmio_sys_info;
- struct flash_info_regs __iomem *mmio_flash_info;
- struct ntb_info_regs __iomem *mmio_ntb;
- struct part_cfg_regs __iomem *mmio_part_cfg;
- struct part_cfg_regs __iomem *mmio_part_cfg_all;
- struct pff_csr_regs __iomem *mmio_pff_csr;
-
- /*
- * The mrpc mutex must be held when accessing the other
- * mrpc_ fields, alive flag and stuser->state field
- */
- struct mutex mrpc_mutex;
- struct list_head mrpc_queue;
- int mrpc_busy;
- struct work_struct mrpc_work;
- struct delayed_work mrpc_timeout;
- bool alive;
-
- wait_queue_head_t event_wq;
- atomic_t event_cnt;
-};
-
-static struct switchtec_dev *to_stdev(struct device *dev)
-{
- return container_of(dev, struct switchtec_dev, dev);
-}
-
enum mrpc_state {
MRPC_IDLE = 0,
MRPC_QUEUED,
diff --git a/include/linux/switchtec.h b/include/linux/switchtec.h
new file mode 100644
index 000000000000..508cda78a430
--- /dev/null
+++ b/include/linux/switchtec.h
@@ -0,0 +1,270 @@
+/*
+ * Microsemi Switchtec PCIe Driver
+ * Copyright (c) 2017, Microsemi Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ */
+
+#ifndef _SWITCHTEC_H
+#define _SWITCHTEC_H
+
+#include <linux/pci.h>
+#include <linux/cdev.h>
+
+#define MICROSEMI_VENDOR_ID 0x11f8
+#define MICROSEMI_NTB_CLASSCODE 0x068000
+#define MICROSEMI_MGMT_CLASSCODE 0x058000
+
+#define SWITCHTEC_MRPC_PAYLOAD_SIZE 1024
+#define SWITCHTEC_MAX_PFF_CSR 48
+
+#define SWITCHTEC_EVENT_OCCURRED BIT(0)
+#define SWITCHTEC_EVENT_CLEAR BIT(0)
+#define SWITCHTEC_EVENT_EN_LOG BIT(1)
+#define SWITCHTEC_EVENT_EN_CLI BIT(2)
+#define SWITCHTEC_EVENT_EN_IRQ BIT(3)
+#define SWITCHTEC_EVENT_FATAL BIT(4)
+
+enum {
+ SWITCHTEC_GAS_MRPC_OFFSET = 0x0000,
+ SWITCHTEC_GAS_TOP_CFG_OFFSET = 0x1000,
+ SWITCHTEC_GAS_SW_EVENT_OFFSET = 0x1800,
+ SWITCHTEC_GAS_SYS_INFO_OFFSET = 0x2000,
+ SWITCHTEC_GAS_FLASH_INFO_OFFSET = 0x2200,
+ SWITCHTEC_GAS_PART_CFG_OFFSET = 0x4000,
+ SWITCHTEC_GAS_NTB_OFFSET = 0x10000,
+ SWITCHTEC_GAS_PFF_CSR_OFFSET = 0x134000,
+};
+
+struct mrpc_regs {
+ u8 input_data[SWITCHTEC_MRPC_PAYLOAD_SIZE];
+ u8 output_data[SWITCHTEC_MRPC_PAYLOAD_SIZE];
+ u32 cmd;
+ u32 status;
+ u32 ret_value;
+} __packed;
+
+enum mrpc_status {
+ SWITCHTEC_MRPC_STATUS_INPROGRESS = 1,
+ SWITCHTEC_MRPC_STATUS_DONE = 2,
+ SWITCHTEC_MRPC_STATUS_ERROR = 0xFF,
+ SWITCHTEC_MRPC_STATUS_INTERRUPTED = 0x100,
+};
+
+struct sw_event_regs {
+ u64 event_report_ctrl;
+ u64 reserved1;
+ u64 part_event_bitmap;
+ u64 reserved2;
+ u32 global_summary;
+ u32 reserved3[3];
+ u32 stack_error_event_hdr;
+ u32 stack_error_event_data;
+ u32 reserved4[4];
+ u32 ppu_error_event_hdr;
+ u32 ppu_error_event_data;
+ u32 reserved5[4];
+ u32 isp_error_event_hdr;
+ u32 isp_error_event_data;
+ u32 reserved6[4];
+ u32 sys_reset_event_hdr;
+ u32 reserved7[5];
+ u32 fw_exception_hdr;
+ u32 reserved8[5];
+ u32 fw_nmi_hdr;
+ u32 reserved9[5];
+ u32 fw_non_fatal_hdr;
+ u32 reserved10[5];
+ u32 fw_fatal_hdr;
+ u32 reserved11[5];
+ u32 twi_mrpc_comp_hdr;
+ u32 twi_mrpc_comp_data;
+ u32 reserved12[4];
+ u32 twi_mrpc_comp_async_hdr;
+ u32 twi_mrpc_comp_async_data;
+ u32 reserved13[4];
+ u32 cli_mrpc_comp_hdr;
+ u32 cli_mrpc_comp_data;
+ u32 reserved14[4];
+ u32 cli_mrpc_comp_async_hdr;
+ u32 cli_mrpc_comp_async_data;
+ u32 reserved15[4];
+ u32 gpio_interrupt_hdr;
+ u32 gpio_interrupt_data;
+ u32 reserved16[4];
+} __packed;
+
+struct sys_info_regs {
+ u32 device_id;
+ u32 device_version;
+ u32 firmware_version;
+ u32 reserved1;
+ u32 vendor_table_revision;
+ u32 table_format_version;
+ u32 partition_id;
+ u32 cfg_file_fmt_version;
+ u32 reserved2[58];
+ char vendor_id[8];
+ char product_id[16];
+ char product_revision[4];
+ char component_vendor[8];
+ u16 component_id;
+ u8 component_revision;
+} __packed;
+
+struct flash_info_regs {
+ u32 flash_part_map_upd_idx;
+
+ struct active_partition_info {
+ u32 address;
+ u32 build_version;
+ u32 build_string;
+ } active_img;
+
+ struct active_partition_info active_cfg;
+ struct active_partition_info inactive_img;
+ struct active_partition_info inactive_cfg;
+
+ u32 flash_length;
+
+ struct partition_info {
+ u32 address;
+ u32 length;
+ } cfg0;
+
+ struct partition_info cfg1;
+ struct partition_info img0;
+ struct partition_info img1;
+ struct partition_info nvlog;
+ struct partition_info vendor[8];
+};
+
+struct ntb_info_regs {
+ u8 partition_count;
+ u8 partition_id;
+ u16 reserved1;
+ u64 ep_map;
+ u16 requester_id;
+} __packed;
+
+struct part_cfg_regs {
+ u32 status;
+ u32 state;
+ u32 port_cnt;
+ u32 usp_port_mode;
+ u32 usp_pff_inst_id;
+ u32 vep_pff_inst_id;
+ u32 dsp_pff_inst_id[47];
+ u32 reserved1[11];
+ u16 vep_vector_number;
+ u16 usp_vector_number;
+ u32 port_event_bitmap;
+ u32 reserved2[3];
+ u32 part_event_summary;
+ u32 reserved3[3];
+ u32 part_reset_hdr;
+ u32 part_reset_data[5];
+ u32 mrpc_comp_hdr;
+ u32 mrpc_comp_data[5];
+ u32 mrpc_comp_async_hdr;
+ u32 mrpc_comp_async_data[5];
+ u32 dyn_binding_hdr;
+ u32 dyn_binding_data[5];
+ u32 reserved4[159];
+} __packed;
+
+enum {
+ SWITCHTEC_PART_CFG_EVENT_RESET = 1 << 0,
+ SWITCHTEC_PART_CFG_EVENT_MRPC_CMP = 1 << 1,
+ SWITCHTEC_PART_CFG_EVENT_MRPC_ASYNC_CMP = 1 << 2,
+ SWITCHTEC_PART_CFG_EVENT_DYN_PART_CMP = 1 << 3,
+};
+
+struct pff_csr_regs {
+ u16 vendor_id;
+ u16 device_id;
+ u32 pci_cfg_header[15];
+ u32 pci_cap_region[48];
+ u32 pcie_cap_region[448];
+ u32 indirect_gas_window[128];
+ u32 indirect_gas_window_off;
+ u32 reserved[127];
+ u32 pff_event_summary;
+ u32 reserved2[3];
+ u32 aer_in_p2p_hdr;
+ u32 aer_in_p2p_data[5];
+ u32 aer_in_vep_hdr;
+ u32 aer_in_vep_data[5];
+ u32 dpc_hdr;
+ u32 dpc_data[5];
+ u32 cts_hdr;
+ u32 cts_data[5];
+ u32 reserved3[6];
+ u32 hotplug_hdr;
+ u32 hotplug_data[5];
+ u32 ier_hdr;
+ u32 ier_data[5];
+ u32 threshold_hdr;
+ u32 threshold_data[5];
+ u32 power_mgmt_hdr;
+ u32 power_mgmt_data[5];
+ u32 tlp_throttling_hdr;
+ u32 tlp_throttling_data[5];
+ u32 force_speed_hdr;
+ u32 force_speed_data[5];
+ u32 credit_timeout_hdr;
+ u32 credit_timeout_data[5];
+ u32 link_state_hdr;
+ u32 link_state_data[5];
+ u32 reserved4[174];
+} __packed;
+
+struct switchtec_dev {
+ struct pci_dev *pdev;
+ struct device dev;
+ struct cdev cdev;
+
+ int partition;
+ int partition_count;
+ int pff_csr_count;
+ char pff_local[SWITCHTEC_MAX_PFF_CSR];
+
+ void __iomem *mmio;
+ struct mrpc_regs __iomem *mmio_mrpc;
+ struct sw_event_regs __iomem *mmio_sw_event;
+ struct sys_info_regs __iomem *mmio_sys_info;
+ struct flash_info_regs __iomem *mmio_flash_info;
+ struct ntb_info_regs __iomem *mmio_ntb;
+ struct part_cfg_regs __iomem *mmio_part_cfg;
+ struct part_cfg_regs __iomem *mmio_part_cfg_all;
+ struct pff_csr_regs __iomem *mmio_pff_csr;
+
+ /*
+ * The mrpc mutex must be held when accessing the other
+ * mrpc_ fields, alive flag and stuser->state field
+ */
+ struct mutex mrpc_mutex;
+ struct list_head mrpc_queue;
+ int mrpc_busy;
+ struct work_struct mrpc_work;
+ struct delayed_work mrpc_timeout;
+ bool alive;
+
+ wait_queue_head_t event_wq;
+ atomic_t event_cnt;
+};
+
+static inline struct switchtec_dev *to_stdev(struct device *dev)
+{
+ return container_of(dev, struct switchtec_dev, dev);
+}
+
+#endif
--
2.11.0
[toc] | [prev] | [next] | [standalone]
| From | Logan Gunthorpe <logang@deltatee.com> |
|---|---|
| Date | 2017-06-29 05:40 +0200 |
| Subject | [PATCH 04/16] switchtec: add link event notifier callback |
| Message-ID | <tXytb-2ks-1@gated-at.bofh.it> |
| In reply to | #1677347 |
In order for the switchtec NTB code to handle link change events we
create a notifier callback in the switchtec code which gets called
whenever an appropriate event interrupt occurs.
In order to preserve userspace's ability to follow these events,
we compare the event count with a stored copy from last time we
checked.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Stephen Bates <sbates@raithlin.com>
Reviewed-by: Kurt Schwemmer <kurt.schwemmer@microsemi.com>
---
drivers/pci/switch/switchtec.c | 51 ++++++++++++++++++++++++++++++++++++++++++
include/linux/switchtec.h | 4 ++++
2 files changed, 55 insertions(+)
diff --git a/drivers/pci/switch/switchtec.c b/drivers/pci/switch/switchtec.c
index fc139ee3fa73..8fb01d7a793c 100644
--- a/drivers/pci/switch/switchtec.c
+++ b/drivers/pci/switch/switchtec.c
@@ -969,6 +969,49 @@ static const struct file_operations switchtec_fops = {
.compat_ioctl = switchtec_dev_ioctl,
};
+static void link_event_work(struct work_struct *work)
+{
+ struct switchtec_dev *stdev;
+
+ stdev = container_of(work, struct switchtec_dev, link_event_work);
+
+ if (stdev->link_notifier)
+ stdev->link_notifier(stdev);
+}
+
+static void check_link_state_events(struct switchtec_dev *stdev)
+{
+ int idx;
+ u32 reg;
+ int count;
+ int occurred = 0;
+
+ for (idx = 0; idx < stdev->pff_csr_count; idx++) {
+ reg = ioread32(&stdev->mmio_pff_csr[idx].link_state_hdr);
+ dev_dbg(&stdev->dev, "link_state: %d->%08x\n", idx, reg);
+ count = (reg >> 5) & 0xFF;
+
+ if (count != stdev->link_event_count[idx]) {
+ occurred = 1;
+ stdev->link_event_count[idx] = count;
+ }
+ }
+
+ if (occurred)
+ schedule_work(&stdev->link_event_work);
+}
+
+static void enable_link_state_events(struct switchtec_dev *stdev)
+{
+ int idx;
+
+ for (idx = 0; idx < stdev->pff_csr_count; idx++) {
+ iowrite32(SWITCHTEC_EVENT_CLEAR |
+ SWITCHTEC_EVENT_EN_IRQ,
+ &stdev->mmio_pff_csr[idx].link_state_hdr);
+ }
+}
+
static void stdev_release(struct device *dev)
{
struct switchtec_dev *stdev = to_stdev(dev);
@@ -1021,6 +1064,7 @@ static struct switchtec_dev *stdev_create(struct pci_dev *pdev)
stdev->mrpc_busy = 0;
INIT_WORK(&stdev->mrpc_work, mrpc_event_work);
INIT_DELAYED_WORK(&stdev->mrpc_timeout, mrpc_timeout_work);
+ INIT_WORK(&stdev->link_event_work, link_event_work);
init_waitqueue_head(&stdev->event_wq);
atomic_set(&stdev->event_cnt, 0);
@@ -1064,6 +1108,9 @@ static int mask_event(struct switchtec_dev *stdev, int eid, int idx)
if (!(hdr & SWITCHTEC_EVENT_OCCURRED && hdr & SWITCHTEC_EVENT_EN_IRQ))
return 0;
+ if (eid == SWITCHTEC_IOCTL_EVENT_LINK_STATE)
+ return 0;
+
dev_dbg(&stdev->dev, "%s: %d %d %x\n", __func__, eid, idx, hdr);
hdr &= ~(SWITCHTEC_EVENT_EN_IRQ | SWITCHTEC_EVENT_OCCURRED);
iowrite32(hdr, hdr_reg);
@@ -1083,6 +1130,7 @@ static int mask_all_events(struct switchtec_dev *stdev, int eid)
for (idx = 0; idx < stdev->pff_csr_count; idx++) {
if (!stdev->pff_local[idx])
continue;
+
count += mask_event(stdev, eid, idx);
}
} else {
@@ -1107,6 +1155,8 @@ static irqreturn_t switchtec_event_isr(int irq, void *dev)
iowrite32(reg, &stdev->mmio_part_cfg->mrpc_comp_hdr);
}
+ check_link_state_events(stdev);
+
for (eid = 0; eid < SWITCHTEC_IOCTL_MAX_EVENTS; eid++)
event_count += mask_all_events(stdev, eid);
@@ -1233,6 +1283,7 @@ static int switchtec_pci_probe(struct pci_dev *pdev,
iowrite32(SWITCHTEC_EVENT_CLEAR |
SWITCHTEC_EVENT_EN_IRQ,
&stdev->mmio_part_cfg->mrpc_comp_hdr);
+ enable_link_state_events(stdev);
rc = cdev_device_add(&stdev->cdev, &stdev->dev);
if (rc)
diff --git a/include/linux/switchtec.h b/include/linux/switchtec.h
index a75f3d7e6876..6a056ffbc439 100644
--- a/include/linux/switchtec.h
+++ b/include/linux/switchtec.h
@@ -344,6 +344,10 @@ struct switchtec_dev {
wait_queue_head_t event_wq;
atomic_t event_cnt;
+
+ struct work_struct link_event_work;
+ void (*link_notifier)(struct switchtec_dev *stdev);
+ u8 link_event_count[SWITCHTEC_MAX_PFF_CSR];
};
static inline struct switchtec_dev *to_stdev(struct device *dev)
--
2.11.0
[toc] | [prev] | [next] | [standalone]
| From | Logan Gunthorpe <logang@deltatee.com> |
|---|---|
| Date | 2017-06-29 05:40 +0200 |
| Subject | [PATCH 05/16] ntb: ensure ntb_mw_get_align is only called when the link is up |
| Message-ID | <tXytb-2ks-5@gated-at.bofh.it> |
| In reply to | #1677347 |
With switchtec hardware it's impossible to get the alignment parameters
for a peer's memory window until the peer's driver has configured it's
windows. Strictly speaking, the link doesn't have to be up for this,
but the link being up is the only way the client can tell that
the otherside has been configured.
This patch converts ntb_transport and ntb_perf to use this function after
the link goes up. This simplifies these clients slightly because they
no longer have to store the alignment parameters. It also tweaks
ntb_tool so that peer_mw_trans will print zero if it is run before
the link goes up.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
---
drivers/ntb/ntb_transport.c | 20 ++++++++++----------
drivers/ntb/test/ntb_perf.c | 18 +++++++++---------
drivers/ntb/test/ntb_tool.c | 6 +++---
3 files changed, 22 insertions(+), 22 deletions(-)
diff --git a/drivers/ntb/ntb_transport.c b/drivers/ntb/ntb_transport.c
index b29558ddfe95..7ed745aaa213 100644
--- a/drivers/ntb/ntb_transport.c
+++ b/drivers/ntb/ntb_transport.c
@@ -191,8 +191,6 @@ struct ntb_transport_qp {
struct ntb_transport_mw {
phys_addr_t phys_addr;
resource_size_t phys_size;
- resource_size_t xlat_align;
- resource_size_t xlat_align_size;
void __iomem *vbase;
size_t xlat_size;
size_t buff_size;
@@ -687,13 +685,20 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw,
struct ntb_transport_mw *mw = &nt->mw_vec[num_mw];
struct pci_dev *pdev = nt->ndev->pdev;
size_t xlat_size, buff_size;
+ resource_size_t xlat_align;
+ resource_size_t xlat_align_size;
int rc;
if (!size)
return -EINVAL;
- xlat_size = round_up(size, mw->xlat_align_size);
- buff_size = round_up(size, mw->xlat_align);
+ rc = ntb_mw_get_align(nt->ndev, PIDX, num_mw, &xlat_align,
+ &xlat_align_size, NULL);
+ if (rc)
+ return rc;
+
+ xlat_size = round_up(size, xlat_align_size);
+ buff_size = round_up(size, xlat_align);
/* No need to re-setup */
if (mw->xlat_size == xlat_size)
@@ -722,7 +727,7 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw,
* is a requirement of the hardware. It is recommended to setup CMA
* for BAR sizes equal or greater than 4MB.
*/
- if (!IS_ALIGNED(mw->dma_addr, mw->xlat_align)) {
+ if (!IS_ALIGNED(mw->dma_addr, xlat_align)) {
dev_err(&pdev->dev, "DMA memory %pad is not aligned\n",
&mw->dma_addr);
ntb_free_mw(nt, num_mw);
@@ -1106,11 +1111,6 @@ static int ntb_transport_probe(struct ntb_client *self, struct ntb_dev *ndev)
for (i = 0; i < mw_count; i++) {
mw = &nt->mw_vec[i];
- rc = ntb_mw_get_align(ndev, PIDX, i, &mw->xlat_align,
- &mw->xlat_align_size, NULL);
- if (rc)
- goto err1;
-
rc = ntb_peer_mw_get_addr(ndev, i, &mw->phys_addr,
&mw->phys_size);
if (rc)
diff --git a/drivers/ntb/test/ntb_perf.c b/drivers/ntb/test/ntb_perf.c
index 759f772fa00c..427112cf101a 100644
--- a/drivers/ntb/test/ntb_perf.c
+++ b/drivers/ntb/test/ntb_perf.c
@@ -108,8 +108,6 @@ MODULE_PARM_DESC(on_node, "Run threads only on NTB device node (default: true)")
struct perf_mw {
phys_addr_t phys_addr;
resource_size_t phys_size;
- resource_size_t xlat_align;
- resource_size_t xlat_align_size;
void __iomem *vbase;
size_t xlat_size;
size_t buf_size;
@@ -472,13 +470,20 @@ static int perf_set_mw(struct perf_ctx *perf, resource_size_t size)
{
struct perf_mw *mw = &perf->mw;
size_t xlat_size, buf_size;
+ resource_size_t xlat_align;
+ resource_size_t xlat_align_size;
int rc;
if (!size)
return -EINVAL;
- xlat_size = round_up(size, mw->xlat_align_size);
- buf_size = round_up(size, mw->xlat_align);
+ rc = ntb_mw_get_align(perf->ntb, PIDX, 0, &xlat_align,
+ &xlat_align_size, NULL);
+ if (rc)
+ return rc;
+
+ xlat_size = round_up(size, xlat_align_size);
+ buf_size = round_up(size, xlat_align);
if (mw->xlat_size == xlat_size)
return 0;
@@ -567,11 +572,6 @@ static int perf_setup_mw(struct ntb_dev *ntb, struct perf_ctx *perf)
mw = &perf->mw;
- rc = ntb_mw_get_align(ntb, PIDX, 0, &mw->xlat_align,
- &mw->xlat_align_size, NULL);
- if (rc)
- return rc;
-
rc = ntb_peer_mw_get_addr(ntb, 0, &mw->phys_addr, &mw->phys_size);
if (rc)
return rc;
diff --git a/drivers/ntb/test/ntb_tool.c b/drivers/ntb/test/ntb_tool.c
index a69815c45ce6..91526a986caa 100644
--- a/drivers/ntb/test/ntb_tool.c
+++ b/drivers/ntb/test/ntb_tool.c
@@ -753,9 +753,9 @@ static ssize_t tool_peer_mw_trans_read(struct file *filep,
phys_addr_t base;
resource_size_t mw_size;
- resource_size_t align_addr;
- resource_size_t align_size;
- resource_size_t max_size;
+ resource_size_t align_addr = 0;
+ resource_size_t align_size = 0;
+ resource_size_t max_size = 0;
buf_size = min_t(size_t, size, 512);
--
2.11.0
[toc] | [prev] | [next] | [standalone]
| From | "Allen Hubbe" <Allen.Hubbe@dell.com> |
|---|---|
| Date | 2017-06-29 20:20 +0200 |
| Subject | RE: [PATCH 05/16] ntb: ensure ntb_mw_get_align is only called when the link is up |
| Message-ID | <tXMcN-2zD-13@gated-at.bofh.it> |
| In reply to | #1677353 |
From: Logan Gunthorpe
> With switchtec hardware it's impossible to get the alignment parameters
> for a peer's memory window until the peer's driver has configured it's
> windows. Strictly speaking, the link doesn't have to be up for this,
> but the link being up is the only way the client can tell that
> the otherside has been configured.
>
> This patch converts ntb_transport and ntb_perf to use this function after
> the link goes up. This simplifies these clients slightly because they
> no longer have to store the alignment parameters. It also tweaks
> ntb_tool so that peer_mw_trans will print zero if it is run before
> the link goes up.
>
> Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Acked-by: Allen Hubbe <Allen.Hubbe@dell.com>
> ---
> drivers/ntb/ntb_transport.c | 20 ++++++++++----------
> drivers/ntb/test/ntb_perf.c | 18 +++++++++---------
> drivers/ntb/test/ntb_tool.c | 6 +++---
> 3 files changed, 22 insertions(+), 22 deletions(-)
>
> diff --git a/drivers/ntb/ntb_transport.c b/drivers/ntb/ntb_transport.c
> index b29558ddfe95..7ed745aaa213 100644
> --- a/drivers/ntb/ntb_transport.c
> +++ b/drivers/ntb/ntb_transport.c
> @@ -191,8 +191,6 @@ struct ntb_transport_qp {
> struct ntb_transport_mw {
> phys_addr_t phys_addr;
> resource_size_t phys_size;
> - resource_size_t xlat_align;
> - resource_size_t xlat_align_size;
> void __iomem *vbase;
> size_t xlat_size;
> size_t buff_size;
> @@ -687,13 +685,20 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw,
> struct ntb_transport_mw *mw = &nt->mw_vec[num_mw];
> struct pci_dev *pdev = nt->ndev->pdev;
> size_t xlat_size, buff_size;
> + resource_size_t xlat_align;
> + resource_size_t xlat_align_size;
> int rc;
>
> if (!size)
> return -EINVAL;
>
> - xlat_size = round_up(size, mw->xlat_align_size);
> - buff_size = round_up(size, mw->xlat_align);
> + rc = ntb_mw_get_align(nt->ndev, PIDX, num_mw, &xlat_align,
> + &xlat_align_size, NULL);
> + if (rc)
> + return rc;
> +
> + xlat_size = round_up(size, xlat_align_size);
> + buff_size = round_up(size, xlat_align);
>
> /* No need to re-setup */
> if (mw->xlat_size == xlat_size)
> @@ -722,7 +727,7 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw,
> * is a requirement of the hardware. It is recommended to setup CMA
> * for BAR sizes equal or greater than 4MB.
> */
> - if (!IS_ALIGNED(mw->dma_addr, mw->xlat_align)) {
> + if (!IS_ALIGNED(mw->dma_addr, xlat_align)) {
> dev_err(&pdev->dev, "DMA memory %pad is not aligned\n",
> &mw->dma_addr);
> ntb_free_mw(nt, num_mw);
> @@ -1106,11 +1111,6 @@ static int ntb_transport_probe(struct ntb_client *self, struct ntb_dev *ndev)
> for (i = 0; i < mw_count; i++) {
> mw = &nt->mw_vec[i];
>
> - rc = ntb_mw_get_align(ndev, PIDX, i, &mw->xlat_align,
> - &mw->xlat_align_size, NULL);
> - if (rc)
> - goto err1;
> -
> rc = ntb_peer_mw_get_addr(ndev, i, &mw->phys_addr,
> &mw->phys_size);
> if (rc)
> diff --git a/drivers/ntb/test/ntb_perf.c b/drivers/ntb/test/ntb_perf.c
> index 759f772fa00c..427112cf101a 100644
> --- a/drivers/ntb/test/ntb_perf.c
> +++ b/drivers/ntb/test/ntb_perf.c
> @@ -108,8 +108,6 @@ MODULE_PARM_DESC(on_node, "Run threads only on NTB device node (default: true)")
> struct perf_mw {
> phys_addr_t phys_addr;
> resource_size_t phys_size;
> - resource_size_t xlat_align;
> - resource_size_t xlat_align_size;
> void __iomem *vbase;
> size_t xlat_size;
> size_t buf_size;
> @@ -472,13 +470,20 @@ static int perf_set_mw(struct perf_ctx *perf, resource_size_t size)
> {
> struct perf_mw *mw = &perf->mw;
> size_t xlat_size, buf_size;
> + resource_size_t xlat_align;
> + resource_size_t xlat_align_size;
> int rc;
>
> if (!size)
> return -EINVAL;
>
> - xlat_size = round_up(size, mw->xlat_align_size);
> - buf_size = round_up(size, mw->xlat_align);
> + rc = ntb_mw_get_align(perf->ntb, PIDX, 0, &xlat_align,
> + &xlat_align_size, NULL);
> + if (rc)
> + return rc;
> +
> + xlat_size = round_up(size, xlat_align_size);
> + buf_size = round_up(size, xlat_align);
>
> if (mw->xlat_size == xlat_size)
> return 0;
> @@ -567,11 +572,6 @@ static int perf_setup_mw(struct ntb_dev *ntb, struct perf_ctx *perf)
>
> mw = &perf->mw;
>
> - rc = ntb_mw_get_align(ntb, PIDX, 0, &mw->xlat_align,
> - &mw->xlat_align_size, NULL);
> - if (rc)
> - return rc;
> -
> rc = ntb_peer_mw_get_addr(ntb, 0, &mw->phys_addr, &mw->phys_size);
> if (rc)
> return rc;
> diff --git a/drivers/ntb/test/ntb_tool.c b/drivers/ntb/test/ntb_tool.c
> index a69815c45ce6..91526a986caa 100644
> --- a/drivers/ntb/test/ntb_tool.c
> +++ b/drivers/ntb/test/ntb_tool.c
> @@ -753,9 +753,9 @@ static ssize_t tool_peer_mw_trans_read(struct file *filep,
>
> phys_addr_t base;
> resource_size_t mw_size;
> - resource_size_t align_addr;
> - resource_size_t align_size;
> - resource_size_t max_size;
> + resource_size_t align_addr = 0;
> + resource_size_t align_size = 0;
> + resource_size_t max_size = 0;
>
> buf_size = min_t(size_t, size, 512);
>
> --
> 2.11.0
[toc] | [prev] | [next] | [standalone]
| From | Logan Gunthorpe <logang@deltatee.com> |
|---|---|
| Date | 2017-06-29 05:40 +0200 |
| Subject | [PATCH 12/16] switchtec_ntb: add link management |
| Message-ID | <tXytb-2ks-9@gated-at.bofh.it> |
| In reply to | #1677347 |
switchtec_ntb checks for a link by looking at the shared memory
window. If the magic number is correct and the otherside indicates
their link is enabled then we take the link to be up.
Whenever we change our local link status we send a msg to the
otherside to check whether it's up and change their status.
The current status is maintained in a flag so ntb_is_link_up
can return quickly.
We utilize switchtec's link status notifier to also check link changes
when the switch notices a port changes state.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Stephen Bates <sbates@raithlin.com>
Reviewed-by: Kurt Schwemmer <kurt.schwemmer@microsemi.com>
---
drivers/ntb/hw/mscc/switchtec_ntb.c | 130 +++++++++++++++++++++++++++++++++++-
1 file changed, 129 insertions(+), 1 deletion(-)
diff --git a/drivers/ntb/hw/mscc/switchtec_ntb.c b/drivers/ntb/hw/mscc/switchtec_ntb.c
index 0587b2380bcc..8ef84d45bda6 100644
--- a/drivers/ntb/hw/mscc/switchtec_ntb.c
+++ b/drivers/ntb/hw/mscc/switchtec_ntb.c
@@ -64,6 +64,7 @@ static inline void _iowrite64(u64 val, void __iomem *mmio)
struct shared_mw {
u32 magic;
+ u32 link_sta;
u32 partition_id;
u16 nr_direct_mw;
u16 nr_lut_mw;
@@ -102,8 +103,17 @@ struct switchtec_ntb {
int nr_direct_mw;
int nr_lut_mw;
int direct_mw_to_bar[MAX_DIRECT_MW];
+
+ bool link_is_up;
+ enum ntb_speed link_speed;
+ enum ntb_width link_width;
};
+static struct switchtec_ntb *ntb_sndev(struct ntb_dev *ntb)
+{
+ return container_of(ntb, struct switchtec_ntb, ntb);
+}
+
static int switchtec_ntb_part_op(struct switchtec_ntb *sndev,
struct ntb_ctrl_regs __iomem *ctl,
u32 op, int wait_status)
@@ -161,6 +171,17 @@ static int switchtec_ntb_part_op(struct switchtec_ntb *sndev,
return -EIO;
}
+static int switchtec_ntb_send_msg(struct switchtec_ntb *sndev, int idx,
+ u32 val)
+{
+ if (idx < 0 || idx >= ARRAY_SIZE(sndev->mmio_self_dbmsg->omsg))
+ return -EINVAL;
+
+ iowrite32(val, &sndev->mmio_self_dbmsg->omsg[idx].msg);
+
+ return 0;
+}
+
static int switchtec_ntb_mw_count(struct ntb_dev *ntb, int pidx)
{
return 0;
@@ -192,22 +213,124 @@ static int switchtec_ntb_peer_mw_get_addr(struct ntb_dev *ntb, int idx,
return 0;
}
+static void switchtec_ntb_part_link_speed(struct switchtec_ntb *sndev,
+ int partition,
+ enum ntb_speed *speed,
+ enum ntb_width *width)
+{
+ struct switchtec_dev *stdev = sndev->stdev;
+
+ u32 pff = ioread32(&stdev->mmio_part_cfg[partition].vep_pff_inst_id);
+ u32 linksta = ioread32(&stdev->mmio_pff_csr[pff].pci_cap_region[13]);
+
+ if (speed)
+ *speed = (linksta >> 16) & 0xF;
+
+ if (width)
+ *width = (linksta >> 20) & 0x3F;
+}
+
+static void switchtec_ntb_set_link_speed(struct switchtec_ntb *sndev)
+{
+ enum ntb_speed self_speed, peer_speed;
+ enum ntb_width self_width, peer_width;
+
+ if (!sndev->link_is_up) {
+ sndev->link_speed = NTB_SPEED_NONE;
+ sndev->link_width = NTB_WIDTH_NONE;
+ return;
+ }
+
+ switchtec_ntb_part_link_speed(sndev, sndev->self_partition,
+ &self_speed, &self_width);
+ switchtec_ntb_part_link_speed(sndev, sndev->peer_partition,
+ &peer_speed, &peer_width);
+
+ sndev->link_speed = min(self_speed, peer_speed);
+ sndev->link_width = min(self_width, peer_width);
+}
+
+enum {
+ LINK_MESSAGE = 0,
+ MSG_LINK_UP = 1,
+ MSG_LINK_DOWN = 2,
+ MSG_CHECK_LINK = 3,
+};
+
+static void switchtec_ntb_check_link(struct switchtec_ntb *sndev)
+{
+ int link_sta;
+ int old = sndev->link_is_up;
+
+ link_sta = sndev->self_shared->link_sta;
+ if (link_sta) {
+ u64 peer = ioread64(&sndev->peer_shared->magic);
+
+ if ((peer & 0xFFFFFFFF) == SWITCHTEC_NTB_MAGIC)
+ link_sta = peer >> 32;
+ else
+ link_sta = 0;
+ }
+
+ sndev->link_is_up = link_sta;
+ switchtec_ntb_set_link_speed(sndev);
+
+ if (link_sta != old) {
+ switchtec_ntb_send_msg(sndev, LINK_MESSAGE, MSG_CHECK_LINK);
+ ntb_link_event(&sndev->ntb);
+ dev_info(&sndev->stdev->dev, "ntb link %s",
+ link_sta ? "up" : "down");
+ }
+}
+
+static void switchtec_ntb_link_notification(struct switchtec_dev *stdev)
+{
+ struct switchtec_ntb *sndev = stdev->sndev;
+
+ switchtec_ntb_check_link(sndev);
+}
+
static u64 switchtec_ntb_link_is_up(struct ntb_dev *ntb,
enum ntb_speed *speed,
enum ntb_width *width)
{
- return 0;
+ struct switchtec_ntb *sndev = ntb_sndev(ntb);
+
+ if (speed)
+ *speed = sndev->link_speed;
+ if (width)
+ *width = sndev->link_width;
+
+ return sndev->link_is_up;
}
static int switchtec_ntb_link_enable(struct ntb_dev *ntb,
enum ntb_speed max_speed,
enum ntb_width max_width)
{
+ struct switchtec_ntb *sndev = ntb_sndev(ntb);
+
+ dev_dbg(&sndev->stdev->dev, "enabling link");
+
+ sndev->self_shared->link_sta = 1;
+ switchtec_ntb_send_msg(sndev, LINK_MESSAGE, MSG_LINK_UP);
+
+ switchtec_ntb_check_link(sndev);
+
return 0;
}
static int switchtec_ntb_link_disable(struct ntb_dev *ntb)
{
+ struct switchtec_ntb *sndev = ntb_sndev(ntb);
+
+ dev_dbg(&sndev->stdev->dev, "disabling link");
+
+ sndev->self_shared->link_sta = 0;
+ switchtec_ntb_send_msg(sndev, LINK_MESSAGE, MSG_LINK_UP);
+
+ switchtec_ntb_check_link(sndev);
+
return 0;
}
@@ -555,6 +678,9 @@ static irqreturn_t switchtec_ntb_message_isr(int irq, void *dev)
dev_dbg(&sndev->stdev->dev, "message: %d %08x\n", i,
(u32)msg);
iowrite8(1, &sndev->mmio_self_dbmsg->imsg[i].status);
+
+ if (i == LINK_MESSAGE)
+ switchtec_ntb_check_link(sndev);
}
}
@@ -659,6 +785,7 @@ static int switchtec_ntb_add(struct device *dev,
goto deinit_and_exit;
stdev->sndev = sndev;
+ stdev->link_notifier = switchtec_ntb_link_notification;
dev_info(dev, "NTB device registered");
return 0;
@@ -682,6 +809,7 @@ void switchtec_ntb_remove(struct device *dev,
if (!sndev)
return;
+ stdev->link_notifier = NULL;
stdev->sndev = NULL;
ntb_unregister_device(&sndev->ntb);
switchtec_ntb_deinit_db_msg_irq(sndev);
--
2.11.0
[toc] | [prev] | [next] | [standalone]
| From | "Allen Hubbe" <Allen.Hubbe@dell.com> |
|---|---|
| Date | 2017-06-29 20:20 +0200 |
| Subject | RE: [PATCH 12/16] switchtec_ntb: add link management |
| Message-ID | <tXMcN-2zD-3@gated-at.bofh.it> |
| In reply to | #1677354 |
From: Logan Gunthorpe
> switchtec_ntb checks for a link by looking at the shared memory
> window. If the magic number is correct and the otherside indicates
> their link is enabled then we take the link to be up.
>
> Whenever we change our local link status we send a msg to the
> otherside to check whether it's up and change their status.
>
> The current status is maintained in a flag so ntb_is_link_up
> can return quickly.
>
> We utilize switchtec's link status notifier to also check link changes
> when the switch notices a port changes state.
>
> Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
> Reviewed-by: Stephen Bates <sbates@raithlin.com>
> Reviewed-by: Kurt Schwemmer <kurt.schwemmer@microsemi.com>
> ---
> drivers/ntb/hw/mscc/switchtec_ntb.c | 130 +++++++++++++++++++++++++++++++++++-
> 1 file changed, 129 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/ntb/hw/mscc/switchtec_ntb.c b/drivers/ntb/hw/mscc/switchtec_ntb.c
> index 0587b2380bcc..8ef84d45bda6 100644
> --- a/drivers/ntb/hw/mscc/switchtec_ntb.c
> +++ b/drivers/ntb/hw/mscc/switchtec_ntb.c
> @@ -64,6 +64,7 @@ static inline void _iowrite64(u64 val, void __iomem *mmio)
>
> struct shared_mw {
> u32 magic;
> + u32 link_sta;
> u32 partition_id;
> u16 nr_direct_mw;
> u16 nr_lut_mw;
> @@ -102,8 +103,17 @@ struct switchtec_ntb {
> int nr_direct_mw;
> int nr_lut_mw;
> int direct_mw_to_bar[MAX_DIRECT_MW];
> +
> + bool link_is_up;
> + enum ntb_speed link_speed;
> + enum ntb_width link_width;
> };
>
> +static struct switchtec_ntb *ntb_sndev(struct ntb_dev *ntb)
> +{
> + return container_of(ntb, struct switchtec_ntb, ntb);
> +}
> +
> static int switchtec_ntb_part_op(struct switchtec_ntb *sndev,
> struct ntb_ctrl_regs __iomem *ctl,
> u32 op, int wait_status)
> @@ -161,6 +171,17 @@ static int switchtec_ntb_part_op(struct switchtec_ntb *sndev,
> return -EIO;
> }
>
> +static int switchtec_ntb_send_msg(struct switchtec_ntb *sndev, int idx,
> + u32 val)
> +{
> + if (idx < 0 || idx >= ARRAY_SIZE(sndev->mmio_self_dbmsg->omsg))
> + return -EINVAL;
> +
> + iowrite32(val, &sndev->mmio_self_dbmsg->omsg[idx].msg);
> +
> + return 0;
> +}
> +
> static int switchtec_ntb_mw_count(struct ntb_dev *ntb, int pidx)
> {
> return 0;
> @@ -192,22 +213,124 @@ static int switchtec_ntb_peer_mw_get_addr(struct ntb_dev *ntb, int idx,
> return 0;
> }
>
> +static void switchtec_ntb_part_link_speed(struct switchtec_ntb *sndev,
> + int partition,
> + enum ntb_speed *speed,
> + enum ntb_width *width)
> +{
> + struct switchtec_dev *stdev = sndev->stdev;
> +
> + u32 pff = ioread32(&stdev->mmio_part_cfg[partition].vep_pff_inst_id);
> + u32 linksta = ioread32(&stdev->mmio_pff_csr[pff].pci_cap_region[13]);
> +
> + if (speed)
> + *speed = (linksta >> 16) & 0xF;
> +
> + if (width)
> + *width = (linksta >> 20) & 0x3F;
> +}
> +
> +static void switchtec_ntb_set_link_speed(struct switchtec_ntb *sndev)
> +{
> + enum ntb_speed self_speed, peer_speed;
> + enum ntb_width self_width, peer_width;
> +
> + if (!sndev->link_is_up) {
> + sndev->link_speed = NTB_SPEED_NONE;
> + sndev->link_width = NTB_WIDTH_NONE;
> + return;
> + }
> +
> + switchtec_ntb_part_link_speed(sndev, sndev->self_partition,
> + &self_speed, &self_width);
> + switchtec_ntb_part_link_speed(sndev, sndev->peer_partition,
> + &peer_speed, &peer_width);
Should we only set self_partition? I think each peer should be able to set preferred speed, and negotiate down. As written here, the last peer to set the speed overrides the setting on the peer, and even that is not atomic if they race.
> +
> + sndev->link_speed = min(self_speed, peer_speed);
> + sndev->link_width = min(self_width, peer_width);
> +}
> +
> +enum {
> + LINK_MESSAGE = 0,
> + MSG_LINK_UP = 1,
> + MSG_LINK_DOWN = 2,
> + MSG_CHECK_LINK = 3,
> +};
> +
> +static void switchtec_ntb_check_link(struct switchtec_ntb *sndev)
> +{
> + int link_sta;
> + int old = sndev->link_is_up;
> +
> + link_sta = sndev->self_shared->link_sta;
> + if (link_sta) {
> + u64 peer = ioread64(&sndev->peer_shared->magic);
> +
> + if ((peer & 0xFFFFFFFF) == SWITCHTEC_NTB_MAGIC)
> + link_sta = peer >> 32;
> + else
> + link_sta = 0;
> + }
> +
> + sndev->link_is_up = link_sta;
> + switchtec_ntb_set_link_speed(sndev);
> +
> + if (link_sta != old) {
> + switchtec_ntb_send_msg(sndev, LINK_MESSAGE, MSG_CHECK_LINK);
> + ntb_link_event(&sndev->ntb);
> + dev_info(&sndev->stdev->dev, "ntb link %s",
> + link_sta ? "up" : "down");
> + }
> +}
> +
> +static void switchtec_ntb_link_notification(struct switchtec_dev *stdev)
> +{
> + struct switchtec_ntb *sndev = stdev->sndev;
> +
> + switchtec_ntb_check_link(sndev);
> +}
> +
> static u64 switchtec_ntb_link_is_up(struct ntb_dev *ntb,
> enum ntb_speed *speed,
> enum ntb_width *width)
> {
> - return 0;
> + struct switchtec_ntb *sndev = ntb_sndev(ntb);
> +
> + if (speed)
> + *speed = sndev->link_speed;
> + if (width)
> + *width = sndev->link_width;
> +
> + return sndev->link_is_up;
> }
>
> static int switchtec_ntb_link_enable(struct ntb_dev *ntb,
> enum ntb_speed max_speed,
> enum ntb_width max_width)
> {
> + struct switchtec_ntb *sndev = ntb_sndev(ntb);
> +
> + dev_dbg(&sndev->stdev->dev, "enabling link");
> +
> + sndev->self_shared->link_sta = 1;
> + switchtec_ntb_send_msg(sndev, LINK_MESSAGE, MSG_LINK_UP);
> +
> + switchtec_ntb_check_link(sndev);
> +
> return 0;
> }
>
> static int switchtec_ntb_link_disable(struct ntb_dev *ntb)
> {
> + struct switchtec_ntb *sndev = ntb_sndev(ntb);
> +
> + dev_dbg(&sndev->stdev->dev, "disabling link");
> +
> + sndev->self_shared->link_sta = 0;
> + switchtec_ntb_send_msg(sndev, LINK_MESSAGE, MSG_LINK_UP);
> +
> + switchtec_ntb_check_link(sndev);
> +
> return 0;
> }
>
> @@ -555,6 +678,9 @@ static irqreturn_t switchtec_ntb_message_isr(int irq, void *dev)
> dev_dbg(&sndev->stdev->dev, "message: %d %08x\n", i,
> (u32)msg);
> iowrite8(1, &sndev->mmio_self_dbmsg->imsg[i].status);
> +
> + if (i == LINK_MESSAGE)
> + switchtec_ntb_check_link(sndev);
> }
> }
>
> @@ -659,6 +785,7 @@ static int switchtec_ntb_add(struct device *dev,
> goto deinit_and_exit;
>
> stdev->sndev = sndev;
> + stdev->link_notifier = switchtec_ntb_link_notification;
> dev_info(dev, "NTB device registered");
>
> return 0;
> @@ -682,6 +809,7 @@ void switchtec_ntb_remove(struct device *dev,
> if (!sndev)
> return;
>
> + stdev->link_notifier = NULL;
> stdev->sndev = NULL;
> ntb_unregister_device(&sndev->ntb);
> switchtec_ntb_deinit_db_msg_irq(sndev);
> --
> 2.11.0
[toc] | [prev] | [next] | [standalone]
| From | Logan Gunthorpe <logang@deltatee.com> |
|---|---|
| Date | 2017-06-29 20:40 +0200 |
| Subject | Re: [PATCH 12/16] switchtec_ntb: add link management |
| Message-ID | <tXMwb-2FO-37@gated-at.bofh.it> |
| In reply to | #1678034 |
On 6/29/2017 12:11 PM, Allen Hubbe wrote: > Should we only set self_partition? I think each peer should be able to set preferred speed, and negotiate down. As written here, the last peer to set the speed overrides the setting on the peer, and even that is not atomic if they race. This function isn't actually setting the link speed. (Though I can understand the confusion from the name.) It's just reporting the slowest speed on the link between the peers for the link_is_up function. (Each port will have negotiated it's own speed). I don't think switchtec has the ability to actually change the port speed or width. (Seeing they are negotiated on boot). Logan
[toc] | [prev] | [next] | [standalone]
| From | Logan Gunthorpe <logang@deltatee.com> |
|---|---|
| Date | 2017-06-29 05:40 +0200 |
| Subject | [PATCH 11/16] switchtec_ntb: add skeleton ntb driver |
| Message-ID | <tXytc-2ks-21@gated-at.bofh.it> |
| In reply to | #1677347 |
This patch simply adds a skeleton NTB driver which will be filled
out in subsequent patches.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Stephen Bates <sbates@raithlin.com>
Reviewed-by: Kurt Schwemmer <kurt.schwemmer@microsemi.com>
---
drivers/ntb/hw/mscc/switchtec_ntb.c | 148 +++++++++++++++++++++++++++++++++++-
include/linux/ntb.h | 3 +
2 files changed, 150 insertions(+), 1 deletion(-)
diff --git a/drivers/ntb/hw/mscc/switchtec_ntb.c b/drivers/ntb/hw/mscc/switchtec_ntb.c
index 4f48b34ce0b5..0587b2380bcc 100644
--- a/drivers/ntb/hw/mscc/switchtec_ntb.c
+++ b/drivers/ntb/hw/mscc/switchtec_ntb.c
@@ -18,6 +18,7 @@
#include <linux/delay.h>
#include <linux/kthread.h>
#include <linux/interrupt.h>
+#include <linux/ntb.h>
MODULE_DESCRIPTION("Microsemi Switchtec(tm) NTB Driver");
MODULE_VERSION("0.1");
@@ -73,6 +74,7 @@ struct shared_mw {
#define LUT_SIZE SZ_64K
struct switchtec_ntb {
+ struct ntb_dev ntb;
struct switchtec_dev *stdev;
int self_partition;
@@ -159,10 +161,148 @@ static int switchtec_ntb_part_op(struct switchtec_ntb *sndev,
return -EIO;
}
+static int switchtec_ntb_mw_count(struct ntb_dev *ntb, int pidx)
+{
+ return 0;
+}
+
+static int switchtec_ntb_mw_get_align(struct ntb_dev *ntb, int pidx,
+ int widx, resource_size_t *addr_align,
+ resource_size_t *size_align,
+ resource_size_t *size_max)
+{
+ return 0;
+}
+
+static int switchtec_ntb_mw_set_trans(struct ntb_dev *ntb, int pidx, int widx,
+ dma_addr_t addr, resource_size_t size)
+{
+ return 0;
+}
+
+static int switchtec_ntb_peer_mw_count(struct ntb_dev *ntb)
+{
+ return 0;
+}
+
+static int switchtec_ntb_peer_mw_get_addr(struct ntb_dev *ntb, int idx,
+ phys_addr_t *base,
+ resource_size_t *size)
+{
+ return 0;
+}
+
+static u64 switchtec_ntb_link_is_up(struct ntb_dev *ntb,
+ enum ntb_speed *speed,
+ enum ntb_width *width)
+{
+ return 0;
+}
+
+static int switchtec_ntb_link_enable(struct ntb_dev *ntb,
+ enum ntb_speed max_speed,
+ enum ntb_width max_width)
+{
+ return 0;
+}
+
+static int switchtec_ntb_link_disable(struct ntb_dev *ntb)
+{
+ return 0;
+}
+
+static u64 switchtec_ntb_db_valid_mask(struct ntb_dev *ntb)
+{
+ return 0;
+}
+
+static int switchtec_ntb_db_vector_count(struct ntb_dev *ntb)
+{
+ return 0;
+}
+
+static u64 switchtec_ntb_db_vector_mask(struct ntb_dev *ntb, int db_vector)
+{
+ return 0;
+}
+
+static u64 switchtec_ntb_db_read(struct ntb_dev *ntb)
+{
+ return 0;
+}
+
+static int switchtec_ntb_db_clear(struct ntb_dev *ntb, u64 db_bits)
+{
+ return 0;
+}
+
+static int switchtec_ntb_db_set_mask(struct ntb_dev *ntb, u64 db_bits)
+{
+ return 0;
+}
+
+static int switchtec_ntb_db_clear_mask(struct ntb_dev *ntb, u64 db_bits)
+{
+ return 0;
+}
+
+static int switchtec_ntb_peer_db_set(struct ntb_dev *ntb, u64 db_bits)
+{
+ return 0;
+}
+
+static int switchtec_ntb_spad_count(struct ntb_dev *ntb)
+{
+ return 0;
+}
+
+static u32 switchtec_ntb_spad_read(struct ntb_dev *ntb, int idx)
+{
+ return 0;
+}
+
+static int switchtec_ntb_spad_write(struct ntb_dev *ntb, int idx, u32 val)
+{
+ return 0;
+}
+
+static int switchtec_ntb_peer_spad_write(struct ntb_dev *ntb, int pidx,
+ int sidx, u32 val)
+{
+ return 0;
+}
+
+static const struct ntb_dev_ops switchtec_ntb_ops = {
+ .mw_count = switchtec_ntb_mw_count,
+ .mw_get_align = switchtec_ntb_mw_get_align,
+ .mw_set_trans = switchtec_ntb_mw_set_trans,
+ .peer_mw_count = switchtec_ntb_peer_mw_count,
+ .peer_mw_get_addr = switchtec_ntb_peer_mw_get_addr,
+ .link_is_up = switchtec_ntb_link_is_up,
+ .link_enable = switchtec_ntb_link_enable,
+ .link_disable = switchtec_ntb_link_disable,
+ .db_valid_mask = switchtec_ntb_db_valid_mask,
+ .db_vector_count = switchtec_ntb_db_vector_count,
+ .db_vector_mask = switchtec_ntb_db_vector_mask,
+ .db_read = switchtec_ntb_db_read,
+ .db_clear = switchtec_ntb_db_clear,
+ .db_set_mask = switchtec_ntb_db_set_mask,
+ .db_clear_mask = switchtec_ntb_db_clear_mask,
+ .peer_db_set = switchtec_ntb_peer_db_set,
+ .spad_count = switchtec_ntb_spad_count,
+ .spad_read = switchtec_ntb_spad_read,
+ .spad_write = switchtec_ntb_spad_write,
+ .peer_spad_write = switchtec_ntb_peer_spad_write,
+};
+
static void switchtec_ntb_init_sndev(struct switchtec_ntb *sndev)
{
u64 part_map;
+ sndev->ntb.pdev = sndev->stdev->pdev;
+ sndev->ntb.topo = NTB_TOPO_SWITCH;
+ sndev->ntb.ops = &switchtec_ntb_ops;
+
sndev->self_partition = sndev->stdev->partition;
sndev->mmio_ntb = sndev->stdev->mmio_ntb;
@@ -495,7 +635,6 @@ static int switchtec_ntb_add(struct device *dev,
return -ENOMEM;
sndev->stdev = stdev;
-
switchtec_ntb_init_sndev(sndev);
switchtec_ntb_init_mw(sndev);
switchtec_ntb_init_db(sndev);
@@ -515,11 +654,17 @@ static int switchtec_ntb_add(struct device *dev,
switchtec_ntb_init_mw(sndev);
+ rc = ntb_register_device(&sndev->ntb);
+ if (rc)
+ goto deinit_and_exit;
+
stdev->sndev = sndev;
dev_info(dev, "NTB device registered");
return 0;
+deinit_and_exit:
+ switchtec_ntb_deinit_db_msg_irq(sndev);
deinit_shared_and_exit:
switchtec_ntb_deinit_shared_mw(sndev);
free_and_exit:
@@ -538,6 +683,7 @@ void switchtec_ntb_remove(struct device *dev,
return;
stdev->sndev = NULL;
+ ntb_unregister_device(&sndev->ntb);
switchtec_ntb_deinit_db_msg_irq(sndev);
switchtec_ntb_deinit_shared_mw(sndev);
kfree(sndev);
diff --git a/include/linux/ntb.h b/include/linux/ntb.h
index de4f800c82b6..bf3dd2d2a821 100644
--- a/include/linux/ntb.h
+++ b/include/linux/ntb.h
@@ -70,6 +70,7 @@ struct pci_dev;
* @NTB_TOPO_SEC: On secondary side of remote ntb.
* @NTB_TOPO_B2B_USD: On primary side of local ntb upstream of remote ntb.
* @NTB_TOPO_B2B_DSD: On primary side of local ntb downstream of remote ntb.
+ * @NTB_TOPO_SWITCH: Connected via a switch which supports ntb.
*/
enum ntb_topo {
NTB_TOPO_NONE = -1,
@@ -77,6 +78,7 @@ enum ntb_topo {
NTB_TOPO_SEC,
NTB_TOPO_B2B_USD,
NTB_TOPO_B2B_DSD,
+ NTB_TOPO_SWITCH,
};
static inline int ntb_topo_is_b2b(enum ntb_topo topo)
@@ -97,6 +99,7 @@ static inline char *ntb_topo_string(enum ntb_topo topo)
case NTB_TOPO_SEC: return "NTB_TOPO_SEC";
case NTB_TOPO_B2B_USD: return "NTB_TOPO_B2B_USD";
case NTB_TOPO_B2B_DSD: return "NTB_TOPO_B2B_DSD";
+ case NTB_TOPO_SWITCH: return "NTB_TOPO_SWITCH";
}
return "NTB_TOPO_INVALID";
}
--
2.11.0
[toc] | [prev] | [next] | [standalone]
| From | Logan Gunthorpe <logang@deltatee.com> |
|---|---|
| Date | 2017-06-29 05:40 +0200 |
| Subject | [PATCH 10/16] switchtec_ntb: initialize hardware for doorbells and messages |
| Message-ID | <tXytd-2ks-25@gated-at.bofh.it> |
| In reply to | #1677347 |
This patch sets up some hardware registers and creates interrupt service
routines for the doorbells and messages.
There are 64 doorbells in the switch that are shared between all
partitions. The upper 4 doorbells are also shared with the messages
and are there for not used. Thus, this code provides 28 doorbells
for each partition.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Stephen Bates <sbates@raithlin.com>
Reviewed-by: Kurt Schwemmer <kurt.schwemmer@microsemi.com>
---
drivers/ntb/hw/mscc/switchtec_ntb.c | 144 ++++++++++++++++++++++++++++++++++++
1 file changed, 144 insertions(+)
diff --git a/drivers/ntb/hw/mscc/switchtec_ntb.c b/drivers/ntb/hw/mscc/switchtec_ntb.c
index ff03424153d9..4f48b34ce0b5 100644
--- a/drivers/ntb/hw/mscc/switchtec_ntb.c
+++ b/drivers/ntb/hw/mscc/switchtec_ntb.c
@@ -17,6 +17,7 @@
#include <linux/module.h>
#include <linux/delay.h>
#include <linux/kthread.h>
+#include <linux/interrupt.h>
MODULE_DESCRIPTION("Microsemi Switchtec(tm) NTB Driver");
MODULE_VERSION("0.1");
@@ -77,6 +78,9 @@ struct switchtec_ntb {
int self_partition;
int peer_partition;
+ int doorbell_irq;
+ int message_irq;
+
struct ntb_info_regs __iomem *mmio_ntb;
struct ntb_ctrl_regs __iomem *mmio_ctrl;
struct ntb_dbmsg_regs __iomem *mmio_dbmsg;
@@ -88,6 +92,11 @@ struct switchtec_ntb {
struct shared_mw __iomem *peer_shared;
dma_addr_t self_shared_dma;
+ u64 db_mask;
+ u64 db_valid_mask;
+ int db_shift;
+ int db_peer_shift;
+
int nr_direct_mw;
int nr_lut_mw;
int direct_mw_to_bar[MAX_DIRECT_MW];
@@ -196,6 +205,49 @@ static void switchtec_ntb_init_mw(struct switchtec_ntb *sndev)
sndev->nr_direct_mw, cnt - sndev->nr_direct_mw);
}
+/*
+ * There are 64 doorbells in the switch hardware but this is
+ * shared among all partitions. So we must split them in half
+ * (32 for each partition). However, the message interrupts are
+ * also shared with the top 4 doorbells so we just limit this to
+ * 28 doorbells per partition
+ */
+static void switchtec_ntb_init_db(struct switchtec_ntb *sndev)
+{
+ sndev->db_valid_mask = 0x0FFFFFFF;
+
+ if (sndev->self_partition < sndev->peer_partition) {
+ sndev->db_shift = 0;
+ sndev->db_peer_shift = 32;
+ } else {
+ sndev->db_shift = 32;
+ sndev->db_peer_shift = 0;
+ }
+
+ sndev->db_mask = 0x0FFFFFFFFFFFFFFFULL;
+ iowrite64(~sndev->db_mask, &sndev->mmio_self_dbmsg->idb_mask);
+ iowrite64(sndev->db_valid_mask << sndev->db_peer_shift,
+ &sndev->mmio_self_dbmsg->odb_mask);
+}
+
+static void switchtec_ntb_init_msgs(struct switchtec_ntb *sndev)
+{
+ int i;
+ u32 msg_map = 0;
+
+ for (i = 0; i < ARRAY_SIZE(sndev->mmio_self_dbmsg->imsg); i++) {
+ int m = i | sndev->peer_partition << 2;
+
+ msg_map |= m << i * 8;
+ }
+
+ iowrite32(msg_map, &sndev->mmio_self_dbmsg->msg_map);
+
+ for (i = 0; i < ARRAY_SIZE(sndev->mmio_self_dbmsg->imsg); i++)
+ iowrite64(NTB_DBMSG_IMSG_STATUS | NTB_DBMSG_IMSG_MASK,
+ &sndev->mmio_self_dbmsg->imsg[i]);
+}
+
static int switchtec_ntb_init_req_id_table(struct switchtec_ntb *sndev)
{
int rc = 0;
@@ -342,6 +394,87 @@ static void switchtec_ntb_deinit_shared_mw(struct switchtec_ntb *sndev)
sndev->self_shared_dma);
}
+static irqreturn_t switchtec_ntb_doorbell_isr(int irq, void *dev)
+{
+ struct switchtec_ntb *sndev = dev;
+
+ dev_dbg(&sndev->stdev->dev, "doorbell\n");
+
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t switchtec_ntb_message_isr(int irq, void *dev)
+{
+ int i;
+ struct switchtec_ntb *sndev = dev;
+
+ for (i = 0; i < ARRAY_SIZE(sndev->mmio_self_dbmsg->imsg); i++) {
+ u64 msg = ioread64(&sndev->mmio_self_dbmsg->imsg[i]);
+
+ if (msg & NTB_DBMSG_IMSG_STATUS) {
+ dev_dbg(&sndev->stdev->dev, "message: %d %08x\n", i,
+ (u32)msg);
+ iowrite8(1, &sndev->mmio_self_dbmsg->imsg[i].status);
+ }
+ }
+
+ return IRQ_HANDLED;
+}
+
+static int switchtec_ntb_init_db_msg_irq(struct switchtec_ntb *sndev)
+{
+ int i;
+ int rc;
+ int doorbell_irq = 0;
+ int message_irq = 0;
+ int event_irq;
+ int idb_vecs = sizeof(sndev->mmio_self_dbmsg->idb_vec_map);
+
+ event_irq = ioread32(&sndev->stdev->mmio_part_cfg->vep_vector_number);
+
+ while (doorbell_irq == event_irq)
+ doorbell_irq++;
+ while (message_irq == doorbell_irq ||
+ message_irq == event_irq)
+ message_irq++;
+
+ dev_dbg(&sndev->stdev->dev, "irqs - event: %d, db: %d, msgs: %d",
+ event_irq, doorbell_irq, message_irq);
+
+ for (i = 0; i < idb_vecs - 4; i++)
+ iowrite8(doorbell_irq,
+ &sndev->mmio_self_dbmsg->idb_vec_map[i]);
+
+ for (; i < idb_vecs; i++)
+ iowrite8(message_irq,
+ &sndev->mmio_self_dbmsg->idb_vec_map[i]);
+
+ sndev->doorbell_irq = pci_irq_vector(sndev->stdev->pdev, doorbell_irq);
+ sndev->message_irq = pci_irq_vector(sndev->stdev->pdev, message_irq);
+
+ rc = request_irq(sndev->doorbell_irq,
+ switchtec_ntb_doorbell_isr, 0,
+ "switchtec_ntb_doorbell", sndev);
+ if (rc)
+ return rc;
+
+ rc = request_irq(sndev->message_irq,
+ switchtec_ntb_message_isr, 0,
+ "switchtec_ntb_message", sndev);
+ if (rc) {
+ free_irq(sndev->doorbell_irq, sndev);
+ return rc;
+ }
+
+ return 0;
+}
+
+static void switchtec_ntb_deinit_db_msg_irq(struct switchtec_ntb *sndev)
+{
+ free_irq(sndev->doorbell_irq, sndev);
+ free_irq(sndev->message_irq, sndev);
+}
+
static int switchtec_ntb_add(struct device *dev,
struct class_interface *class_intf)
{
@@ -365,6 +498,8 @@ static int switchtec_ntb_add(struct device *dev,
switchtec_ntb_init_sndev(sndev);
switchtec_ntb_init_mw(sndev);
+ switchtec_ntb_init_db(sndev);
+ switchtec_ntb_init_msgs(sndev);
rc = switchtec_ntb_init_req_id_table(sndev);
if (rc)
@@ -374,11 +509,19 @@ static int switchtec_ntb_add(struct device *dev,
if (rc)
goto free_and_exit;
+ rc = switchtec_ntb_init_db_msg_irq(sndev);
+ if (rc)
+ goto deinit_shared_and_exit;
+
+ switchtec_ntb_init_mw(sndev);
+
stdev->sndev = sndev;
dev_info(dev, "NTB device registered");
return 0;
+deinit_shared_and_exit:
+ switchtec_ntb_deinit_shared_mw(sndev);
free_and_exit:
kfree(sndev);
dev_err(dev, "failed to register ntb device: %d", rc);
@@ -395,6 +538,7 @@ void switchtec_ntb_remove(struct device *dev,
return;
stdev->sndev = NULL;
+ switchtec_ntb_deinit_db_msg_irq(sndev);
switchtec_ntb_deinit_shared_mw(sndev);
kfree(sndev);
dev_info(dev, "ntb device unregistered");
--
2.11.0
[toc] | [prev] | [next] | [standalone]
| From | Greg Kroah-Hartman <gregkh@linuxfoundation.org> |
|---|---|
| Date | 2017-06-29 17:20 +0200 |
| Message-ID | <tXJoC-LR-17@gated-at.bofh.it> |
| In reply to | #1677347 |
On Wed, Jun 28, 2017 at 09:26:32PM -0600, Logan Gunthorpe wrote: > Changes since the rfc: I think your subject: line is missing something :)
[toc] | [prev] | [next] | [standalone]
| From | Logan Gunthorpe <logang@deltatee.com> |
|---|---|
| Date | 2017-06-29 17:40 +0200 |
| Message-ID | <tXJHZ-Sn-25@gated-at.bofh.it> |
| In reply to | #1677858 |
On 29/06/17 09:15 AM, Greg Kroah-Hartman wrote: > On Wed, Jun 28, 2017 at 09:26:32PM -0600, Logan Gunthorpe wrote: >> Changes since the rfc: > > I think your subject: line is missing something :) Yeah, sorry. A small oversight. Logan
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web