Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1540656 > unrolled thread
| Started by | Serge Semin <fancer.lancer@gmail.com> |
|---|---|
| First post | 2016-12-12 22:10 +0100 |
| Last post | 2016-12-14 01:00 +0100 |
| Articles | 7 on this page of 27 — 2 participants |
Back to article view | Back to linux.kernel
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
[PATCH v2 0/9] NTB: Alter kernel API to support multi-port devices Serge Semin <fancer.lancer@gmail.com> - 2016-12-12 22:10 +0100
[PATCH v2 2/9] NTB: Add indexed ports NTB API Serge Semin <fancer.lancer@gmail.com> - 2016-12-12 22:10 +0100
[PATCH v2 7/9] NTB: Add new Memory Windows API documentation Serge Semin <fancer.lancer@gmail.com> - 2016-12-12 22:10 +0100
[PATCH v2 3/9] NTB: Alter link-state API to support multi-port devices Serge Semin <fancer.lancer@gmail.com> - 2016-12-12 22:20 +0100
[PATCH v2 8/9] NTB: Add PCIe Gen4 link speed Serge Semin <fancer.lancer@gmail.com> - 2016-12-12 22:20 +0100
[PATCH v2 5/9] NTB: Alter Scratchpads API to support multi-ports devices Serge Semin <fancer.lancer@gmail.com> - 2016-12-12 22:20 +0100
[PATCH v2 1/9] NTB: Make link-state API being declared first Serge Semin <fancer.lancer@gmail.com> - 2016-12-12 22:20 +0100
[PATCH v2 4/9] NTB: Alter MW API to support multi-ports devices Serge Semin <fancer.lancer@gmail.com> - 2016-12-12 22:20 +0100
[PATCH v3 8/9] NTB: Add PCIe Gen4 link speed Serge Semin <fancer.lancer@gmail.com> - 2016-12-14 01:00 +0100
[PATCH v3 1/9] NTB: Make link-state API being declared first Serge Semin <fancer.lancer@gmail.com> - 2016-12-14 01:00 +0100
RE: [PATCH v3 1/9] NTB: Make link-state API being declared first "Allen Hubbe" <Allen.Hubbe@dell.com> - 2016-12-14 08:10 +0100
[PATCH v3 5/9] NTB: Alter Scratchpads API to support multi-ports devices Serge Semin <fancer.lancer@gmail.com> - 2016-12-14 01:00 +0100
RE: [PATCH v3 5/9] NTB: Alter Scratchpads API to support multi-ports devices "Allen Hubbe" <Allen.Hubbe@dell.com> - 2016-12-14 08:10 +0100
[PATCH v3 9/9] NTB: Add ntb.h comments Serge Semin <fancer.lancer@gmail.com> - 2016-12-14 01:00 +0100
RE: [PATCH v3 9/9] NTB: Add ntb.h comments "Allen Hubbe" <Allen.Hubbe@dell.com> - 2016-12-14 08:20 +0100
[PATCH v3 9/9] NTB: Add ntb.h comments Serge Semin <fancer.lancer@gmail.com> - 2016-12-14 08:20 +0100
RE: [PATCH v3 9/9] NTB: Add ntb.h comments "Allen Hubbe" <Allen.Hubbe@dell.com> - 2016-12-14 08:20 +0100
[PATCH v3 9/9] NTB: Add ntb.h comments Serge Semin <fancer.lancer@gmail.com> - 2016-12-20 11:00 +0100
[PATCH v3 0/9] NTB: Alter kernel API to support multi-port devices Serge Semin <fancer.lancer@gmail.com> - 2016-12-14 01:00 +0100
[PATCH v3 2/9] NTB: Add indexed ports NTB API Serge Semin <fancer.lancer@gmail.com> - 2016-12-14 01:00 +0100
RE: [PATCH v3 2/9] NTB: Add indexed ports NTB API "Allen Hubbe" <Allen.Hubbe@dell.com> - 2016-12-14 08:10 +0100
[PATCH v3 4/9] NTB: Alter MW API to support multi-ports devices Serge Semin <fancer.lancer@gmail.com> - 2016-12-14 01:00 +0100
RE: [PATCH v3 4/9] NTB: Alter MW API to support multi-ports devices "Allen Hubbe" <Allen.Hubbe@dell.com> - 2016-12-14 08:10 +0100
[PATCH v3 3/9] NTB: Alter link-state API to support multi-port devices Serge Semin <fancer.lancer@gmail.com> - 2016-12-14 01:00 +0100
[PATCH v3 6/9] NTB: Add Messaging NTB API Serge Semin <fancer.lancer@gmail.com> - 2016-12-14 01:00 +0100
[PATCH v3 6/9] NTB: Add Messaging NTB API Serge Semin <fancer.lancer@gmail.com> - 2016-12-20 10:50 +0100
[PATCH v3 7/9] NTB: Add new Memory Windows API documentation Serge Semin <fancer.lancer@gmail.com> - 2016-12-14 01:00 +0100
Page 2 of 2 — ← Prev page 1 [2]
| From | "Allen Hubbe" <Allen.Hubbe@dell.com> |
|---|---|
| Date | 2016-12-14 08:10 +0100 |
| Subject | RE: [PATCH v3 2/9] NTB: Add indexed ports NTB API |
| Message-ID | <sObRn-3ug-11@gated-at.bofh.it> |
| In reply to | #1541575 |
From: Serge Semin
> There is some NTB hardware, which can combine more than just two domains
> over NTB. For instance, some IDT PCIe-switches can have NTB-functions
> activated on more than two-ports. The different domains are distinguished
> by ports they are connected to. So the new port-related methods are added to
> the NTB API:
> ntb_port_number() - return local port
> ntb_peer_port_count() - return number of peers local port can connect to
> ntb_peer_port_number(pdix) - return port number by it index
> ntb_peer_port_idx(port) - return port index by it number
>
> Current test-drivers aren't changed much. They still support two-ports devices
> for the time being while multi-ports hardware drivers aren't added.
>
> By default port-related API is declared for two-ports hardware.
> So corresponding hardware drivers won't need to implement it.
>
> Signed-off-by: Serge Semin <fancer.lancer@gmail.com>
Acked-by: Allen Hubbe <Allen.Hubbe@dell.com>
> ---
> drivers/ntb/ntb.c | 54 ++++++++++++++
> drivers/ntb/ntb_transport.c | 6 ++
> drivers/ntb/test/ntb_perf.c | 4 ++
> drivers/ntb/test/ntb_pingpong.c | 6 ++
> drivers/ntb/test/ntb_tool.c | 5 ++
> include/linux/ntb.h | 156 ++++++++++++++++++++++++++++++++++++++++
> 6 files changed, 231 insertions(+)
>
> diff --git a/drivers/ntb/ntb.c b/drivers/ntb/ntb.c
> index 2e25307..1e92e52 100644
> --- a/drivers/ntb/ntb.c
> +++ b/drivers/ntb/ntb.c
> @@ -191,6 +191,60 @@ void ntb_db_event(struct ntb_dev *ntb, int vector)
> }
> EXPORT_SYMBOL(ntb_db_event);
>
> +int ntb_default_port_number(struct ntb_dev *ntb)
> +{
> + switch (ntb->topo) {
> + case NTB_TOPO_PRI:
> + case NTB_TOPO_B2B_USD:
> + return NTB_PORT_PRI_USD;
> + case NTB_TOPO_SEC:
> + case NTB_TOPO_B2B_DSD:
> + return NTB_PORT_SEC_DSD;
> + default:
> + break;
> + }
> +
> + return -EINVAL;
> +}
> +EXPORT_SYMBOL(ntb_default_port_number);
> +
> +int ntb_default_peer_port_count(struct ntb_dev *ntb)
> +{
> + return NTB_DEF_PEER_CNT;
> +}
> +EXPORT_SYMBOL(ntb_default_peer_port_count);
> +
> +int ntb_default_peer_port_number(struct ntb_dev *ntb, int pidx)
> +{
> + if (pidx != NTB_DEF_PEER_IDX)
> + return -EINVAL;
> +
> + switch (ntb->topo) {
> + case NTB_TOPO_PRI:
> + case NTB_TOPO_B2B_USD:
> + return NTB_PORT_SEC_DSD;
> + case NTB_TOPO_SEC:
> + case NTB_TOPO_B2B_DSD:
> + return NTB_PORT_PRI_USD;
> + default:
> + break;
> + }
> +
> + return -EINVAL;
> +}
> +EXPORT_SYMBOL(ntb_default_peer_port_number);
> +
> +int ntb_default_peer_port_idx(struct ntb_dev *ntb, int port)
> +{
> + int peer_port = ntb_default_peer_port_number(ntb, NTB_DEF_PEER_IDX);
> +
> + if (peer_port == -EINVAL || port != peer_port)
> + return -EINVAL;
> +
> + return 0;
> +}
> +EXPORT_SYMBOL(ntb_default_peer_port_idx);
> +
> static int ntb_probe(struct device *dev)
> {
> struct ntb_dev *ntb;
> diff --git a/drivers/ntb/ntb_transport.c b/drivers/ntb/ntb_transport.c
> index 4eb8adb..10518b7 100644
> --- a/drivers/ntb/ntb_transport.c
> +++ b/drivers/ntb/ntb_transport.c
> @@ -94,6 +94,9 @@ MODULE_PARM_DESC(use_dma, "Use DMA engine to perform large data copy");
>
> static struct dentry *nt_debugfs_dir;
>
> +/* Only two-ports NTB devices are supported */
> +#define PIDX NTB_DEF_PEER_IDX
> +
> struct ntb_queue_entry {
> /* ntb_queue list reference */
> struct list_head entry;
> @@ -1083,6 +1086,9 @@ static int ntb_transport_probe(struct ntb_client *self, struct
> ntb_dev *ndev)
> dev_dbg(&ndev->dev,
> "scratchpad is unsafe, proceed anyway...\n");
>
> + if (ntb_peer_port_count(ndev) != NTB_DEF_PEER_CNT)
> + dev_warn(&ndev->dev, "Multi-port NTB devices unsupported\n");
> +
> node = dev_to_node(&ndev->dev);
>
> nt = kzalloc_node(sizeof(*nt), GFP_KERNEL, node);
> diff --git a/drivers/ntb/test/ntb_perf.c b/drivers/ntb/test/ntb_perf.c
> index e75d4fd..c908b3a 100644
> --- a/drivers/ntb/test/ntb_perf.c
> +++ b/drivers/ntb/test/ntb_perf.c
> @@ -76,6 +76,7 @@
> #define DMA_RETRIES 20
> #define SZ_4G (1ULL << 32)
> #define MAX_SEG_ORDER 20 /* no larger than 1M for kmalloc buffer */
> +#define PIDX NTB_DEF_PEER_IDX
>
> MODULE_LICENSE(DRIVER_LICENSE);
> MODULE_VERSION(DRIVER_VERSION);
> @@ -764,6 +765,9 @@ static int perf_probe(struct ntb_client *client, struct ntb_dev *ntb)
> return -EIO;
> }
>
> + if (ntb_peer_port_count(ntb) != NTB_DEF_PEER_CNT)
> + dev_warn(&ntb->dev, "Multi-port NTB devices unsupported\n");
> +
> node = dev_to_node(&pdev->dev);
>
> perf = kzalloc_node(sizeof(*perf), GFP_KERNEL, node);
> diff --git a/drivers/ntb/test/ntb_pingpong.c b/drivers/ntb/test/ntb_pingpong.c
> index 4358611..12f8b40 100644
> --- a/drivers/ntb/test/ntb_pingpong.c
> +++ b/drivers/ntb/test/ntb_pingpong.c
> @@ -90,6 +90,9 @@ static unsigned long db_init = 0x7;
> module_param(db_init, ulong, 0644);
> MODULE_PARM_DESC(db_init, "Initial doorbell bits to ring on the peer");
>
> +/* Only two-ports NTB devices are supported */
> +#define PIDX NTB_DEF_PEER_IDX
> +
> struct pp_ctx {
> struct ntb_dev *ntb;
> u64 db_bits;
> @@ -230,6 +233,9 @@ static int pp_probe(struct ntb_client *client,
> }
> }
>
> + if (ntb_peer_port_count(ntb) != NTB_DEF_PEER_CNT)
> + dev_warn(&ntb->dev, "multi-port NTB is unsupported\n");
> +
> pp = kmalloc(sizeof(*pp), GFP_KERNEL);
> if (!pp) {
> rc = -ENOMEM;
> diff --git a/drivers/ntb/test/ntb_tool.c b/drivers/ntb/test/ntb_tool.c
> index 61bf2ef..690862d 100644
> --- a/drivers/ntb/test/ntb_tool.c
> +++ b/drivers/ntb/test/ntb_tool.c
> @@ -120,6 +120,8 @@ MODULE_AUTHOR(DRIVER_AUTHOR);
> MODULE_DESCRIPTION(DRIVER_DESCRIPTION);
>
> #define MAX_MWS 16
> +/* Only two-ports devices are supported */
> +#define PIDX NTB_DEF_PEER_IDX
>
> static struct dentry *tool_dbgfs;
>
> @@ -919,6 +921,9 @@ static int tool_probe(struct ntb_client *self, struct ntb_dev *ntb)
> if (ntb_spad_is_unsafe(ntb))
> dev_dbg(&ntb->dev, "scratchpad is unsafe\n");
>
> + if (ntb_peer_port_count(ntb) != NTB_DEF_PEER_CNT)
> + dev_warn(&ntb->dev, "multi-port NTB is unsupported\n");
> +
> tc = kzalloc(sizeof(*tc), GFP_KERNEL);
> if (!tc) {
> rc = -ENOMEM;
> diff --git a/include/linux/ntb.h b/include/linux/ntb.h
> index 5d1f260..fe175c7 100644
> --- a/include/linux/ntb.h
> +++ b/include/linux/ntb.h
> @@ -140,6 +140,20 @@ enum ntb_width {
> };
>
> /**
> + * enum ntb_default_port - NTB default port number
> + * @NTB_PORT_PRI_USD: Default port of the NTB_TOPO_PRI/NTB_TOPO_B2B_USD
> + * topologies
> + * @NTB_PORT_SEC_DSD: Default port of the NTB_TOPO_SEC/NTB_TOPO_B2B_DSD
> + * topologies
> + */
> +enum ntb_default_port {
> + NTB_PORT_PRI_USD,
> + NTB_PORT_SEC_DSD
> +};
> +#define NTB_DEF_PEER_CNT (1)
> +#define NTB_DEF_PEER_IDX (0)
> +
> +/**
> * struct ntb_client_ops - ntb client operations
> * @probe: Notify client of a new device.
> * @remove: Notify client to remove a device.
> @@ -179,6 +193,10 @@ static inline int ntb_ctx_ops_is_valid(const struct ntb_ctx_ops *ops)
>
> /**
> * struct ntb_ctx_ops - ntb device operations
> + * @port_number: See ntb_port_number().
> + * @peer_port_count: See ntb_peer_port_count().
> + * @peer_port_number: See ntb_peer_port_number().
> + * @peer_port_idx: See ntb_peer_port_idx().
> * @link_is_up: See ntb_link_is_up().
> * @link_enable: See ntb_link_enable().
> * @link_disable: See ntb_link_disable().
> @@ -212,6 +230,11 @@ static inline int ntb_ctx_ops_is_valid(const struct ntb_ctx_ops *ops)
> * @peer_spad_write: See ntb_peer_spad_write().
> */
> struct ntb_dev_ops {
> + int (*port_number)(struct ntb_dev *ntb);
> + int (*peer_port_count)(struct ntb_dev *ntb);
> + int (*peer_port_number)(struct ntb_dev *ntb, int pidx);
> + int (*peer_port_idx)(struct ntb_dev *ntb, int port);
> +
> int (*link_is_up)(struct ntb_dev *ntb,
> enum ntb_speed *speed, enum ntb_width *width);
> int (*link_enable)(struct ntb_dev *ntb,
> @@ -265,6 +288,9 @@ static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops *ops)
> {
> /* commented callbacks are not required: */
> return
> + !ops->peer_port_count == !ops->port_number &&
> + !ops->peer_port_number == !ops->port_number &&
> + !ops->peer_port_idx == !ops->port_number &&
> ops->link_is_up &&
> ops->link_enable &&
> ops->link_disable &&
> @@ -442,6 +468,136 @@ void ntb_link_event(struct ntb_dev *ntb);
> void ntb_db_event(struct ntb_dev *ntb, int vector);
>
> /**
> + * ntb_default_port_number() - get the default local port number
> + * @ntb: NTB device context.
> + *
> + * If hardware driver doesn't specify port_number() callback method, the NTB
> + * is considered with just two ports. So this method returns default local
> + * port number in compliance with topology.
> + *
> + * NOTE Don't call this method directly. The ntb_port_number() function should
> + * be used instead.
> + *
> + * Return: the default local port number
> + */
> +int ntb_default_port_number(struct ntb_dev *ntb);
> +
> +/**
> + * ntb_default_port_count() - get the default number of peer device ports
> + * @ntb: NTB device context.
> + *
> + * By default hardware driver supports just one peer device.
> + *
> + * NOTE Don't call this method directly. The ntb_peer_port_count() function
> + * should be used instead.
> + *
> + * Return: the default number of peer ports
> + */
> +int ntb_default_peer_port_count(struct ntb_dev *ntb);
> +
> +/**
> + * ntb_default_peer_port_number() - get the default peer port by given index
> + * @ntb: NTB device context.
> + * @idx: Peer port index (should not differ from zero).
> + *
> + * By default hardware driver supports just one peer device, so this method
> + * shall return the corresponding value from enum ntb_default_port.
> + *
> + * NOTE Don't call this method directly. The ntb_peer_port_number() function
> + * should be used instead.
> + *
> + * Return: the peer device port or negative value indicating an error
> + */
> +int ntb_default_peer_port_number(struct ntb_dev *ntb, int pidx);
> +
> +/**
> + * ntb_default_peer_port_idx() - get the default peer device port index by
> + * given port number
> + * @ntb: NTB device context.
> + * @port: Peer port number (should be one of enum ntb_default_port).
> + *
> + * By default hardware driver supports just one peer device, so while
> + * specified port-argument indicates peer port from enum ntb_default_port,
> + * the return value shall be zero.
> + *
> + * NOTE Don't call this method directly. The ntb_peer_port_idx() function
> + * should be used instead.
> + *
> + * Return: the peer port index or negative value indicating an error
> + */
> +int ntb_default_peer_port_idx(struct ntb_dev *ntb, int port);
> +
> +/**
> + * ntb_port_number() - get the local port number
> + * @ntb: NTB device context.
> + *
> + * Hardware must support at least simple two-ports ntb connection
> + *
> + * Return: the local port number
> + */
> +static inline int ntb_port_number(struct ntb_dev *ntb)
> +{
> + if (!ntb->ops->port_number)
> + return ntb_default_port_number(ntb);
> +
> + return ntb->ops->port_number(ntb);
> +}
> +
> +/**
> + * ntb_peer_port_count() - get the number of peer device ports
> + * @ntb: NTB device context.
> + *
> + * Hardware may support an access to memory of several remote domains
> + * over multi-port NTB devices. This method returns the number of peers,
> + * local device can have shared memory with.
> + *
> + * Return: the number of peer ports
> + */
> +static inline int ntb_peer_port_count(struct ntb_dev *ntb)
> +{
> + if (!ntb->ops->peer_port_count)
> + return ntb_default_peer_port_count(ntb);
> +
> + return ntb->ops->peer_port_count(ntb);
> +}
> +
> +/**
> + * ntb_peer_port_number() - get the peer port by given index
> + * @ntb: NTB device context.
> + * @pidx: Peer port index.
> + *
> + * Peer ports are continuously enumerated by NTB API logic, so this method
> + * lets to retrieve port real number by its index.
> + *
> + * Return: the peer device port or negative value indicating an error
> + */
> +static inline int ntb_peer_port_number(struct ntb_dev *ntb, int pidx)
> +{
> + if (!ntb->ops->peer_port_number)
> + return ntb_default_peer_port_number(ntb, pidx);
> +
> + return ntb->ops->peer_port_number(ntb, pidx);
> +}
> +
> +/**
> + * ntb_peer_port_idx() - get the peer device port index by given port number
> + * @ntb: NTB device context.
> + * @port: Peer port number.
> + *
> + * Inverse operation of ntb_peer_port_number(), so one can get port index
> + * by specified port number.
> + *
> + * Return: the peer port index or negative value indicating an error
> + */
> +static inline int ntb_peer_port_idx(struct ntb_dev *ntb, int port)
> +{
> + if (!ntb->ops->peer_port_idx)
> + return ntb_default_peer_port_idx(ntb, port);
> +
> + return ntb->ops->peer_port_idx(ntb, port);
> +}
> +
> +/**
> * ntb_link_is_up() - get the current ntb link state
> * @ntb: NTB device context.
> * @speed: OUT - The link speed expressed as PCIe generation number.
> --
> 2.6.6
[toc] | [prev] | [next] | [standalone]
| From | Serge Semin <fancer.lancer@gmail.com> |
|---|---|
| Date | 2016-12-14 01:00 +0100 |
| Subject | [PATCH v3 4/9] NTB: Alter MW API to support multi-ports devices |
| Message-ID | <sO59g-7zL-17@gated-at.bofh.it> |
| In reply to | #1541574 |
Multi-port NTB devices permit to share a memory between all accessible peers.
Memory Windows API is altered to correspondingly initialize and map memory
windows for such devices:
ntb_mw_count(pidx); - number of inbound memory windows, which can be allocated
for shared buffer with specified peer device.
ntb_mw_get_align(pidx, widx); - get alignment and size restriction parameters
to properly allocate inbound memory region.
ntb_peer_mw_count(); - get number of outbound memory windows.
ntb_peer_mw_get_addr(widx); - get mapping address of an outbound memory window
If hardware supports inbound translation configured on the local ntb port:
ntb_mw_set_trans(pidx, widx); - set translation address of allocated inbound
memory window so a peer device could access it.
ntb_mw_clear_trans(pidx, widx); - clear the translation address of an inbound
memory window.
If hardware supports outbound translation configured on the peer ntb port:
ntb_peer_mw_set_trans(pidx, widx); - set translation address of a memory
window retrieved from a peer device
ntb_peer_mw_clear_trans(pidx, widx); - clear the translation address of an
outbound memory window
Signed-off-by: Serge Semin <fancer.lancer@gmail.com>
---
drivers/ntb/hw/amd/ntb_hw_amd.c | 68 +++++++++---
drivers/ntb/hw/intel/ntb_hw_intel.c | 90 ++++++++++++----
drivers/ntb/ntb.c | 2 +
drivers/ntb/ntb_transport.c | 21 +++-
drivers/ntb/test/ntb_perf.c | 17 ++-
drivers/ntb/test/ntb_tool.c | 43 +++++---
include/linux/ntb.h | 208 ++++++++++++++++++++++++++++--------
7 files changed, 342 insertions(+), 107 deletions(-)
diff --git a/drivers/ntb/hw/amd/ntb_hw_amd.c b/drivers/ntb/hw/amd/ntb_hw_amd.c
index 4d8d0bd..6a41c38 100644
--- a/drivers/ntb/hw/amd/ntb_hw_amd.c
+++ b/drivers/ntb/hw/amd/ntb_hw_amd.c
@@ -5,6 +5,7 @@
* GPL LICENSE SUMMARY
*
* Copyright (C) 2016 Advanced Micro Devices, Inc. All Rights Reserved.
+ * Copyright (C) 2016 T-Platforms. All Rights Reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -13,6 +14,7 @@
* BSD LICENSE
*
* Copyright (C) 2016 Advanced Micro Devices, Inc. All Rights Reserved.
+ * Copyright (C) 2016 T-Platforms. All Rights Reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
@@ -79,40 +81,42 @@ static int ndev_mw_to_bar(struct amd_ntb_dev *ndev, int idx)
return 1 << idx;
}
-static int amd_ntb_mw_count(struct ntb_dev *ntb)
+static int amd_ntb_mw_count(struct ntb_dev *ntb, int pidx)
{
+ if (pidx != NTB_DEF_PEER_IDX)
+ return -EINVAL;
+
return ntb_ndev(ntb)->mw_count;
}
-static int amd_ntb_mw_get_range(struct ntb_dev *ntb, int idx,
- phys_addr_t *base,
- resource_size_t *size,
- resource_size_t *align,
- resource_size_t *align_size)
+static int amd_ntb_mw_get_align(struct ntb_dev *ntb, int pidx, int idx,
+ resource_size_t *addr_align,
+ resource_size_t *size_align,
+ resource_size_t *size_max)
{
struct amd_ntb_dev *ndev = ntb_ndev(ntb);
int bar;
+ if (pidx != NTB_DEF_PEER_IDX)
+ return -EINVAL;
+
bar = ndev_mw_to_bar(ndev, idx);
if (bar < 0)
return bar;
- if (base)
- *base = pci_resource_start(ndev->ntb.pdev, bar);
-
- if (size)
- *size = pci_resource_len(ndev->ntb.pdev, bar);
+ if (addr_align)
+ *addr_align = SZ_4K;
- if (align)
- *align = SZ_4K;
+ if (size_align)
+ *size_align = 1;
- if (align_size)
- *align_size = 1;
+ if (size_max)
+ *size_max = pci_resource_len(ndev->ntb.pdev, bar);
return 0;
}
-static int amd_ntb_mw_set_trans(struct ntb_dev *ntb, int idx,
+static int amd_ntb_mw_set_trans(struct ntb_dev *ntb, int pidx, int idx,
dma_addr_t addr, resource_size_t size)
{
struct amd_ntb_dev *ndev = ntb_ndev(ntb);
@@ -122,6 +126,9 @@ static int amd_ntb_mw_set_trans(struct ntb_dev *ntb, int idx,
u64 base_addr, limit, reg_val;
int bar;
+ if (pidx != NTB_DEF_PEER_IDX)
+ return -EINVAL;
+
bar = ndev_mw_to_bar(ndev, idx);
if (bar < 0)
return bar;
@@ -285,6 +292,31 @@ static int amd_ntb_link_disable(struct ntb_dev *ntb)
return 0;
}
+static int amd_ntb_peer_mw_count(struct ntb_dev *ntb)
+{
+ /* The same as for inbound MWs */
+ return ntb_ndev(ntb)->mw_count;
+}
+
+static int amd_ntb_peer_mw_get_addr(struct ntb_dev *ntb, int idx,
+ phys_addr_t *base, resource_size_t *size)
+{
+ struct amd_ntb_dev *ndev = ntb_ndev(ntb);
+ int bar;
+
+ bar = ndev_mw_to_bar(ndev, idx);
+ if (bar < 0)
+ return bar;
+
+ if (base)
+ *base = pci_resource_start(ndev->ntb.pdev, bar);
+
+ if (size)
+ *size = pci_resource_len(ndev->ntb.pdev, bar);
+
+ return 0;
+}
+
static u64 amd_ntb_db_valid_mask(struct ntb_dev *ntb)
{
return ntb_ndev(ntb)->db_valid_mask;
@@ -432,8 +464,10 @@ static int amd_ntb_peer_spad_write(struct ntb_dev *ntb,
static const struct ntb_dev_ops amd_ntb_ops = {
.mw_count = amd_ntb_mw_count,
- .mw_get_range = amd_ntb_mw_get_range,
+ .mw_get_align = amd_ntb_mw_get_align,
.mw_set_trans = amd_ntb_mw_set_trans,
+ .peer_mw_count = amd_ntb_peer_mw_count,
+ .peer_mw_get_addr = amd_ntb_peer_mw_get_addr,
.link_is_up = amd_ntb_link_is_up,
.link_enable = amd_ntb_link_enable,
.link_disable = amd_ntb_link_disable,
diff --git a/drivers/ntb/hw/intel/ntb_hw_intel.c b/drivers/ntb/hw/intel/ntb_hw_intel.c
index 725ffa4..4b84012 100644
--- a/drivers/ntb/hw/intel/ntb_hw_intel.c
+++ b/drivers/ntb/hw/intel/ntb_hw_intel.c
@@ -6,6 +6,7 @@
*
* Copyright(c) 2012 Intel Corporation. All rights reserved.
* Copyright (C) 2015 EMC Corporation. All Rights Reserved.
+ * Copyright (C) 2016 T-Platforms. All Rights Reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -15,6 +16,7 @@
*
* Copyright(c) 2012 Intel Corporation. All rights reserved.
* Copyright (C) 2015 EMC Corporation. All Rights Reserved.
+ * Copyright (C) 2016 T-Platforms. All Rights Reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
@@ -1035,20 +1037,26 @@ static void ndev_deinit_debugfs(struct intel_ntb_dev *ndev)
debugfs_remove_recursive(ndev->debugfs_dir);
}
-static int intel_ntb_mw_count(struct ntb_dev *ntb)
+static int intel_ntb_mw_count(struct ntb_dev *ntb, int pidx)
{
+ if (pidx != NTB_DEF_PEER_IDX)
+ return -EINVAL;
+
return ntb_ndev(ntb)->mw_count;
}
-static int intel_ntb_mw_get_range(struct ntb_dev *ntb, int idx,
- phys_addr_t *base,
- resource_size_t *size,
- resource_size_t *align,
- resource_size_t *align_size)
+static int intel_ntb_mw_get_align(struct ntb_dev *ntb, int pidx, int idx,
+ resource_size_t *addr_align,
+ resource_size_t *size_align,
+ resource_size_t *size_max)
{
struct intel_ntb_dev *ndev = ntb_ndev(ntb);
+ resource_size_t bar_size, mw_size;
int bar;
+ if (pidx != NTB_DEF_PEER_IDX)
+ return -EINVAL;
+
if (idx >= ndev->b2b_idx && !ndev->b2b_off)
idx += 1;
@@ -1056,24 +1064,26 @@ static int intel_ntb_mw_get_range(struct ntb_dev *ntb, int idx,
if (bar < 0)
return bar;
- if (base)
- *base = pci_resource_start(ndev->ntb.pdev, bar) +
- (idx == ndev->b2b_idx ? ndev->b2b_off : 0);
+ bar_size = pci_resource_len(ndev->ntb.pdev, bar);
- if (size)
- *size = pci_resource_len(ndev->ntb.pdev, bar) -
- (idx == ndev->b2b_idx ? ndev->b2b_off : 0);
+ if (idx == ndev->b2b_idx)
+ mw_size = bar_size - ndev->b2b_off;
+ else
+ mw_size = bar_size;
+
+ if (addr_align)
+ *addr_align = pci_resource_len(ndev->ntb.pdev, bar);
- if (align)
- *align = pci_resource_len(ndev->ntb.pdev, bar);
+ if (size_align)
+ *size_align = 1;
- if (align_size)
- *align_size = 1;
+ if (size_max)
+ *size_max = mw_size;
return 0;
}
-static int intel_ntb_mw_set_trans(struct ntb_dev *ntb, int idx,
+static int intel_ntb_mw_set_trans(struct ntb_dev *ntb, int pidx, int idx,
dma_addr_t addr, resource_size_t size)
{
struct intel_ntb_dev *ndev = ntb_ndev(ntb);
@@ -1083,6 +1093,9 @@ static int intel_ntb_mw_set_trans(struct ntb_dev *ntb, int idx,
u64 base, limit, reg_val;
int bar;
+ if (pidx != NTB_DEF_PEER_IDX)
+ return -EINVAL;
+
if (idx >= ndev->b2b_idx && !ndev->b2b_off)
idx += 1;
@@ -1249,6 +1262,36 @@ static int intel_ntb_link_disable(struct ntb_dev *ntb)
return 0;
}
+static int intel_ntb_peer_mw_count(struct ntb_dev *ntb)
+{
+ /* Numbers of inbound and outbound memory windows match */
+ return ntb_ndev(ntb)->mw_count;
+}
+
+static int intel_ntb_peer_mw_get_addr(struct ntb_dev *ntb, int idx,
+ phys_addr_t *base, resource_size_t *size)
+{
+ struct intel_ntb_dev *ndev = ntb_ndev(ntb);
+ int bar;
+
+ if (idx >= ndev->b2b_idx && !ndev->b2b_off)
+ idx += 1;
+
+ bar = ndev_mw_to_bar(ndev, idx);
+ if (bar < 0)
+ return bar;
+
+ if (base)
+ *base = pci_resource_start(ndev->ntb.pdev, bar) +
+ (idx == ndev->b2b_idx ? ndev->b2b_off : 0);
+
+ if (size)
+ *size = pci_resource_len(ndev->ntb.pdev, bar) -
+ (idx == ndev->b2b_idx ? ndev->b2b_off : 0);
+
+ return 0;
+}
+
static int intel_ntb_db_is_unsafe(struct ntb_dev *ntb)
{
return ndev_ignore_unsafe(ntb_ndev(ntb), NTB_UNSAFE_DB);
@@ -1880,7 +1923,7 @@ static int intel_ntb3_link_enable(struct ntb_dev *ntb,
return 0;
}
-static int intel_ntb3_mw_set_trans(struct ntb_dev *ntb, int idx,
+static int intel_ntb3_mw_set_trans(struct ntb_dev *ntb, int pidx, int idx,
dma_addr_t addr, resource_size_t size)
{
struct intel_ntb_dev *ndev = ntb_ndev(ntb);
@@ -1890,6 +1933,9 @@ static int intel_ntb3_mw_set_trans(struct ntb_dev *ntb, int idx,
u64 base, limit, reg_val;
int bar;
+ if (pidx != NTB_DEF_PEER_IDX)
+ return -EINVAL;
+
if (idx >= ndev->b2b_idx && !ndev->b2b_off)
idx += 1;
@@ -2884,8 +2930,10 @@ static const struct intel_ntb_xlat_reg skx_sec_xlat = {
/* operations for primary side of local ntb */
static const struct ntb_dev_ops intel_ntb_ops = {
.mw_count = intel_ntb_mw_count,
- .mw_get_range = intel_ntb_mw_get_range,
+ .mw_get_align = intel_ntb_mw_get_align,
.mw_set_trans = intel_ntb_mw_set_trans,
+ .peer_mw_count = intel_ntb_peer_mw_count,
+ .peer_mw_get_addr = intel_ntb_peer_mw_get_addr,
.link_is_up = intel_ntb_link_is_up,
.link_enable = intel_ntb_link_enable,
.link_disable = intel_ntb_link_disable,
@@ -2910,8 +2958,10 @@ static const struct ntb_dev_ops intel_ntb_ops = {
static const struct ntb_dev_ops intel_ntb3_ops = {
.mw_count = intel_ntb_mw_count,
- .mw_get_range = intel_ntb_mw_get_range,
+ .mw_get_align = intel_ntb_mw_get_align,
.mw_set_trans = intel_ntb3_mw_set_trans,
+ .peer_mw_count = intel_ntb_peer_mw_count,
+ .peer_mw_get_addr = intel_ntb_peer_mw_get_addr,
.link_is_up = intel_ntb_link_is_up,
.link_enable = intel_ntb3_link_enable,
.link_disable = intel_ntb_link_disable,
diff --git a/drivers/ntb/ntb.c b/drivers/ntb/ntb.c
index 1e92e52..2551bb2 100644
--- a/drivers/ntb/ntb.c
+++ b/drivers/ntb/ntb.c
@@ -5,6 +5,7 @@
* GPL LICENSE SUMMARY
*
* Copyright (C) 2015 EMC Corporation. All Rights Reserved.
+ * Copyright (C) 2016 T-Platforms. All Rights Reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -18,6 +19,7 @@
* BSD LICENSE
*
* Copyright (C) 2015 EMC Corporation. All Rights Reserved.
+ * Copyright (C) 2016 T-Platforms. All Rights Reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
diff --git a/drivers/ntb/ntb_transport.c b/drivers/ntb/ntb_transport.c
index 10518b7..4d5b160 100644
--- a/drivers/ntb/ntb_transport.c
+++ b/drivers/ntb/ntb_transport.c
@@ -685,7 +685,7 @@ static void ntb_free_mw(struct ntb_transport_ctx *nt, int num_mw)
if (!mw->virt_addr)
return;
- ntb_mw_clear_trans(nt->ndev, num_mw);
+ ntb_mw_clear_trans(nt->ndev, PIDX, num_mw);
dma_free_coherent(&pdev->dev, mw->buff_size,
mw->virt_addr, mw->dma_addr);
mw->xlat_size = 0;
@@ -742,7 +742,8 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw,
}
/* Notify HW the memory location of the receive buffer */
- rc = ntb_mw_set_trans(nt->ndev, num_mw, mw->dma_addr, mw->xlat_size);
+ rc = ntb_mw_set_trans(nt->ndev, PIDX, num_mw, mw->dma_addr,
+ mw->xlat_size);
if (rc) {
dev_err(&pdev->dev, "Unable to set mw%d translation", num_mw);
ntb_free_mw(nt, num_mw);
@@ -1072,13 +1073,18 @@ static int ntb_transport_probe(struct ntb_client *self, struct ntb_dev *ndev)
int node;
int rc, i;
- mw_count = ntb_mw_count(ndev);
+ mw_count = ntb_mw_count(ndev, PIDX);
if (ntb_spad_count(ndev) < (NUM_MWS + 1 + mw_count * 2)) {
dev_err(&ndev->dev, "Not enough scratch pad registers for %s",
NTB_TRANSPORT_NAME);
return -EIO;
}
+ if (!ndev->ops->mw_set_trans) {
+ dev_err(&ndev->dev, "Inbound MW based NTB API is required\n");
+ return -EINVAL;
+ }
+
if (ntb_db_is_unsafe(ndev))
dev_dbg(&ndev->dev,
"doorbell is unsafe, proceed anyway...\n");
@@ -1109,8 +1115,13 @@ static int ntb_transport_probe(struct ntb_client *self, struct ntb_dev *ndev)
for (i = 0; i < mw_count; i++) {
mw = &nt->mw_vec[i];
- rc = ntb_mw_get_range(ndev, i, &mw->phys_addr, &mw->phys_size,
- &mw->xlat_align, &mw->xlat_align_size);
+ rc = ntb_mw_get_align(ndev, PIDX, i, &mw->xlat_align,
+ &mw->xlat_align_size, NULL);
+ if (rc)
+ goto err1;
+
+ rc = ntb_peer_mw_get_addr(ndev, i, &mw->phys_addr,
+ &mw->phys_size);
if (rc)
goto err1;
diff --git a/drivers/ntb/test/ntb_perf.c b/drivers/ntb/test/ntb_perf.c
index c908b3a..cbff0b4 100644
--- a/drivers/ntb/test/ntb_perf.c
+++ b/drivers/ntb/test/ntb_perf.c
@@ -453,7 +453,7 @@ static void perf_free_mw(struct perf_ctx *perf)
if (!mw->virt_addr)
return;
- ntb_mw_clear_trans(perf->ntb, 0);
+ ntb_mw_clear_trans(perf->ntb, PIDX, 0);
dma_free_coherent(&pdev->dev, mw->buf_size,
mw->virt_addr, mw->dma_addr);
mw->xlat_size = 0;
@@ -489,7 +489,7 @@ static int perf_set_mw(struct perf_ctx *perf, resource_size_t size)
mw->buf_size = 0;
}
- rc = ntb_mw_set_trans(perf->ntb, 0, mw->dma_addr, mw->xlat_size);
+ rc = ntb_mw_set_trans(perf->ntb, PIDX, 0, mw->dma_addr, mw->xlat_size);
if (rc) {
dev_err(&perf->ntb->dev, "Unable to set mw0 translation\n");
perf_free_mw(perf);
@@ -560,8 +560,12 @@ static int perf_setup_mw(struct ntb_dev *ntb, struct perf_ctx *perf)
mw = &perf->mw;
- rc = ntb_mw_get_range(ntb, 0, &mw->phys_addr, &mw->phys_size,
- &mw->xlat_align, &mw->xlat_align_size);
+ rc = ntb_mw_get_align(ntb, PIDX, 0, &mw->xlat_align,
+ &mw->xlat_align_size, NULL);
+ if (rc)
+ return rc;
+
+ rc = ntb_peer_mw_get_addr(ntb, 0, &mw->phys_addr, &mw->phys_size);
if (rc)
return rc;
@@ -765,6 +769,11 @@ static int perf_probe(struct ntb_client *client, struct ntb_dev *ntb)
return -EIO;
}
+ if (!ntb->ops->mw_set_trans) {
+ dev_err(&ntb->dev, "Need inbound MW based NTB API\n");
+ return -EINVAL;
+ }
+
if (ntb_peer_port_count(ntb) != NTB_DEF_PEER_CNT)
dev_warn(&ntb->dev, "Multi-port NTB devices unsupported\n");
diff --git a/drivers/ntb/test/ntb_tool.c b/drivers/ntb/test/ntb_tool.c
index 690862d..cb69247 100644
--- a/drivers/ntb/test/ntb_tool.c
+++ b/drivers/ntb/test/ntb_tool.c
@@ -119,7 +119,8 @@ MODULE_VERSION(DRIVER_VERSION);
MODULE_AUTHOR(DRIVER_AUTHOR);
MODULE_DESCRIPTION(DRIVER_DESCRIPTION);
-#define MAX_MWS 16
+/* It is rare to have hadrware with greater than six MWs */
+#define MAX_MWS 6
/* Only two-ports devices are supported */
#define PIDX NTB_DEF_PEER_IDX
@@ -670,28 +671,27 @@ static int tool_setup_mw(struct tool_ctx *tc, int idx, size_t req_size)
{
int rc;
struct tool_mw *mw = &tc->mws[idx];
- phys_addr_t base;
- resource_size_t size, align, align_size;
+ resource_size_t size, align_addr, align_size;
char buf[16];
if (mw->peer)
return 0;
- rc = ntb_mw_get_range(tc->ntb, idx, &base, &size, &align,
- &align_size);
+ rc = ntb_mw_get_align(tc->ntb, PIDX, idx, &align_addr,
+ &align_size, &size);
if (rc)
return rc;
mw->size = min_t(resource_size_t, req_size, size);
- mw->size = round_up(mw->size, align);
+ mw->size = round_up(mw->size, align_addr);
mw->size = round_up(mw->size, align_size);
mw->peer = dma_alloc_coherent(&tc->ntb->pdev->dev, mw->size,
&mw->peer_dma, GFP_KERNEL);
- if (!mw->peer)
+ if (!mw->peer || !IS_ALIGNED(mw->peer_dma, align_addr))
return -ENOMEM;
- rc = ntb_mw_set_trans(tc->ntb, idx, mw->peer_dma, mw->size);
+ rc = ntb_mw_set_trans(tc->ntb, PIDX, idx, mw->peer_dma, mw->size);
if (rc)
goto err_free_dma;
@@ -718,7 +718,7 @@ static void tool_free_mw(struct tool_ctx *tc, int idx)
struct tool_mw *mw = &tc->mws[idx];
if (mw->peer) {
- ntb_mw_clear_trans(tc->ntb, idx);
+ ntb_mw_clear_trans(tc->ntb, PIDX, idx);
dma_free_coherent(&tc->ntb->pdev->dev, mw->size,
mw->peer,
mw->peer_dma);
@@ -744,8 +744,9 @@ static ssize_t tool_peer_mw_trans_read(struct file *filep,
phys_addr_t base;
resource_size_t mw_size;
- resource_size_t align;
+ resource_size_t align_addr;
resource_size_t align_size;
+ resource_size_t max_size;
buf_size = min_t(size_t, size, 512);
@@ -753,8 +754,9 @@ static ssize_t tool_peer_mw_trans_read(struct file *filep,
if (!buf)
return -ENOMEM;
- ntb_mw_get_range(mw->tc->ntb, mw->idx,
- &base, &mw_size, &align, &align_size);
+ ntb_mw_get_align(mw->tc->ntb, PIDX, mw->idx,
+ &align_addr, &align_size, &max_size);
+ ntb_peer_mw_get_addr(mw->tc->ntb, mw->idx, &base, &mw_size);
off += scnprintf(buf + off, buf_size - off,
"Peer MW %d Information:\n", mw->idx);
@@ -769,13 +771,17 @@ static ssize_t tool_peer_mw_trans_read(struct file *filep,
off += scnprintf(buf + off, buf_size - off,
"Alignment \t%lld\n",
- (unsigned long long)align);
+ (unsigned long long)align_addr);
off += scnprintf(buf + off, buf_size - off,
"Size Alignment \t%lld\n",
(unsigned long long)align_size);
off += scnprintf(buf + off, buf_size - off,
+ "Size Max \t%lld\n",
+ (unsigned long long)max_size);
+
+ off += scnprintf(buf + off, buf_size - off,
"Ready \t%c\n",
(mw->peer) ? 'Y' : 'N');
@@ -829,8 +835,7 @@ static int tool_init_mw(struct tool_ctx *tc, int idx)
phys_addr_t base;
int rc;
- rc = ntb_mw_get_range(tc->ntb, idx, &base, &mw->win_size,
- NULL, NULL);
+ rc = ntb_peer_mw_get_addr(tc->ntb, idx, &base, &mw->win_size);
if (rc)
return rc;
@@ -915,6 +920,12 @@ static int tool_probe(struct ntb_client *self, struct ntb_dev *ntb)
int rc;
int i;
+ if (!ntb->ops->mw_set_trans) {
+ dev_dbg(&ntb->dev, "need inbound MW based NTB API\n");
+ rc = -EINVAL;
+ goto err_tc;
+ }
+
if (ntb_db_is_unsafe(ntb))
dev_dbg(&ntb->dev, "doorbell is unsafe\n");
@@ -933,7 +944,7 @@ static int tool_probe(struct ntb_client *self, struct ntb_dev *ntb)
tc->ntb = ntb;
init_waitqueue_head(&tc->link_wq);
- tc->mw_count = min(ntb_mw_count(tc->ntb), MAX_MWS);
+ tc->mw_count = min(ntb_mw_count(tc->ntb, PIDX), MAX_MWS);
for (i = 0; i < tc->mw_count; i++) {
rc = tool_init_mw(tc, i);
if (rc)
diff --git a/include/linux/ntb.h b/include/linux/ntb.h
index 6eef109..f6ec88f 100644
--- a/include/linux/ntb.h
+++ b/include/linux/ntb.h
@@ -5,6 +5,7 @@
* GPL LICENSE SUMMARY
*
* Copyright (C) 2015 EMC Corporation. All Rights Reserved.
+ * Copyright (C) 2016 T-Platforms. All Rights Reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -18,6 +19,7 @@
* BSD LICENSE
*
* Copyright (C) 2015 EMC Corporation. All Rights Reserved.
+ * Copyright (C) 2016 T-Platforms. All Rights Reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
@@ -201,9 +203,13 @@ static inline int ntb_ctx_ops_is_valid(const struct ntb_ctx_ops *ops)
* @link_enable: See ntb_link_enable().
* @link_disable: See ntb_link_disable().
* @mw_count: See ntb_mw_count().
- * @mw_get_range: See ntb_mw_get_range().
+ * @mw_get_align: See ntb_mw_get_align().
* @mw_set_trans: See ntb_mw_set_trans().
* @mw_clear_trans: See ntb_mw_clear_trans().
+ * @peer_mw_count: See ntb_peer_mw_count().
+ * @peer_mw_get_addr: See ntb_peer_mw_get_addr().
+ * @peer_mw_set_trans: See ntb_peer_mw_set_trans().
+ * @peer_mw_clear_trans:See ntb_peer_mw_clear_trans().
* @db_is_unsafe: See ntb_db_is_unsafe().
* @db_valid_mask: See ntb_db_valid_mask().
* @db_vector_count: See ntb_db_vector_count().
@@ -241,13 +247,20 @@ struct ntb_dev_ops {
enum ntb_speed max_speed, enum ntb_width max_width);
int (*link_disable)(struct ntb_dev *ntb);
- int (*mw_count)(struct ntb_dev *ntb);
- int (*mw_get_range)(struct ntb_dev *ntb, int idx,
- phys_addr_t *base, resource_size_t *size,
- resource_size_t *align, resource_size_t *align_size);
- int (*mw_set_trans)(struct ntb_dev *ntb, int idx,
+ int (*mw_count)(struct ntb_dev *ntb, int pidx);
+ int (*mw_get_align)(struct ntb_dev *ntb, int pidx, int widx,
+ resource_size_t *addr_align,
+ resource_size_t *size_align,
+ resource_size_t *size_max);
+ int (*mw_set_trans)(struct ntb_dev *ntb, int pidx, int widx,
dma_addr_t addr, resource_size_t size);
- int (*mw_clear_trans)(struct ntb_dev *ntb, int idx);
+ int (*mw_clear_trans)(struct ntb_dev *ntb, int pidx, int widx);
+ int (*peer_mw_count)(struct ntb_dev *ntb);
+ int (*peer_mw_get_addr)(struct ntb_dev *ntb, int widx,
+ phys_addr_t *base, resource_size_t *size);
+ int (*peer_mw_set_trans)(struct ntb_dev *ntb, int pidx, int widx,
+ u64 addr, resource_size_t size);
+ int (*peer_mw_clear_trans)(struct ntb_dev *ntb, int pidx, int widx);
int (*db_is_unsafe)(struct ntb_dev *ntb);
u64 (*db_valid_mask)(struct ntb_dev *ntb);
@@ -295,9 +308,13 @@ static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops *ops)
ops->link_enable &&
ops->link_disable &&
ops->mw_count &&
- ops->mw_get_range &&
- ops->mw_set_trans &&
+ ops->mw_get_align &&
+ (ops->mw_set_trans ||
+ ops->peer_mw_set_trans) &&
/* ops->mw_clear_trans && */
+ ops->peer_mw_count &&
+ ops->peer_mw_get_addr &&
+ /* ops->peer_mw_clear_trans && */
/* ops->db_is_unsafe && */
ops->db_valid_mask &&
@@ -655,79 +672,180 @@ static inline int ntb_link_disable(struct ntb_dev *ntb)
}
/**
- * ntb_mw_count() - get the number of memory windows
+ * ntb_mw_count() - get the number of inbound memory windows, which could
+ * be created for a specified peer device
* @ntb: NTB device context.
+ * @pidx: Port index of peer device.
*
* Hardware and topology may support a different number of memory windows.
+ * Moreover different peer devices can support different number of memory
+ * windows. Simply speaking this method returns the number of possible inbound
+ * memory windows to share with specified peer device.
*
* Return: the number of memory windows.
*/
-static inline int ntb_mw_count(struct ntb_dev *ntb)
+static inline int ntb_mw_count(struct ntb_dev *ntb, int pidx)
{
- return ntb->ops->mw_count(ntb);
+ return ntb->ops->mw_count(ntb, pidx);
}
/**
- * ntb_mw_get_range() - get the range of a memory window
+ * ntb_mw_get_align() - get the restriction parameters of inbound memory window
* @ntb: NTB device context.
- * @idx: Memory window number.
- * @base: OUT - the base address for mapping the memory window
- * @size: OUT - the size for mapping the memory window
- * @align: OUT - the base alignment for translating the memory window
- * @align_size: OUT - the size alignment for translating the memory window
- *
- * Get the range of a memory window. NULL may be given for any output
- * parameter if the value is not needed. The base and size may be used for
- * mapping the memory window, to access the peer memory. The alignment and
- * size may be used for translating the memory window, for the peer to access
- * memory on the local system.
- *
- * Return: Zero on success, otherwise an error number.
+ * @pidx: Port index of peer device.
+ * @widx: Memory window index.
+ * @addr_align: OUT - the base alignment for translating the memory window
+ * @size_align: OUT - the size alignment for translating the memory window
+ * @size_max: OUT - the maximum size of the memory window
+ *
+ * Get the alignments of an inbound memory window with specified index.
+ * NULL may be given for any output parameter if the value is not needed.
+ * The alignment and size parameters may be used for allocation of proper
+ * shared memory.
+ *
+ * Return: Zero on success, otherwise a negative error number.
*/
-static inline int ntb_mw_get_range(struct ntb_dev *ntb, int idx,
- phys_addr_t *base, resource_size_t *size,
- resource_size_t *align, resource_size_t *align_size)
+static inline int ntb_mw_get_align(struct ntb_dev *ntb, int pidx, int widx,
+ resource_size_t *addr_align,
+ resource_size_t *size_align,
+ resource_size_t *size_max)
{
- return ntb->ops->mw_get_range(ntb, idx, base, size,
- align, align_size);
+ return ntb->ops->mw_get_align(ntb, pidx, widx, addr_align, size_align,
+ size_max);
}
/**
- * ntb_mw_set_trans() - set the translation of a memory window
+ * ntb_mw_set_trans() - set the translation of an inbound memory window
* @ntb: NTB device context.
- * @idx: Memory window number.
- * @addr: The dma address local memory to expose to the peer.
+ * @pidx: Port index of peer device.
+ * @widx: Memory window index.
+ * @addr: The dma address of local memory to expose to the peer.
* @size: The size of the local memory to expose to the peer.
*
* Set the translation of a memory window. The peer may access local memory
* through the window starting at the address, up to the size. The address
- * must be aligned to the alignment specified by ntb_mw_get_range(). The size
- * must be aligned to the size alignment specified by ntb_mw_get_range().
+ * and size must be aligned in compliance with restrictions of
+ * ntb_mw_get_align(). The region size should not exceed the size_max parameter
+ * of that method.
+ *
+ * This method may not be implemented due to the hardware specific memory
+ * windows interface.
*
* Return: Zero on success, otherwise an error number.
*/
-static inline int ntb_mw_set_trans(struct ntb_dev *ntb, int idx,
+static inline int ntb_mw_set_trans(struct ntb_dev *ntb, int pidx, int widx,
dma_addr_t addr, resource_size_t size)
{
- return ntb->ops->mw_set_trans(ntb, idx, addr, size);
+ if (!ntb->ops->mw_set_trans)
+ return 0;
+
+ return ntb->ops->mw_set_trans(ntb, pidx, widx, addr, size);
}
/**
- * ntb_mw_clear_trans() - clear the translation of a memory window
+ * ntb_mw_clear_trans() - clear the translation address of an inbound memory
+ * window
* @ntb: NTB device context.
- * @idx: Memory window number.
+ * @pidx: Port index of peer device.
+ * @widx: Memory window index.
*
- * Clear the translation of a memory window. The peer may no longer access
- * local memory through the window.
+ * Clear the translation of an inbound memory window. The peer may no longer
+ * access local memory through the window.
*
* Return: Zero on success, otherwise an error number.
*/
-static inline int ntb_mw_clear_trans(struct ntb_dev *ntb, int idx)
+static inline int ntb_mw_clear_trans(struct ntb_dev *ntb, int pidx, int widx)
{
if (!ntb->ops->mw_clear_trans)
- return ntb->ops->mw_set_trans(ntb, idx, 0, 0);
+ return ntb_mw_set_trans(ntb, pidx, widx, 0, 0);
+
+ return ntb->ops->mw_clear_trans(ntb, pidx, widx);
+}
+
+/**
+ * ntb_peer_mw_count() - get the number of outbound memory windows, which could
+ * be mapped to access a shared memory
+ * @ntb: NTB device context.
+ *
+ * Hardware and topology may support a different number of memory windows.
+ * This method returns the number of outbound memory windows supported by
+ * local device.
+ *
+ * Return: the number of memory windows.
+ */
+static inline int ntb_peer_mw_count(struct ntb_dev *ntb)
+{
+ return ntb->ops->peer_mw_count(ntb);
+}
+
+/**
+ * ntb_peer_mw_get_addr() - get map address of an outbound memory window
+ * @ntb: NTB device context.
+ * @widx: Memory window index (within ntb_peer_mw_count() return value).
+ * @base: OUT - the base address of mapping region.
+ * @size: OUT - the size of mapping region.
+ *
+ * Get base and size of memory region to map. NULL may be given for any output
+ * parameter if the value is not needed. The base and size may be used for
+ * mapping the memory window, to access the peer memory.
+ *
+ * Return: Zero on success, otherwise a negative error number.
+ */
+static inline int ntb_peer_mw_get_addr(struct ntb_dev *ntb, int widx,
+ phys_addr_t *base, resource_size_t *size)
+{
+ return ntb->ops->peer_mw_get_addr(ntb, widx, base, size);
+}
+
+/**
+ * ntb_peer_mw_set_trans() - set a translation address of a memory window
+ * retrieved from a peer device
+ * @ntb: NTB device context.
+ * @pidx: Port index of peer device the translation address received from.
+ * @widx: Memory window index.
+ * @addr: The dma address of the shared memory to access.
+ * @size: The size of the shared memory to access.
+ *
+ * Set the translation of an outbound memory window. The local device may
+ * access shared memory allocated by a peer device sent the address.
+ *
+ * This method may not be implemented due to the hardware specific memory
+ * windows interface, so a translation address can be only set on the side,
+ * where shared memory (inbound memory windows) is allocated.
+ *
+ * Return: Zero on success, otherwise an error number.
+ */
+static inline int ntb_peer_mw_set_trans(struct ntb_dev *ntb, int pidx, int widx,
+ u64 addr, resource_size_t size)
+{
+ if (!ntb->ops->peer_mw_set_trans)
+ return 0;
+
+ return ntb->ops->peer_mw_set_trans(ntb, pidx, widx, addr, size);
+}
+
+/**
+ * ntb_peer_mw_clear_trans() - clear the translation address of an outbound
+ * memory window
+ * @ntb: NTB device context.
+ * @pidx: Port index of peer device.
+ * @widx: Memory window index.
+ *
+ * Clear the translation of a outbound memory window. The local device may no
+ * longer access a shared memory through the window.
+ *
+ * This method may not be implemented due to the hardware specific memory
+ * windows interface.
+ *
+ * Return: Zero on success, otherwise an error number.
+ */
+static inline int ntb_peer_mw_clear_trans(struct ntb_dev *ntb, int pidx,
+ int widx)
+{
+ if (!ntb->ops->peer_mw_clear_trans)
+ return ntb_peer_mw_set_trans(ntb, pidx, widx, 0, 0);
- return ntb->ops->mw_clear_trans(ntb, idx);
+ return ntb->ops->peer_mw_clear_trans(ntb, pidx, widx);
}
/**
--
2.6.6
[toc] | [prev] | [next] | [standalone]
| From | "Allen Hubbe" <Allen.Hubbe@dell.com> |
|---|---|
| Date | 2016-12-14 08:10 +0100 |
| Subject | RE: [PATCH v3 4/9] NTB: Alter MW API to support multi-ports devices |
| Message-ID | <sObRo-3ug-19@gated-at.bofh.it> |
| In reply to | #1541576 |
From: Serge Semin
> Multi-port NTB devices permit to share a memory between all accessible peers.
> Memory Windows API is altered to correspondingly initialize and map memory
> windows for such devices:
> ntb_mw_count(pidx); - number of inbound memory windows, which can be allocated
> for shared buffer with specified peer device.
> ntb_mw_get_align(pidx, widx); - get alignment and size restriction parameters
> to properly allocate inbound memory region.
> ntb_peer_mw_count(); - get number of outbound memory windows.
> ntb_peer_mw_get_addr(widx); - get mapping address of an outbound memory window
>
> If hardware supports inbound translation configured on the local ntb port:
> ntb_mw_set_trans(pidx, widx); - set translation address of allocated inbound
> memory window so a peer device could access it.
> ntb_mw_clear_trans(pidx, widx); - clear the translation address of an inbound
> memory window.
>
> If hardware supports outbound translation configured on the peer ntb port:
> ntb_peer_mw_set_trans(pidx, widx); - set translation address of a memory
> window retrieved from a peer device
> ntb_peer_mw_clear_trans(pidx, widx); - clear the translation address of an
> outbound memory window
>
> Signed-off-by: Serge Semin <fancer.lancer@gmail.com>
Acked-by: Allen Hubbe <Allen.Hubbe@dell.com>
> ---
> drivers/ntb/hw/amd/ntb_hw_amd.c | 68 +++++++++---
> drivers/ntb/hw/intel/ntb_hw_intel.c | 90 ++++++++++++----
> drivers/ntb/ntb.c | 2 +
> drivers/ntb/ntb_transport.c | 21 +++-
> drivers/ntb/test/ntb_perf.c | 17 ++-
> drivers/ntb/test/ntb_tool.c | 43 +++++---
> include/linux/ntb.h | 208 ++++++++++++++++++++++++++++--------
> 7 files changed, 342 insertions(+), 107 deletions(-)
>
> diff --git a/drivers/ntb/hw/amd/ntb_hw_amd.c b/drivers/ntb/hw/amd/ntb_hw_amd.c
> index 4d8d0bd..6a41c38 100644
> --- a/drivers/ntb/hw/amd/ntb_hw_amd.c
> +++ b/drivers/ntb/hw/amd/ntb_hw_amd.c
> @@ -5,6 +5,7 @@
> * GPL LICENSE SUMMARY
> *
> * Copyright (C) 2016 Advanced Micro Devices, Inc. All Rights Reserved.
> + * Copyright (C) 2016 T-Platforms. All Rights Reserved.
> *
> * This program is free software; you can redistribute it and/or modify
> * it under the terms of version 2 of the GNU General Public License as
> @@ -13,6 +14,7 @@
> * BSD LICENSE
> *
> * Copyright (C) 2016 Advanced Micro Devices, Inc. All Rights Reserved.
> + * Copyright (C) 2016 T-Platforms. All Rights Reserved.
> *
> * Redistribution and use in source and binary forms, with or without
> * modification, are permitted provided that the following conditions
> @@ -79,40 +81,42 @@ static int ndev_mw_to_bar(struct amd_ntb_dev *ndev, int idx)
> return 1 << idx;
> }
>
> -static int amd_ntb_mw_count(struct ntb_dev *ntb)
> +static int amd_ntb_mw_count(struct ntb_dev *ntb, int pidx)
> {
> + if (pidx != NTB_DEF_PEER_IDX)
> + return -EINVAL;
> +
> return ntb_ndev(ntb)->mw_count;
> }
>
> -static int amd_ntb_mw_get_range(struct ntb_dev *ntb, int idx,
> - phys_addr_t *base,
> - resource_size_t *size,
> - resource_size_t *align,
> - resource_size_t *align_size)
> +static int amd_ntb_mw_get_align(struct ntb_dev *ntb, int pidx, int idx,
> + resource_size_t *addr_align,
> + resource_size_t *size_align,
> + resource_size_t *size_max)
> {
> struct amd_ntb_dev *ndev = ntb_ndev(ntb);
> int bar;
>
> + if (pidx != NTB_DEF_PEER_IDX)
> + return -EINVAL;
> +
> bar = ndev_mw_to_bar(ndev, idx);
> if (bar < 0)
> return bar;
>
> - if (base)
> - *base = pci_resource_start(ndev->ntb.pdev, bar);
> -
> - if (size)
> - *size = pci_resource_len(ndev->ntb.pdev, bar);
> + if (addr_align)
> + *addr_align = SZ_4K;
>
> - if (align)
> - *align = SZ_4K;
> + if (size_align)
> + *size_align = 1;
>
> - if (align_size)
> - *align_size = 1;
> + if (size_max)
> + *size_max = pci_resource_len(ndev->ntb.pdev, bar);
>
> return 0;
> }
>
> -static int amd_ntb_mw_set_trans(struct ntb_dev *ntb, int idx,
> +static int amd_ntb_mw_set_trans(struct ntb_dev *ntb, int pidx, int idx,
> dma_addr_t addr, resource_size_t size)
> {
> struct amd_ntb_dev *ndev = ntb_ndev(ntb);
> @@ -122,6 +126,9 @@ static int amd_ntb_mw_set_trans(struct ntb_dev *ntb, int idx,
> u64 base_addr, limit, reg_val;
> int bar;
>
> + if (pidx != NTB_DEF_PEER_IDX)
> + return -EINVAL;
> +
> bar = ndev_mw_to_bar(ndev, idx);
> if (bar < 0)
> return bar;
> @@ -285,6 +292,31 @@ static int amd_ntb_link_disable(struct ntb_dev *ntb)
> return 0;
> }
>
> +static int amd_ntb_peer_mw_count(struct ntb_dev *ntb)
> +{
> + /* The same as for inbound MWs */
> + return ntb_ndev(ntb)->mw_count;
> +}
> +
> +static int amd_ntb_peer_mw_get_addr(struct ntb_dev *ntb, int idx,
> + phys_addr_t *base, resource_size_t *size)
> +{
> + struct amd_ntb_dev *ndev = ntb_ndev(ntb);
> + int bar;
> +
> + bar = ndev_mw_to_bar(ndev, idx);
> + if (bar < 0)
> + return bar;
> +
> + if (base)
> + *base = pci_resource_start(ndev->ntb.pdev, bar);
> +
> + if (size)
> + *size = pci_resource_len(ndev->ntb.pdev, bar);
> +
> + return 0;
> +}
> +
> static u64 amd_ntb_db_valid_mask(struct ntb_dev *ntb)
> {
> return ntb_ndev(ntb)->db_valid_mask;
> @@ -432,8 +464,10 @@ static int amd_ntb_peer_spad_write(struct ntb_dev *ntb,
>
> static const struct ntb_dev_ops amd_ntb_ops = {
> .mw_count = amd_ntb_mw_count,
> - .mw_get_range = amd_ntb_mw_get_range,
> + .mw_get_align = amd_ntb_mw_get_align,
> .mw_set_trans = amd_ntb_mw_set_trans,
> + .peer_mw_count = amd_ntb_peer_mw_count,
> + .peer_mw_get_addr = amd_ntb_peer_mw_get_addr,
> .link_is_up = amd_ntb_link_is_up,
> .link_enable = amd_ntb_link_enable,
> .link_disable = amd_ntb_link_disable,
> diff --git a/drivers/ntb/hw/intel/ntb_hw_intel.c b/drivers/ntb/hw/intel/ntb_hw_intel.c
> index 725ffa4..4b84012 100644
> --- a/drivers/ntb/hw/intel/ntb_hw_intel.c
> +++ b/drivers/ntb/hw/intel/ntb_hw_intel.c
> @@ -6,6 +6,7 @@
> *
> * Copyright(c) 2012 Intel Corporation. All rights reserved.
> * Copyright (C) 2015 EMC Corporation. All Rights Reserved.
> + * Copyright (C) 2016 T-Platforms. All Rights Reserved.
> *
> * This program is free software; you can redistribute it and/or modify
> * it under the terms of version 2 of the GNU General Public License as
> @@ -15,6 +16,7 @@
> *
> * Copyright(c) 2012 Intel Corporation. All rights reserved.
> * Copyright (C) 2015 EMC Corporation. All Rights Reserved.
> + * Copyright (C) 2016 T-Platforms. All Rights Reserved.
> *
> * Redistribution and use in source and binary forms, with or without
> * modification, are permitted provided that the following conditions
> @@ -1035,20 +1037,26 @@ static void ndev_deinit_debugfs(struct intel_ntb_dev *ndev)
> debugfs_remove_recursive(ndev->debugfs_dir);
> }
>
> -static int intel_ntb_mw_count(struct ntb_dev *ntb)
> +static int intel_ntb_mw_count(struct ntb_dev *ntb, int pidx)
> {
> + if (pidx != NTB_DEF_PEER_IDX)
> + return -EINVAL;
> +
> return ntb_ndev(ntb)->mw_count;
> }
>
> -static int intel_ntb_mw_get_range(struct ntb_dev *ntb, int idx,
> - phys_addr_t *base,
> - resource_size_t *size,
> - resource_size_t *align,
> - resource_size_t *align_size)
> +static int intel_ntb_mw_get_align(struct ntb_dev *ntb, int pidx, int idx,
> + resource_size_t *addr_align,
> + resource_size_t *size_align,
> + resource_size_t *size_max)
> {
> struct intel_ntb_dev *ndev = ntb_ndev(ntb);
> + resource_size_t bar_size, mw_size;
> int bar;
>
> + if (pidx != NTB_DEF_PEER_IDX)
> + return -EINVAL;
> +
> if (idx >= ndev->b2b_idx && !ndev->b2b_off)
> idx += 1;
>
> @@ -1056,24 +1064,26 @@ static int intel_ntb_mw_get_range(struct ntb_dev *ntb, int idx,
> if (bar < 0)
> return bar;
>
> - if (base)
> - *base = pci_resource_start(ndev->ntb.pdev, bar) +
> - (idx == ndev->b2b_idx ? ndev->b2b_off : 0);
> + bar_size = pci_resource_len(ndev->ntb.pdev, bar);
>
> - if (size)
> - *size = pci_resource_len(ndev->ntb.pdev, bar) -
> - (idx == ndev->b2b_idx ? ndev->b2b_off : 0);
> + if (idx == ndev->b2b_idx)
> + mw_size = bar_size - ndev->b2b_off;
> + else
> + mw_size = bar_size;
> +
> + if (addr_align)
> + *addr_align = pci_resource_len(ndev->ntb.pdev, bar);
>
> - if (align)
> - *align = pci_resource_len(ndev->ntb.pdev, bar);
> + if (size_align)
> + *size_align = 1;
>
> - if (align_size)
> - *align_size = 1;
> + if (size_max)
> + *size_max = mw_size;
>
> return 0;
> }
>
> -static int intel_ntb_mw_set_trans(struct ntb_dev *ntb, int idx,
> +static int intel_ntb_mw_set_trans(struct ntb_dev *ntb, int pidx, int idx,
> dma_addr_t addr, resource_size_t size)
> {
> struct intel_ntb_dev *ndev = ntb_ndev(ntb);
> @@ -1083,6 +1093,9 @@ static int intel_ntb_mw_set_trans(struct ntb_dev *ntb, int idx,
> u64 base, limit, reg_val;
> int bar;
>
> + if (pidx != NTB_DEF_PEER_IDX)
> + return -EINVAL;
> +
> if (idx >= ndev->b2b_idx && !ndev->b2b_off)
> idx += 1;
>
> @@ -1249,6 +1262,36 @@ static int intel_ntb_link_disable(struct ntb_dev *ntb)
> return 0;
> }
>
> +static int intel_ntb_peer_mw_count(struct ntb_dev *ntb)
> +{
> + /* Numbers of inbound and outbound memory windows match */
> + return ntb_ndev(ntb)->mw_count;
> +}
> +
> +static int intel_ntb_peer_mw_get_addr(struct ntb_dev *ntb, int idx,
> + phys_addr_t *base, resource_size_t *size)
> +{
> + struct intel_ntb_dev *ndev = ntb_ndev(ntb);
> + int bar;
> +
> + if (idx >= ndev->b2b_idx && !ndev->b2b_off)
> + idx += 1;
> +
> + bar = ndev_mw_to_bar(ndev, idx);
> + if (bar < 0)
> + return bar;
> +
> + if (base)
> + *base = pci_resource_start(ndev->ntb.pdev, bar) +
> + (idx == ndev->b2b_idx ? ndev->b2b_off : 0);
> +
> + if (size)
> + *size = pci_resource_len(ndev->ntb.pdev, bar) -
> + (idx == ndev->b2b_idx ? ndev->b2b_off : 0);
> +
> + return 0;
> +}
> +
> static int intel_ntb_db_is_unsafe(struct ntb_dev *ntb)
> {
> return ndev_ignore_unsafe(ntb_ndev(ntb), NTB_UNSAFE_DB);
> @@ -1880,7 +1923,7 @@ static int intel_ntb3_link_enable(struct ntb_dev *ntb,
>
> return 0;
> }
> -static int intel_ntb3_mw_set_trans(struct ntb_dev *ntb, int idx,
> +static int intel_ntb3_mw_set_trans(struct ntb_dev *ntb, int pidx, int idx,
> dma_addr_t addr, resource_size_t size)
> {
> struct intel_ntb_dev *ndev = ntb_ndev(ntb);
> @@ -1890,6 +1933,9 @@ static int intel_ntb3_mw_set_trans(struct ntb_dev *ntb, int idx,
> u64 base, limit, reg_val;
> int bar;
>
> + if (pidx != NTB_DEF_PEER_IDX)
> + return -EINVAL;
> +
> if (idx >= ndev->b2b_idx && !ndev->b2b_off)
> idx += 1;
>
> @@ -2884,8 +2930,10 @@ static const struct intel_ntb_xlat_reg skx_sec_xlat = {
> /* operations for primary side of local ntb */
> static const struct ntb_dev_ops intel_ntb_ops = {
> .mw_count = intel_ntb_mw_count,
> - .mw_get_range = intel_ntb_mw_get_range,
> + .mw_get_align = intel_ntb_mw_get_align,
> .mw_set_trans = intel_ntb_mw_set_trans,
> + .peer_mw_count = intel_ntb_peer_mw_count,
> + .peer_mw_get_addr = intel_ntb_peer_mw_get_addr,
> .link_is_up = intel_ntb_link_is_up,
> .link_enable = intel_ntb_link_enable,
> .link_disable = intel_ntb_link_disable,
> @@ -2910,8 +2958,10 @@ static const struct ntb_dev_ops intel_ntb_ops = {
>
> static const struct ntb_dev_ops intel_ntb3_ops = {
> .mw_count = intel_ntb_mw_count,
> - .mw_get_range = intel_ntb_mw_get_range,
> + .mw_get_align = intel_ntb_mw_get_align,
> .mw_set_trans = intel_ntb3_mw_set_trans,
> + .peer_mw_count = intel_ntb_peer_mw_count,
> + .peer_mw_get_addr = intel_ntb_peer_mw_get_addr,
> .link_is_up = intel_ntb_link_is_up,
> .link_enable = intel_ntb3_link_enable,
> .link_disable = intel_ntb_link_disable,
> diff --git a/drivers/ntb/ntb.c b/drivers/ntb/ntb.c
> index 1e92e52..2551bb2 100644
> --- a/drivers/ntb/ntb.c
> +++ b/drivers/ntb/ntb.c
> @@ -5,6 +5,7 @@
> * GPL LICENSE SUMMARY
> *
> * Copyright (C) 2015 EMC Corporation. All Rights Reserved.
> + * Copyright (C) 2016 T-Platforms. All Rights Reserved.
> *
> * This program is free software; you can redistribute it and/or modify
> * it under the terms of version 2 of the GNU General Public License as
> @@ -18,6 +19,7 @@
> * BSD LICENSE
> *
> * Copyright (C) 2015 EMC Corporation. All Rights Reserved.
> + * Copyright (C) 2016 T-Platforms. All Rights Reserved.
> *
> * Redistribution and use in source and binary forms, with or without
> * modification, are permitted provided that the following conditions
> diff --git a/drivers/ntb/ntb_transport.c b/drivers/ntb/ntb_transport.c
> index 10518b7..4d5b160 100644
> --- a/drivers/ntb/ntb_transport.c
> +++ b/drivers/ntb/ntb_transport.c
> @@ -685,7 +685,7 @@ static void ntb_free_mw(struct ntb_transport_ctx *nt, int num_mw)
> if (!mw->virt_addr)
> return;
>
> - ntb_mw_clear_trans(nt->ndev, num_mw);
> + ntb_mw_clear_trans(nt->ndev, PIDX, num_mw);
> dma_free_coherent(&pdev->dev, mw->buff_size,
> mw->virt_addr, mw->dma_addr);
> mw->xlat_size = 0;
> @@ -742,7 +742,8 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw,
> }
>
> /* Notify HW the memory location of the receive buffer */
> - rc = ntb_mw_set_trans(nt->ndev, num_mw, mw->dma_addr, mw->xlat_size);
> + rc = ntb_mw_set_trans(nt->ndev, PIDX, num_mw, mw->dma_addr,
> + mw->xlat_size);
> if (rc) {
> dev_err(&pdev->dev, "Unable to set mw%d translation", num_mw);
> ntb_free_mw(nt, num_mw);
> @@ -1072,13 +1073,18 @@ static int ntb_transport_probe(struct ntb_client *self, struct
> ntb_dev *ndev)
> int node;
> int rc, i;
>
> - mw_count = ntb_mw_count(ndev);
> + mw_count = ntb_mw_count(ndev, PIDX);
> if (ntb_spad_count(ndev) < (NUM_MWS + 1 + mw_count * 2)) {
> dev_err(&ndev->dev, "Not enough scratch pad registers for %s",
> NTB_TRANSPORT_NAME);
> return -EIO;
> }
>
> + if (!ndev->ops->mw_set_trans) {
> + dev_err(&ndev->dev, "Inbound MW based NTB API is required\n");
> + return -EINVAL;
> + }
> +
> if (ntb_db_is_unsafe(ndev))
> dev_dbg(&ndev->dev,
> "doorbell is unsafe, proceed anyway...\n");
> @@ -1109,8 +1115,13 @@ static int ntb_transport_probe(struct ntb_client *self, struct
> ntb_dev *ndev)
> for (i = 0; i < mw_count; i++) {
> mw = &nt->mw_vec[i];
>
> - rc = ntb_mw_get_range(ndev, i, &mw->phys_addr, &mw->phys_size,
> - &mw->xlat_align, &mw->xlat_align_size);
> + rc = ntb_mw_get_align(ndev, PIDX, i, &mw->xlat_align,
> + &mw->xlat_align_size, NULL);
> + if (rc)
> + goto err1;
> +
> + rc = ntb_peer_mw_get_addr(ndev, i, &mw->phys_addr,
> + &mw->phys_size);
> if (rc)
> goto err1;
>
> diff --git a/drivers/ntb/test/ntb_perf.c b/drivers/ntb/test/ntb_perf.c
> index c908b3a..cbff0b4 100644
> --- a/drivers/ntb/test/ntb_perf.c
> +++ b/drivers/ntb/test/ntb_perf.c
> @@ -453,7 +453,7 @@ static void perf_free_mw(struct perf_ctx *perf)
> if (!mw->virt_addr)
> return;
>
> - ntb_mw_clear_trans(perf->ntb, 0);
> + ntb_mw_clear_trans(perf->ntb, PIDX, 0);
> dma_free_coherent(&pdev->dev, mw->buf_size,
> mw->virt_addr, mw->dma_addr);
> mw->xlat_size = 0;
> @@ -489,7 +489,7 @@ static int perf_set_mw(struct perf_ctx *perf, resource_size_t size)
> mw->buf_size = 0;
> }
>
> - rc = ntb_mw_set_trans(perf->ntb, 0, mw->dma_addr, mw->xlat_size);
> + rc = ntb_mw_set_trans(perf->ntb, PIDX, 0, mw->dma_addr, mw->xlat_size);
> if (rc) {
> dev_err(&perf->ntb->dev, "Unable to set mw0 translation\n");
> perf_free_mw(perf);
> @@ -560,8 +560,12 @@ static int perf_setup_mw(struct ntb_dev *ntb, struct perf_ctx *perf)
>
> mw = &perf->mw;
>
> - rc = ntb_mw_get_range(ntb, 0, &mw->phys_addr, &mw->phys_size,
> - &mw->xlat_align, &mw->xlat_align_size);
> + rc = ntb_mw_get_align(ntb, PIDX, 0, &mw->xlat_align,
> + &mw->xlat_align_size, NULL);
> + if (rc)
> + return rc;
> +
> + rc = ntb_peer_mw_get_addr(ntb, 0, &mw->phys_addr, &mw->phys_size);
> if (rc)
> return rc;
>
> @@ -765,6 +769,11 @@ static int perf_probe(struct ntb_client *client, struct ntb_dev *ntb)
> return -EIO;
> }
>
> + if (!ntb->ops->mw_set_trans) {
> + dev_err(&ntb->dev, "Need inbound MW based NTB API\n");
> + return -EINVAL;
> + }
> +
> if (ntb_peer_port_count(ntb) != NTB_DEF_PEER_CNT)
> dev_warn(&ntb->dev, "Multi-port NTB devices unsupported\n");
>
> diff --git a/drivers/ntb/test/ntb_tool.c b/drivers/ntb/test/ntb_tool.c
> index 690862d..cb69247 100644
> --- a/drivers/ntb/test/ntb_tool.c
> +++ b/drivers/ntb/test/ntb_tool.c
> @@ -119,7 +119,8 @@ MODULE_VERSION(DRIVER_VERSION);
> MODULE_AUTHOR(DRIVER_AUTHOR);
> MODULE_DESCRIPTION(DRIVER_DESCRIPTION);
>
> -#define MAX_MWS 16
> +/* It is rare to have hadrware with greater than six MWs */
> +#define MAX_MWS 6
> /* Only two-ports devices are supported */
> #define PIDX NTB_DEF_PEER_IDX
>
> @@ -670,28 +671,27 @@ static int tool_setup_mw(struct tool_ctx *tc, int idx, size_t
> req_size)
> {
> int rc;
> struct tool_mw *mw = &tc->mws[idx];
> - phys_addr_t base;
> - resource_size_t size, align, align_size;
> + resource_size_t size, align_addr, align_size;
> char buf[16];
>
> if (mw->peer)
> return 0;
>
> - rc = ntb_mw_get_range(tc->ntb, idx, &base, &size, &align,
> - &align_size);
> + rc = ntb_mw_get_align(tc->ntb, PIDX, idx, &align_addr,
> + &align_size, &size);
> if (rc)
> return rc;
>
> mw->size = min_t(resource_size_t, req_size, size);
> - mw->size = round_up(mw->size, align);
> + mw->size = round_up(mw->size, align_addr);
> mw->size = round_up(mw->size, align_size);
> mw->peer = dma_alloc_coherent(&tc->ntb->pdev->dev, mw->size,
> &mw->peer_dma, GFP_KERNEL);
>
> - if (!mw->peer)
> + if (!mw->peer || !IS_ALIGNED(mw->peer_dma, align_addr))
> return -ENOMEM;
>
> - rc = ntb_mw_set_trans(tc->ntb, idx, mw->peer_dma, mw->size);
> + rc = ntb_mw_set_trans(tc->ntb, PIDX, idx, mw->peer_dma, mw->size);
> if (rc)
> goto err_free_dma;
>
> @@ -718,7 +718,7 @@ static void tool_free_mw(struct tool_ctx *tc, int idx)
> struct tool_mw *mw = &tc->mws[idx];
>
> if (mw->peer) {
> - ntb_mw_clear_trans(tc->ntb, idx);
> + ntb_mw_clear_trans(tc->ntb, PIDX, idx);
> dma_free_coherent(&tc->ntb->pdev->dev, mw->size,
> mw->peer,
> mw->peer_dma);
> @@ -744,8 +744,9 @@ static ssize_t tool_peer_mw_trans_read(struct file *filep,
>
> phys_addr_t base;
> resource_size_t mw_size;
> - resource_size_t align;
> + resource_size_t align_addr;
> resource_size_t align_size;
> + resource_size_t max_size;
>
> buf_size = min_t(size_t, size, 512);
>
> @@ -753,8 +754,9 @@ static ssize_t tool_peer_mw_trans_read(struct file *filep,
> if (!buf)
> return -ENOMEM;
>
> - ntb_mw_get_range(mw->tc->ntb, mw->idx,
> - &base, &mw_size, &align, &align_size);
> + ntb_mw_get_align(mw->tc->ntb, PIDX, mw->idx,
> + &align_addr, &align_size, &max_size);
> + ntb_peer_mw_get_addr(mw->tc->ntb, mw->idx, &base, &mw_size);
>
> off += scnprintf(buf + off, buf_size - off,
> "Peer MW %d Information:\n", mw->idx);
> @@ -769,13 +771,17 @@ static ssize_t tool_peer_mw_trans_read(struct file *filep,
>
> off += scnprintf(buf + off, buf_size - off,
> "Alignment \t%lld\n",
> - (unsigned long long)align);
> + (unsigned long long)align_addr);
>
> off += scnprintf(buf + off, buf_size - off,
> "Size Alignment \t%lld\n",
> (unsigned long long)align_size);
>
> off += scnprintf(buf + off, buf_size - off,
> + "Size Max \t%lld\n",
> + (unsigned long long)max_size);
> +
> + off += scnprintf(buf + off, buf_size - off,
> "Ready \t%c\n",
> (mw->peer) ? 'Y' : 'N');
>
> @@ -829,8 +835,7 @@ static int tool_init_mw(struct tool_ctx *tc, int idx)
> phys_addr_t base;
> int rc;
>
> - rc = ntb_mw_get_range(tc->ntb, idx, &base, &mw->win_size,
> - NULL, NULL);
> + rc = ntb_peer_mw_get_addr(tc->ntb, idx, &base, &mw->win_size);
> if (rc)
> return rc;
>
> @@ -915,6 +920,12 @@ static int tool_probe(struct ntb_client *self, struct ntb_dev *ntb)
> int rc;
> int i;
>
> + if (!ntb->ops->mw_set_trans) {
> + dev_dbg(&ntb->dev, "need inbound MW based NTB API\n");
> + rc = -EINVAL;
> + goto err_tc;
> + }
> +
> if (ntb_db_is_unsafe(ntb))
> dev_dbg(&ntb->dev, "doorbell is unsafe\n");
>
> @@ -933,7 +944,7 @@ static int tool_probe(struct ntb_client *self, struct ntb_dev *ntb)
> tc->ntb = ntb;
> init_waitqueue_head(&tc->link_wq);
>
> - tc->mw_count = min(ntb_mw_count(tc->ntb), MAX_MWS);
> + tc->mw_count = min(ntb_mw_count(tc->ntb, PIDX), MAX_MWS);
> for (i = 0; i < tc->mw_count; i++) {
> rc = tool_init_mw(tc, i);
> if (rc)
> diff --git a/include/linux/ntb.h b/include/linux/ntb.h
> index 6eef109..f6ec88f 100644
> --- a/include/linux/ntb.h
> +++ b/include/linux/ntb.h
> @@ -5,6 +5,7 @@
> * GPL LICENSE SUMMARY
> *
> * Copyright (C) 2015 EMC Corporation. All Rights Reserved.
> + * Copyright (C) 2016 T-Platforms. All Rights Reserved.
> *
> * This program is free software; you can redistribute it and/or modify
> * it under the terms of version 2 of the GNU General Public License as
> @@ -18,6 +19,7 @@
> * BSD LICENSE
> *
> * Copyright (C) 2015 EMC Corporation. All Rights Reserved.
> + * Copyright (C) 2016 T-Platforms. All Rights Reserved.
> *
> * Redistribution and use in source and binary forms, with or without
> * modification, are permitted provided that the following conditions
> @@ -201,9 +203,13 @@ static inline int ntb_ctx_ops_is_valid(const struct ntb_ctx_ops *ops)
> * @link_enable: See ntb_link_enable().
> * @link_disable: See ntb_link_disable().
> * @mw_count: See ntb_mw_count().
> - * @mw_get_range: See ntb_mw_get_range().
> + * @mw_get_align: See ntb_mw_get_align().
> * @mw_set_trans: See ntb_mw_set_trans().
> * @mw_clear_trans: See ntb_mw_clear_trans().
> + * @peer_mw_count: See ntb_peer_mw_count().
> + * @peer_mw_get_addr: See ntb_peer_mw_get_addr().
> + * @peer_mw_set_trans: See ntb_peer_mw_set_trans().
> + * @peer_mw_clear_trans:See ntb_peer_mw_clear_trans().
> * @db_is_unsafe: See ntb_db_is_unsafe().
> * @db_valid_mask: See ntb_db_valid_mask().
> * @db_vector_count: See ntb_db_vector_count().
> @@ -241,13 +247,20 @@ struct ntb_dev_ops {
> enum ntb_speed max_speed, enum ntb_width max_width);
> int (*link_disable)(struct ntb_dev *ntb);
>
> - int (*mw_count)(struct ntb_dev *ntb);
> - int (*mw_get_range)(struct ntb_dev *ntb, int idx,
> - phys_addr_t *base, resource_size_t *size,
> - resource_size_t *align, resource_size_t *align_size);
> - int (*mw_set_trans)(struct ntb_dev *ntb, int idx,
> + int (*mw_count)(struct ntb_dev *ntb, int pidx);
> + int (*mw_get_align)(struct ntb_dev *ntb, int pidx, int widx,
> + resource_size_t *addr_align,
> + resource_size_t *size_align,
> + resource_size_t *size_max);
> + int (*mw_set_trans)(struct ntb_dev *ntb, int pidx, int widx,
> dma_addr_t addr, resource_size_t size);
> - int (*mw_clear_trans)(struct ntb_dev *ntb, int idx);
> + int (*mw_clear_trans)(struct ntb_dev *ntb, int pidx, int widx);
> + int (*peer_mw_count)(struct ntb_dev *ntb);
> + int (*peer_mw_get_addr)(struct ntb_dev *ntb, int widx,
> + phys_addr_t *base, resource_size_t *size);
> + int (*peer_mw_set_trans)(struct ntb_dev *ntb, int pidx, int widx,
> + u64 addr, resource_size_t size);
> + int (*peer_mw_clear_trans)(struct ntb_dev *ntb, int pidx, int widx);
>
> int (*db_is_unsafe)(struct ntb_dev *ntb);
> u64 (*db_valid_mask)(struct ntb_dev *ntb);
> @@ -295,9 +308,13 @@ static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops *ops)
> ops->link_enable &&
> ops->link_disable &&
> ops->mw_count &&
> - ops->mw_get_range &&
> - ops->mw_set_trans &&
> + ops->mw_get_align &&
> + (ops->mw_set_trans ||
> + ops->peer_mw_set_trans) &&
> /* ops->mw_clear_trans && */
> + ops->peer_mw_count &&
> + ops->peer_mw_get_addr &&
> + /* ops->peer_mw_clear_trans && */
>
> /* ops->db_is_unsafe && */
> ops->db_valid_mask &&
> @@ -655,79 +672,180 @@ static inline int ntb_link_disable(struct ntb_dev *ntb)
> }
>
> /**
> - * ntb_mw_count() - get the number of memory windows
> + * ntb_mw_count() - get the number of inbound memory windows, which could
> + * be created for a specified peer device
> * @ntb: NTB device context.
> + * @pidx: Port index of peer device.
> *
> * Hardware and topology may support a different number of memory windows.
> + * Moreover different peer devices can support different number of memory
> + * windows. Simply speaking this method returns the number of possible inbound
> + * memory windows to share with specified peer device.
> *
> * Return: the number of memory windows.
> */
> -static inline int ntb_mw_count(struct ntb_dev *ntb)
> +static inline int ntb_mw_count(struct ntb_dev *ntb, int pidx)
> {
> - return ntb->ops->mw_count(ntb);
> + return ntb->ops->mw_count(ntb, pidx);
> }
>
> /**
> - * ntb_mw_get_range() - get the range of a memory window
> + * ntb_mw_get_align() - get the restriction parameters of inbound memory window
> * @ntb: NTB device context.
> - * @idx: Memory window number.
> - * @base: OUT - the base address for mapping the memory window
> - * @size: OUT - the size for mapping the memory window
> - * @align: OUT - the base alignment for translating the memory window
> - * @align_size: OUT - the size alignment for translating the memory window
> - *
> - * Get the range of a memory window. NULL may be given for any output
> - * parameter if the value is not needed. The base and size may be used for
> - * mapping the memory window, to access the peer memory. The alignment and
> - * size may be used for translating the memory window, for the peer to access
> - * memory on the local system.
> - *
> - * Return: Zero on success, otherwise an error number.
> + * @pidx: Port index of peer device.
> + * @widx: Memory window index.
> + * @addr_align: OUT - the base alignment for translating the memory window
> + * @size_align: OUT - the size alignment for translating the memory window
> + * @size_max: OUT - the maximum size of the memory window
> + *
> + * Get the alignments of an inbound memory window with specified index.
> + * NULL may be given for any output parameter if the value is not needed.
> + * The alignment and size parameters may be used for allocation of proper
> + * shared memory.
> + *
> + * Return: Zero on success, otherwise a negative error number.
> */
> -static inline int ntb_mw_get_range(struct ntb_dev *ntb, int idx,
> - phys_addr_t *base, resource_size_t *size,
> - resource_size_t *align, resource_size_t *align_size)
> +static inline int ntb_mw_get_align(struct ntb_dev *ntb, int pidx, int widx,
> + resource_size_t *addr_align,
> + resource_size_t *size_align,
> + resource_size_t *size_max)
> {
> - return ntb->ops->mw_get_range(ntb, idx, base, size,
> - align, align_size);
> + return ntb->ops->mw_get_align(ntb, pidx, widx, addr_align, size_align,
> + size_max);
> }
>
> /**
> - * ntb_mw_set_trans() - set the translation of a memory window
> + * ntb_mw_set_trans() - set the translation of an inbound memory window
> * @ntb: NTB device context.
> - * @idx: Memory window number.
> - * @addr: The dma address local memory to expose to the peer.
> + * @pidx: Port index of peer device.
> + * @widx: Memory window index.
> + * @addr: The dma address of local memory to expose to the peer.
> * @size: The size of the local memory to expose to the peer.
> *
> * Set the translation of a memory window. The peer may access local memory
> * through the window starting at the address, up to the size. The address
> - * must be aligned to the alignment specified by ntb_mw_get_range(). The size
> - * must be aligned to the size alignment specified by ntb_mw_get_range().
> + * and size must be aligned in compliance with restrictions of
> + * ntb_mw_get_align(). The region size should not exceed the size_max parameter
> + * of that method.
> + *
> + * This method may not be implemented due to the hardware specific memory
> + * windows interface.
> *
> * Return: Zero on success, otherwise an error number.
> */
> -static inline int ntb_mw_set_trans(struct ntb_dev *ntb, int idx,
> +static inline int ntb_mw_set_trans(struct ntb_dev *ntb, int pidx, int widx,
> dma_addr_t addr, resource_size_t size)
> {
> - return ntb->ops->mw_set_trans(ntb, idx, addr, size);
> + if (!ntb->ops->mw_set_trans)
> + return 0;
> +
> + return ntb->ops->mw_set_trans(ntb, pidx, widx, addr, size);
> }
>
> /**
> - * ntb_mw_clear_trans() - clear the translation of a memory window
> + * ntb_mw_clear_trans() - clear the translation address of an inbound memory
> + * window
> * @ntb: NTB device context.
> - * @idx: Memory window number.
> + * @pidx: Port index of peer device.
> + * @widx: Memory window index.
> *
> - * Clear the translation of a memory window. The peer may no longer access
> - * local memory through the window.
> + * Clear the translation of an inbound memory window. The peer may no longer
> + * access local memory through the window.
> *
> * Return: Zero on success, otherwise an error number.
> */
> -static inline int ntb_mw_clear_trans(struct ntb_dev *ntb, int idx)
> +static inline int ntb_mw_clear_trans(struct ntb_dev *ntb, int pidx, int widx)
> {
> if (!ntb->ops->mw_clear_trans)
> - return ntb->ops->mw_set_trans(ntb, idx, 0, 0);
> + return ntb_mw_set_trans(ntb, pidx, widx, 0, 0);
> +
> + return ntb->ops->mw_clear_trans(ntb, pidx, widx);
> +}
> +
> +/**
> + * ntb_peer_mw_count() - get the number of outbound memory windows, which could
> + * be mapped to access a shared memory
> + * @ntb: NTB device context.
> + *
> + * Hardware and topology may support a different number of memory windows.
> + * This method returns the number of outbound memory windows supported by
> + * local device.
> + *
> + * Return: the number of memory windows.
> + */
> +static inline int ntb_peer_mw_count(struct ntb_dev *ntb)
> +{
> + return ntb->ops->peer_mw_count(ntb);
> +}
> +
> +/**
> + * ntb_peer_mw_get_addr() - get map address of an outbound memory window
> + * @ntb: NTB device context.
> + * @widx: Memory window index (within ntb_peer_mw_count() return value).
> + * @base: OUT - the base address of mapping region.
> + * @size: OUT - the size of mapping region.
> + *
> + * Get base and size of memory region to map. NULL may be given for any output
> + * parameter if the value is not needed. The base and size may be used for
> + * mapping the memory window, to access the peer memory.
> + *
> + * Return: Zero on success, otherwise a negative error number.
> + */
> +static inline int ntb_peer_mw_get_addr(struct ntb_dev *ntb, int widx,
> + phys_addr_t *base, resource_size_t *size)
> +{
> + return ntb->ops->peer_mw_get_addr(ntb, widx, base, size);
> +}
> +
> +/**
> + * ntb_peer_mw_set_trans() - set a translation address of a memory window
> + * retrieved from a peer device
> + * @ntb: NTB device context.
> + * @pidx: Port index of peer device the translation address received from.
> + * @widx: Memory window index.
> + * @addr: The dma address of the shared memory to access.
> + * @size: The size of the shared memory to access.
> + *
> + * Set the translation of an outbound memory window. The local device may
> + * access shared memory allocated by a peer device sent the address.
> + *
> + * This method may not be implemented due to the hardware specific memory
> + * windows interface, so a translation address can be only set on the side,
> + * where shared memory (inbound memory windows) is allocated.
> + *
> + * Return: Zero on success, otherwise an error number.
> + */
> +static inline int ntb_peer_mw_set_trans(struct ntb_dev *ntb, int pidx, int widx,
> + u64 addr, resource_size_t size)
> +{
> + if (!ntb->ops->peer_mw_set_trans)
> + return 0;
> +
> + return ntb->ops->peer_mw_set_trans(ntb, pidx, widx, addr, size);
> +}
> +
> +/**
> + * ntb_peer_mw_clear_trans() - clear the translation address of an outbound
> + * memory window
> + * @ntb: NTB device context.
> + * @pidx: Port index of peer device.
> + * @widx: Memory window index.
> + *
> + * Clear the translation of a outbound memory window. The local device may no
> + * longer access a shared memory through the window.
> + *
> + * This method may not be implemented due to the hardware specific memory
> + * windows interface.
> + *
> + * Return: Zero on success, otherwise an error number.
> + */
> +static inline int ntb_peer_mw_clear_trans(struct ntb_dev *ntb, int pidx,
> + int widx)
> +{
> + if (!ntb->ops->peer_mw_clear_trans)
> + return ntb_peer_mw_set_trans(ntb, pidx, widx, 0, 0);
>
> - return ntb->ops->mw_clear_trans(ntb, idx);
> + return ntb->ops->peer_mw_clear_trans(ntb, pidx, widx);
> }
>
> /**
> --
> 2.6.6
[toc] | [prev] | [next] | [standalone]
| From | Serge Semin <fancer.lancer@gmail.com> |
|---|---|
| Date | 2016-12-14 01:00 +0100 |
| Subject | [PATCH v3 3/9] NTB: Alter link-state API to support multi-port devices |
| Message-ID | <sO59g-7zL-23@gated-at.bofh.it> |
| In reply to | #1541574 |
Multi-port devices permit the NTB connections between multiple domains,
so a local device can have NTB link being up with one peer and being
down with another. NTB link-state API is appropriately altered to return
a bitfield of the link-states between the local device and possible peers.
Acked-by: Allen Hubbe <Allen.Hubbe@dell.com>
Signed-off-by: Serge Semin <fancer.lancer@gmail.com>
---
drivers/ntb/hw/amd/ntb_hw_amd.c | 2 +-
drivers/ntb/hw/intel/ntb_hw_intel.c | 2 +-
include/linux/ntb.h | 31 ++++++++++++++++---------------
3 files changed, 18 insertions(+), 17 deletions(-)
diff --git a/drivers/ntb/hw/amd/ntb_hw_amd.c b/drivers/ntb/hw/amd/ntb_hw_amd.c
index 6ccba0d..4d8d0bd 100644
--- a/drivers/ntb/hw/amd/ntb_hw_amd.c
+++ b/drivers/ntb/hw/amd/ntb_hw_amd.c
@@ -213,7 +213,7 @@ static int amd_link_is_up(struct amd_ntb_dev *ndev)
return 0;
}
-static int amd_ntb_link_is_up(struct ntb_dev *ntb,
+static u64 amd_ntb_link_is_up(struct ntb_dev *ntb,
enum ntb_speed *speed,
enum ntb_width *width)
{
diff --git a/drivers/ntb/hw/intel/ntb_hw_intel.c b/drivers/ntb/hw/intel/ntb_hw_intel.c
index d2ce280..725ffa4 100644
--- a/drivers/ntb/hw/intel/ntb_hw_intel.c
+++ b/drivers/ntb/hw/intel/ntb_hw_intel.c
@@ -1171,7 +1171,7 @@ static int intel_ntb_mw_set_trans(struct ntb_dev *ntb, int idx,
return 0;
}
-static int intel_ntb_link_is_up(struct ntb_dev *ntb,
+static u64 intel_ntb_link_is_up(struct ntb_dev *ntb,
enum ntb_speed *speed,
enum ntb_width *width)
{
diff --git a/include/linux/ntb.h b/include/linux/ntb.h
index fe175c7..6eef109 100644
--- a/include/linux/ntb.h
+++ b/include/linux/ntb.h
@@ -235,7 +235,7 @@ struct ntb_dev_ops {
int (*peer_port_number)(struct ntb_dev *ntb, int pidx);
int (*peer_port_idx)(struct ntb_dev *ntb, int port);
- int (*link_is_up)(struct ntb_dev *ntb,
+ u64 (*link_is_up)(struct ntb_dev *ntb,
enum ntb_speed *speed, enum ntb_width *width);
int (*link_enable)(struct ntb_dev *ntb,
enum ntb_speed max_speed, enum ntb_width max_width);
@@ -607,25 +607,26 @@ static inline int ntb_peer_port_idx(struct ntb_dev *ntb, int port)
* state once after every link event. It is safe to query the link state in
* the context of the link event callback.
*
- * Return: One if the link is up, zero if the link is down, otherwise a
- * negative value indicating the error number.
+ * Return: bitfield of indexed ports link state: bit is set/cleared if the
+ * link is up/down respectively.
*/
-static inline int ntb_link_is_up(struct ntb_dev *ntb,
+static inline u64 ntb_link_is_up(struct ntb_dev *ntb,
enum ntb_speed *speed, enum ntb_width *width)
{
return ntb->ops->link_is_up(ntb, speed, width);
}
/**
- * ntb_link_enable() - enable the link on the secondary side of the ntb
+ * ntb_link_enable() - enable the local port ntb connection
* @ntb: NTB device context.
* @max_speed: The maximum link speed expressed as PCIe generation number.
* @max_width: The maximum link width expressed as the number of PCIe lanes.
*
- * Enable the link on the secondary side of the ntb. This can only be done
- * from the primary side of the ntb in primary or b2b topology. The ntb device
- * should train the link to its maximum speed and width, or the requested speed
- * and width, whichever is smaller, if supported.
+ * Enable the NTB/PCIe link on the local or remote (for bridge-to-bridge
+ * topology) side of the bridge. If it's supported the ntb device should train
+ * the link to its maximum speed and width, or the requested speed and width,
+ * whichever is smaller. Some hardware doesn't support PCIe link training, so
+ * the last two arguments will be ignored then.
*
* Return: Zero on success, otherwise an error number.
*/
@@ -637,14 +638,14 @@ static inline int ntb_link_enable(struct ntb_dev *ntb,
}
/**
- * ntb_link_disable() - disable the link on the secondary side of the ntb
+ * ntb_link_disable() - disable the local port ntb connection
* @ntb: NTB device context.
*
- * Disable the link on the secondary side of the ntb. This can only be
- * done from the primary side of the ntb in primary or b2b topology. The ntb
- * device should disable the link. Returning from this call must indicate that
- * a barrier has passed, though with no more writes may pass in either
- * direction across the link, except if this call returns an error number.
+ * Disable the link on the local or remote (for b2b topology) of the ntb.
+ * The ntb device should disable the link. Returning from this call must
+ * indicate that a barrier has passed, though with no more writes may pass in
+ * either direction across the link, except if this call returns an error
+ * number.
*
* Return: Zero on success, otherwise an error number.
*/
--
2.6.6
[toc] | [prev] | [next] | [standalone]
| From | Serge Semin <fancer.lancer@gmail.com> |
|---|---|
| Date | 2016-12-14 01:00 +0100 |
| Subject | [PATCH v3 6/9] NTB: Add Messaging NTB API |
| Message-ID | <sO59g-7zL-29@gated-at.bofh.it> |
| In reply to | #1541574 |
Some IDT NTB-capable PCIe-switches have message registers to communicate with
peer devices. This patch adds new NTB API callback methods, which can be used
to utilize these registers functionality:
ntb_msg_count(); - get number of message registers
ntb_msg_inbits(); - get bitfield of inbound message registers status
ntb_msg_outbits(); - get bitfield of outbound message registers status
ntb_msg_read_sts(); - read the inbound and outbound message registers status
ntb_msg_clear_sts(); - clear status bits of message registers
ntb_msg_set_mask(); - mask interrupts raised by status bits of message
registers.
ntb_msg_clear_mask(); - clear interrupts mask bits of message registers
ntb_msg_read(midx, *pidx); - read message register with specified index,
additionally getting peer port index which data received from
ntb_msg_write(midx, pidx); - write data to the specified message register
sending it to the passed peer device connected over a pidx port
ntb_msg_event(); - notify driver context of a new message event
Of course there is hardware which doesn't support Message registers, so
this API is made optional.
Acked-by: Allen Hubbe <Allen.Hubbe@dell.com>
Signed-off-by: Serge Semin <fancer.lancer@gmail.com>
---
drivers/ntb/ntb.c | 13 ++++
include/linux/ntb.h | 205 ++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 218 insertions(+)
diff --git a/drivers/ntb/ntb.c b/drivers/ntb/ntb.c
index 2551bb2..03b80d8 100644
--- a/drivers/ntb/ntb.c
+++ b/drivers/ntb/ntb.c
@@ -193,6 +193,19 @@ void ntb_db_event(struct ntb_dev *ntb, int vector)
}
EXPORT_SYMBOL(ntb_db_event);
+void ntb_msg_event(struct ntb_dev *ntb)
+{
+ unsigned long irqflags;
+
+ spin_lock_irqsave(&ntb->ctx_lock, irqflags);
+ {
+ if (ntb->ctx_ops && ntb->ctx_ops->msg_event)
+ ntb->ctx_ops->msg_event(ntb->ctx);
+ }
+ spin_unlock_irqrestore(&ntb->ctx_lock, irqflags);
+}
+EXPORT_SYMBOL(ntb_msg_event);
+
int ntb_default_port_number(struct ntb_dev *ntb)
{
switch (ntb->topo) {
diff --git a/include/linux/ntb.h b/include/linux/ntb.h
index a54e2be..76c56d5 100644
--- a/include/linux/ntb.h
+++ b/include/linux/ntb.h
@@ -178,10 +178,12 @@ static inline int ntb_client_ops_is_valid(const struct ntb_client_ops *ops)
* struct ntb_ctx_ops - ntb driver context operations
* @link_event: See ntb_link_event().
* @db_event: See ntb_db_event().
+ * @msg_event: See ntb_msg_event().
*/
struct ntb_ctx_ops {
void (*link_event)(void *ctx);
void (*db_event)(void *ctx, int db_vector);
+ void (*msg_event)(void *ctx);
};
static inline int ntb_ctx_ops_is_valid(const struct ntb_ctx_ops *ops)
@@ -190,6 +192,7 @@ static inline int ntb_ctx_ops_is_valid(const struct ntb_ctx_ops *ops)
return
/* ops->link_event && */
/* ops->db_event && */
+ /* ops->msg_event && */
1;
}
@@ -234,6 +237,15 @@ static inline int ntb_ctx_ops_is_valid(const struct ntb_ctx_ops *ops)
* @peer_spad_addr: See ntb_peer_spad_addr().
* @peer_spad_read: See ntb_peer_spad_read().
* @peer_spad_write: See ntb_peer_spad_write().
+ * @msg_count: See ntb_msg_count().
+ * @msg_inbits: See ntb_msg_inbits().
+ * @msg_outbits: See ntb_msg_outbits().
+ * @msg_read_sts: See ntb_msg_read_sts().
+ * @msg_clear_sts: See ntb_msg_clear_sts().
+ * @msg_set_mask: See ntb_msg_set_mask().
+ * @msg_clear_mask: See ntb_msg_clear_mask().
+ * @msg_read: See ntb_msg_read().
+ * @msg_write: See ntb_msg_write().
*/
struct ntb_dev_ops {
int (*port_number)(struct ntb_dev *ntb);
@@ -296,6 +308,16 @@ struct ntb_dev_ops {
u32 (*peer_spad_read)(struct ntb_dev *ntb, int pidx, int sidx);
int (*peer_spad_write)(struct ntb_dev *ntb, int pidx, int sidx,
u32 val);
+
+ int (*msg_count)(struct ntb_dev *ntb);
+ u64 (*msg_inbits)(struct ntb_dev *ntb);
+ u64 (*msg_outbits)(struct ntb_dev *ntb);
+ u64 (*msg_read_sts)(struct ntb_dev *ntb);
+ int (*msg_clear_sts)(struct ntb_dev *ntb, u64 sts_bits);
+ int (*msg_set_mask)(struct ntb_dev *ntb, u64 mask_bits);
+ int (*msg_clear_mask)(struct ntb_dev *ntb, u64 mask_bits);
+ int (*msg_read)(struct ntb_dev *ntb, int midx, int *pidx, u32 *msg);
+ int (*msg_write)(struct ntb_dev *ntb, int midx, int pidx, u32 msg);
};
static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops *ops)
@@ -342,6 +364,15 @@ static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops *ops)
/* !ops->peer_spad_addr == !ops->spad_count && */
/* !ops->peer_spad_read == !ops->spad_count && */
!ops->peer_spad_write == !ops->spad_count &&
+
+ !ops->msg_inbits == !ops->msg_count &&
+ !ops->msg_outbits == !ops->msg_count &&
+ !ops->msg_read_sts == !ops->msg_count &&
+ !ops->msg_clear_sts == !ops->msg_count &&
+ /* !ops->msg_set_mask == !ops->msg_count && */
+ /* !ops->msg_clear_mask == !ops->msg_count && */
+ !ops->msg_read == !ops->msg_count &&
+ !ops->msg_write == !ops->msg_count &&
1;
}
@@ -485,6 +516,18 @@ void ntb_link_event(struct ntb_dev *ntb);
void ntb_db_event(struct ntb_dev *ntb, int vector);
/**
+ * ntb_msg_event() - notify driver context of a message event
+ * @ntb: NTB device context.
+ *
+ * Notify the driver context of a message event. If hardware supports
+ * message registers, this event indicates, that a new message arrived in
+ * some incoming message register or last sent message couldn't be delivered.
+ * The events can be masked/unmasked by the methods ntb_msg_set_mask() and
+ * ntb_msg_clear_mask().
+ */
+void ntb_msg_event(struct ntb_dev *ntb);
+
+/**
* ntb_default_port_number() - get the default local port number
* @ntb: NTB device context.
*
@@ -1282,4 +1325,166 @@ static inline int ntb_peer_spad_write(struct ntb_dev *ntb, int pidx, int sidx,
return ntb->ops->peer_spad_write(ntb, pidx, sidx, val);
}
+/**
+ * ntb_msg_count() - get the number of message registers
+ * @ntb: NTB device context.
+ *
+ * Hardware may support a different number of message registers.
+ *
+ * Return: the number of message registers.
+ */
+static inline int ntb_msg_count(struct ntb_dev *ntb)
+{
+ if (!ntb->ops->msg_count)
+ return 0;
+
+ return ntb->ops->msg_count(ntb);
+}
+
+/**
+ * ntb_msg_inbits() - get a bitfield of inbound message registers status
+ * @ntb: NTB device context.
+ *
+ * The method returns the bitfield of status and mask registers, which related
+ * to inbound message registers.
+ *
+ * Return: bitfield of inbound message registers.
+ */
+static inline u64 ntb_msg_inbits(struct ntb_dev *ntb)
+{
+ if (!ntb->ops->msg_inbits)
+ return 0;
+
+ return ntb->ops->msg_inbits(ntb);
+}
+
+/**
+ * ntb_msg_outbits() - get a bitfield of outbound message registers status
+ * @ntb: NTB device context.
+ *
+ * The method returns the bitfield of status and mask registers, which related
+ * to outbound message registers.
+ *
+ * Return: bitfield of outbound message registers.
+ */
+static inline u64 ntb_msg_outbits(struct ntb_dev *ntb)
+{
+ if (!ntb->ops->msg_outbits)
+ return 0;
+
+ return ntb->ops->msg_outbits(ntb);
+}
+
+/**
+ * ntb_msg_read_sts() - read the message registers status
+ * @ntb: NTB device context.
+ *
+ * Read the status of message register. Inbound and outbound message registers
+ * related bits can be filtered by masks retrieved from ntb_msg_inbits() and
+ * ntb_msg_outbits().
+ *
+ * Return: status bits of message registers
+ */
+static inline u64 ntb_msg_read_sts(struct ntb_dev *ntb)
+{
+ if (!ntb->ops->msg_read_sts)
+ return 0;
+
+ return ntb->ops->msg_read_sts(ntb);
+}
+
+/**
+ * ntb_msg_clear_sts() - clear status bits of message registers
+ * @ntb: NTB device context.
+ * @sts_bits: Status bits to clear.
+ *
+ * Clear bits in the status register.
+ *
+ * Return: Zero on success, otherwise a negative error number.
+ */
+static inline int ntb_msg_clear_sts(struct ntb_dev *ntb, u64 sts_bits)
+{
+ if (!ntb->ops->msg_clear_sts)
+ return -EINVAL;
+
+ return ntb->ops->msg_clear_sts(ntb, sts_bits);
+}
+
+/**
+ * ntb_msg_set_mask() - set mask of message register status bits
+ * @ntb: NTB device context.
+ * @mask_bits: Mask bits.
+ *
+ * Mask the message registers status bits from raising the message event.
+ *
+ * Return: Zero on success, otherwise a negative error number.
+ */
+static inline int ntb_msg_set_mask(struct ntb_dev *ntb, u64 mask_bits)
+{
+ if (!ntb->ops->msg_set_mask)
+ return -EINVAL;
+
+ return ntb->ops->msg_set_mask(ntb, mask_bits);
+}
+
+/**
+ * ntb_msg_clear_mask() - clear message registers mask
+ * @ntb: NTB device context.
+ * @mask_bits: Mask bits to clear.
+ *
+ * Clear bits in the message events mask register.
+ *
+ * Return: Zero on success, otherwise a negative error number.
+ */
+static inline int ntb_msg_clear_mask(struct ntb_dev *ntb, u64 mask_bits)
+{
+ if (!ntb->ops->msg_clear_mask)
+ return -EINVAL;
+
+ return ntb->ops->msg_clear_mask(ntb, mask_bits);
+}
+
+/**
+ * ntb_msg_read() - read message register with specified index
+ * @ntb: NTB device context.
+ * @midx: Message register index
+ * @pidx: OUT - Port index of peer device a message retrieved from
+ * @msg: OUT - Data
+ *
+ * Read data from the specified message register. Source port index of a
+ * message is retrieved as well.
+ *
+ * Return: Zero on success, otherwise a negative error number.
+ */
+static inline int ntb_msg_read(struct ntb_dev *ntb, int midx, int *pidx,
+ u32 *msg)
+{
+ if (!ntb->ops->msg_read)
+ return -EINVAL;
+
+ return ntb->ops->msg_read(ntb, midx, pidx, msg);
+}
+
+/**
+ * ntb_msg_write() - write data to the specified message register
+ * @ntb: NTB device context.
+ * @midx: Message register index
+ * @pidx: Port index of peer device a message being sent to
+ * @msg: Data to send
+ *
+ * Send data to a specified peer device using the defined message register.
+ * Message event can be raised if the midx registers isn't empty while
+ * calling this method and the corresponding interrupt isn't masked.
+ *
+ * Return: Zero on success, otherwise a negative error number.
+ */
+static inline int ntb_msg_write(struct ntb_dev *ntb, int midx, int pidx,
+ u32 msg)
+{
+ if (!ntb->ops->msg_write)
+ return -EINVAL;
+
+ return ntb->ops->msg_write(ntb, midx, pidx, msg);
+}
+
#endif
--
2.6.6
[toc] | [prev] | [next] | [standalone]
| From | Serge Semin <fancer.lancer@gmail.com> |
|---|---|
| Date | 2016-12-20 10:50 +0100 |
| Subject | [PATCH v3 6/9] NTB: Add Messaging NTB API |
| Message-ID | <sQpdw-3QC-21@gated-at.bofh.it> |
| In reply to | #1541578 |
Some IDT NTB-capable PCIe-switches have message registers to communicate with
peer devices. This patch adds new NTB API callback methods, which can be used
to utilize these registers functionality:
ntb_msg_count(); - get number of message registers
ntb_msg_inbits(); - get bitfield of inbound message registers status
ntb_msg_outbits(); - get bitfield of outbound message registers status
ntb_msg_read_sts(); - read the inbound and outbound message registers status
ntb_msg_clear_sts(); - clear status bits of message registers
ntb_msg_set_mask(); - mask interrupts raised by status bits of message
registers.
ntb_msg_clear_mask(); - clear interrupts mask bits of message registers
ntb_msg_read(midx, *pidx); - read message register with specified index,
additionally getting peer port index which data received from
ntb_msg_write(midx, pidx); - write data to the specified message register
sending it to the passed peer device connected over a pidx port
ntb_msg_event(); - notify driver context of a new message event
Of course there is hardware which doesn't support Message registers, so
this API is made optional.
Acked-by: Allen Hubbe <Allen.Hubbe@dell.com>
Signed-off-by: Serge Semin <fancer.lancer@gmail.com>
---
drivers/ntb/ntb.c | 13 ++++
include/linux/ntb.h | 205 ++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 218 insertions(+)
diff --git a/drivers/ntb/ntb.c b/drivers/ntb/ntb.c
index 2551bb2..03b80d8 100644
--- a/drivers/ntb/ntb.c
+++ b/drivers/ntb/ntb.c
@@ -193,6 +193,19 @@ void ntb_db_event(struct ntb_dev *ntb, int vector)
}
EXPORT_SYMBOL(ntb_db_event);
+void ntb_msg_event(struct ntb_dev *ntb)
+{
+ unsigned long irqflags;
+
+ spin_lock_irqsave(&ntb->ctx_lock, irqflags);
+ {
+ if (ntb->ctx_ops && ntb->ctx_ops->msg_event)
+ ntb->ctx_ops->msg_event(ntb->ctx);
+ }
+ spin_unlock_irqrestore(&ntb->ctx_lock, irqflags);
+}
+EXPORT_SYMBOL(ntb_msg_event);
+
int ntb_default_port_number(struct ntb_dev *ntb)
{
switch (ntb->topo) {
diff --git a/include/linux/ntb.h b/include/linux/ntb.h
index a54e2be..76c56d5 100644
--- a/include/linux/ntb.h
+++ b/include/linux/ntb.h
@@ -178,10 +178,12 @@ static inline int ntb_client_ops_is_valid(const struct ntb_client_ops *ops)
* struct ntb_ctx_ops - ntb driver context operations
* @link_event: See ntb_link_event().
* @db_event: See ntb_db_event().
+ * @msg_event: See ntb_msg_event().
*/
struct ntb_ctx_ops {
void (*link_event)(void *ctx);
void (*db_event)(void *ctx, int db_vector);
+ void (*msg_event)(void *ctx);
};
static inline int ntb_ctx_ops_is_valid(const struct ntb_ctx_ops *ops)
@@ -190,6 +192,7 @@ static inline int ntb_ctx_ops_is_valid(const struct ntb_ctx_ops *ops)
return
/* ops->link_event && */
/* ops->db_event && */
+ /* ops->msg_event && */
1;
}
@@ -234,6 +237,15 @@ static inline int ntb_ctx_ops_is_valid(const struct ntb_ctx_ops *ops)
* @peer_spad_addr: See ntb_peer_spad_addr().
* @peer_spad_read: See ntb_peer_spad_read().
* @peer_spad_write: See ntb_peer_spad_write().
+ * @msg_count: See ntb_msg_count().
+ * @msg_inbits: See ntb_msg_inbits().
+ * @msg_outbits: See ntb_msg_outbits().
+ * @msg_read_sts: See ntb_msg_read_sts().
+ * @msg_clear_sts: See ntb_msg_clear_sts().
+ * @msg_set_mask: See ntb_msg_set_mask().
+ * @msg_clear_mask: See ntb_msg_clear_mask().
+ * @msg_read: See ntb_msg_read().
+ * @msg_write: See ntb_msg_write().
*/
struct ntb_dev_ops {
int (*port_number)(struct ntb_dev *ntb);
@@ -296,6 +308,16 @@ struct ntb_dev_ops {
u32 (*peer_spad_read)(struct ntb_dev *ntb, int pidx, int sidx);
int (*peer_spad_write)(struct ntb_dev *ntb, int pidx, int sidx,
u32 val);
+
+ int (*msg_count)(struct ntb_dev *ntb);
+ u64 (*msg_inbits)(struct ntb_dev *ntb);
+ u64 (*msg_outbits)(struct ntb_dev *ntb);
+ u64 (*msg_read_sts)(struct ntb_dev *ntb);
+ int (*msg_clear_sts)(struct ntb_dev *ntb, u64 sts_bits);
+ int (*msg_set_mask)(struct ntb_dev *ntb, u64 mask_bits);
+ int (*msg_clear_mask)(struct ntb_dev *ntb, u64 mask_bits);
+ int (*msg_read)(struct ntb_dev *ntb, int midx, int *pidx, u32 *msg);
+ int (*msg_write)(struct ntb_dev *ntb, int midx, int pidx, u32 msg);
};
static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops *ops)
@@ -342,6 +364,15 @@ static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops *ops)
/* !ops->peer_spad_addr == !ops->spad_count && */
/* !ops->peer_spad_read == !ops->spad_count && */
!ops->peer_spad_write == !ops->spad_count &&
+
+ !ops->msg_inbits == !ops->msg_count &&
+ !ops->msg_outbits == !ops->msg_count &&
+ !ops->msg_read_sts == !ops->msg_count &&
+ !ops->msg_clear_sts == !ops->msg_count &&
+ /* !ops->msg_set_mask == !ops->msg_count && */
+ /* !ops->msg_clear_mask == !ops->msg_count && */
+ !ops->msg_read == !ops->msg_count &&
+ !ops->msg_write == !ops->msg_count &&
1;
}
@@ -485,6 +516,18 @@ void ntb_link_event(struct ntb_dev *ntb);
void ntb_db_event(struct ntb_dev *ntb, int vector);
/**
+ * ntb_msg_event() - notify driver context of a message event
+ * @ntb: NTB device context.
+ *
+ * Notify the driver context of a message event. If hardware supports
+ * message registers, this event indicates, that a new message arrived in
+ * some incoming message register or last sent message couldn't be delivered.
+ * The events can be masked/unmasked by the methods ntb_msg_set_mask() and
+ * ntb_msg_clear_mask().
+ */
+void ntb_msg_event(struct ntb_dev *ntb);
+
+/**
* ntb_default_port_number() - get the default local port number
* @ntb: NTB device context.
*
@@ -1282,4 +1325,166 @@ static inline int ntb_peer_spad_write(struct ntb_dev *ntb, int pidx, int sidx,
return ntb->ops->peer_spad_write(ntb, pidx, sidx, val);
}
+/**
+ * ntb_msg_count() - get the number of message registers
+ * @ntb: NTB device context.
+ *
+ * Hardware may support a different number of message registers.
+ *
+ * Return: the number of message registers.
+ */
+static inline int ntb_msg_count(struct ntb_dev *ntb)
+{
+ if (!ntb->ops->msg_count)
+ return 0;
+
+ return ntb->ops->msg_count(ntb);
+}
+
+/**
+ * ntb_msg_inbits() - get a bitfield of inbound message registers status
+ * @ntb: NTB device context.
+ *
+ * The method returns the bitfield of status and mask registers, which related
+ * to inbound message registers.
+ *
+ * Return: bitfield of inbound message registers.
+ */
+static inline u64 ntb_msg_inbits(struct ntb_dev *ntb)
+{
+ if (!ntb->ops->msg_inbits)
+ return 0;
+
+ return ntb->ops->msg_inbits(ntb);
+}
+
+/**
+ * ntb_msg_outbits() - get a bitfield of outbound message registers status
+ * @ntb: NTB device context.
+ *
+ * The method returns the bitfield of status and mask registers, which related
+ * to outbound message registers.
+ *
+ * Return: bitfield of outbound message registers.
+ */
+static inline u64 ntb_msg_outbits(struct ntb_dev *ntb)
+{
+ if (!ntb->ops->msg_outbits)
+ return 0;
+
+ return ntb->ops->msg_outbits(ntb);
+}
+
+/**
+ * ntb_msg_read_sts() - read the message registers status
+ * @ntb: NTB device context.
+ *
+ * Read the status of message register. Inbound and outbound message registers
+ * related bits can be filtered by masks retrieved from ntb_msg_inbits() and
+ * ntb_msg_outbits().
+ *
+ * Return: status bits of message registers
+ */
+static inline u64 ntb_msg_read_sts(struct ntb_dev *ntb)
+{
+ if (!ntb->ops->msg_read_sts)
+ return 0;
+
+ return ntb->ops->msg_read_sts(ntb);
+}
+
+/**
+ * ntb_msg_clear_sts() - clear status bits of message registers
+ * @ntb: NTB device context.
+ * @sts_bits: Status bits to clear.
+ *
+ * Clear bits in the status register.
+ *
+ * Return: Zero on success, otherwise a negative error number.
+ */
+static inline int ntb_msg_clear_sts(struct ntb_dev *ntb, u64 sts_bits)
+{
+ if (!ntb->ops->msg_clear_sts)
+ return -EINVAL;
+
+ return ntb->ops->msg_clear_sts(ntb, sts_bits);
+}
+
+/**
+ * ntb_msg_set_mask() - set mask of message register status bits
+ * @ntb: NTB device context.
+ * @mask_bits: Mask bits.
+ *
+ * Mask the message registers status bits from raising the message event.
+ *
+ * Return: Zero on success, otherwise a negative error number.
+ */
+static inline int ntb_msg_set_mask(struct ntb_dev *ntb, u64 mask_bits)
+{
+ if (!ntb->ops->msg_set_mask)
+ return -EINVAL;
+
+ return ntb->ops->msg_set_mask(ntb, mask_bits);
+}
+
+/**
+ * ntb_msg_clear_mask() - clear message registers mask
+ * @ntb: NTB device context.
+ * @mask_bits: Mask bits to clear.
+ *
+ * Clear bits in the message events mask register.
+ *
+ * Return: Zero on success, otherwise a negative error number.
+ */
+static inline int ntb_msg_clear_mask(struct ntb_dev *ntb, u64 mask_bits)
+{
+ if (!ntb->ops->msg_clear_mask)
+ return -EINVAL;
+
+ return ntb->ops->msg_clear_mask(ntb, mask_bits);
+}
+
+/**
+ * ntb_msg_read() - read message register with specified index
+ * @ntb: NTB device context.
+ * @midx: Message register index
+ * @pidx: OUT - Port index of peer device a message retrieved from
+ * @msg: OUT - Data
+ *
+ * Read data from the specified message register. Source port index of a
+ * message is retrieved as well.
+ *
+ * Return: Zero on success, otherwise a negative error number.
+ */
+static inline int ntb_msg_read(struct ntb_dev *ntb, int midx, int *pidx,
+ u32 *msg)
+{
+ if (!ntb->ops->msg_read)
+ return -EINVAL;
+
+ return ntb->ops->msg_read(ntb, midx, pidx, msg);
+}
+
+/**
+ * ntb_msg_write() - write data to the specified message register
+ * @ntb: NTB device context.
+ * @midx: Message register index
+ * @pidx: Port index of peer device a message being sent to
+ * @msg: Data to send
+ *
+ * Send data to a specified peer device using the defined message register.
+ * Message event can be raised if the midx registers isn't empty while
+ * calling this method and the corresponding interrupt isn't masked.
+ *
+ * Return: Zero on success, otherwise a negative error number.
+ */
+static inline int ntb_msg_write(struct ntb_dev *ntb, int midx, int pidx,
+ u32 msg)
+{
+ if (!ntb->ops->msg_write)
+ return -EINVAL;
+
+ return ntb->ops->msg_write(ntb, midx, pidx, msg);
+}
+
#endif
--
2.6.6
[toc] | [prev] | [next] | [standalone]
| From | Serge Semin <fancer.lancer@gmail.com> |
|---|---|
| Date | 2016-12-14 01:00 +0100 |
| Subject | [PATCH v3 7/9] NTB: Add new Memory Windows API documentation |
| Message-ID | <sO59g-7zL-31@gated-at.bofh.it> |
| In reply to | #1541574 |
Since the new API slightly changes the way a typical NTB client driver works, the documentation file needs to be appropriately updated. Acked-by: Allen Hubbe <Allen.Hubbe@dell.com> Signed-off-by: Serge Semin <fancer.lancer@gmail.com> --- Documentation/ntb.txt | 99 ++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 91 insertions(+), 8 deletions(-) diff --git a/Documentation/ntb.txt b/Documentation/ntb.txt index 1d9bbab..a5af4f0 100644 --- a/Documentation/ntb.txt +++ b/Documentation/ntb.txt @@ -1,14 +1,16 @@ # NTB Drivers NTB (Non-Transparent Bridge) is a type of PCI-Express bridge chip that connects -the separate memory systems of two computers to the same PCI-Express fabric. -Existing NTB hardware supports a common feature set, including scratchpad -registers, doorbell registers, and memory translation windows. Scratchpad -registers are read-and-writable registers that are accessible from either side -of the device, so that peers can exchange a small amount of information at a -fixed address. Doorbell registers provide a way for peers to send interrupt -events. Memory windows allow translated read and write access to the peer -memory. +the separate memory systems of two or more computers to the same PCI-Express +fabric. Existing NTB hardware supports a common feature set: doorbell +registers and memory translation windows, as well as non common features like +scratchpad and message registers. Scratchpad registers are read-and-writable +registers that are accessible from either side of the device, so that peers can +exchange a small amount of information at a fixed address. Message registers can +be utilized for the same purpose. Additionally they are provided with with +special status bits to make sure the information isn't rewritten by another +peer. Doorbell registers provide a way for peers to send interrupt events. +Memory windows allow translated read and write access to the peer memory. ## NTB Core Driver (ntb) @@ -26,6 +28,87 @@ as ntb hardware, or hardware drivers, are inserted and removed. The registration uses the Linux Device framework, so it should feel familiar to anyone who has written a pci driver. +### NTB Typical client driver implementation + +Primary purpose of NTB is to share some peace of memory between at least two +systems. So the NTB device features like Scratchpad/Message registers are +mainly used to perform the proper memory window initialization. Typically +there are two types of memory window interfaces supported by the NTB API: +inbound translation configured on the local ntb port and outbound translation +configured by the peer, on the peer ntb port. The first type is +depicted on the next figure + +Inbound translation: + Memory: Local NTB Port: Peer NTB Port: Peer MMIO: + ____________ + | dma-mapped |-ntb_mw_set_trans(addr) | + | memory | _v____________ | ______________ + | (addr) |<======| MW xlat addr |<====| MW base addr |<== memory-mapped IO + |------------| |--------------| | |--------------| + +So typical scenario of the first type memory window initialization looks: +1) allocate a memory region, 2) put translated address to NTB config, +3) somehow notify a peer device of performed initialization, 4) peer device +maps corresponding outbound memory window so to have access to the shared +memory region. + +The second type of interface, that implies the shared windows being +initialized by a peer device, is depicted on the figure: + +Outbound translation: + Memory: Local NTB Port: Peer NTB Port: Peer MMIO: + ____________ ______________ + | dma-mapped | | | MW base addr |<== memory-mapped IO + | memory | | |--------------| + | (addr) |<===================| MW xlat addr |<-ntb_peer_mw_set_trans(addr) + |------------| | |--------------| + +Typical scenario of the second type interface initialization would be: +1) allocate a memory region, 2) somehow deliver a translated address to a peer +device, 3) peer puts the translated address to NTB config, 4) peer device maps +outbound memory window so to have access to the shared memory region. + +As one can see the described scenarios can be combined in one portable +algorithm. + Local device: + 1) Allocate memory for a shared window + 2) Initialize memory window by translated address of the allocated region + (it may fail if local memory window initialization is unsupported) + 3) Send the translated address and memory window index to a peer device + Peer device: + 1) Initialize memory window with retrieved address of the allocated + by another device memory region (it may fail if peer memory window + initialization is unsupported) + 2) Map outbound memory window + +In accordance with this scenario, the NTB Memory Window API can be used as +follows: + Local device: + 1) ntb_mw_count(pidx) - retrieve number of memory ranges, which can + be allocated for memory windows between local device and peer device + of port with specified index. + 2) ntb_get_align(pidx, midx) - retrieve parameters restricting the + shared memory region alignment and size. Then memory can be properly + allocated. + 3) Allocate physically contiguous memory region in compliance with + restrictions retrieved in 2). + 4) ntb_mw_set_trans(pidx, midx) - try to set translation address of + the memory window with specified index for the defined peer device + (it may fail if local translated address setting is not supported) + 5) Send translated base address (usually together with memory window + number) to the peer device using, for instance, scratchpad or message + registers. + Peer device: + 1) ntb_peer_mw_set_trans(pidx, midx) - try to set received from other + device (related to pidx) translated address for specified memory + window. It may fail if retrieved address, for instance, exceeds + maximum possible address or isn't properly aligned. + 2) ntb_peer_mw_get_addr(widx) - retrieve MMIO address to map the memory + window so to have an access to the shared memory. + +Also it is worth to note, that method ntb_mw_count(pidx) should return the +same value as ntb_peer_mw_count() on the peer with port index - pidx. + ### NTB Transport Client (ntb\_transport) and NTB Netdev (ntb\_netdev) The primary client for NTB is the Transport client, used in tandem with NTB -- 2.6.6
[toc] | [prev] | [standalone]
Page 2 of 2 — ← Prev page 1 [2]
Back to top | Article view | linux.kernel
csiph-web