Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1613003 > unrolled thread
| Started by | Wu Hao <hao.wu@intel.com> |
|---|---|
| First post | 2017-03-30 14:20 +0200 |
| Last post | 2017-04-06 22:30 +0200 |
| Articles | 20 on this page of 61 — 8 participants |
Back to article view | Back to linux.kernel
[PATCH 00/16] Intel FPGA Device Drivers Wu Hao <hao.wu@intel.com> - 2017-03-30 14:20 +0200
[PATCH 02/16] fpga: add FPGA device framework Wu Hao <hao.wu@intel.com> - 2017-03-30 14:20 +0200
Re: [PATCH 02/16] fpga: add FPGA device framework Greg KH <greg@kroah.com> - 2017-03-31 08:10 +0200
Re: [PATCH 02/16] fpga: add FPGA device framework Wu Hao <hao.wu@intel.com> - 2017-03-31 10:00 +0200
Re: [PATCH 02/16] fpga: add FPGA device framework Greg KH <greg@kroah.com> - 2017-03-31 11:10 +0200
Re: [PATCH 02/16] fpga: add FPGA device framework Wu Hao <hao.wu@intel.com> - 2017-03-31 14:30 +0200
Re: [PATCH 02/16] fpga: add FPGA device framework matthew.gerlach@linux.intel.com - 2017-03-31 21:10 +0200
Re: [PATCH 02/16] fpga: add FPGA device framework Wu Hao <hao.wu@intel.com> - 2017-04-01 14:30 +0200
Re: [PATCH 02/16] fpga: add FPGA device framework Greg KH <greg@kroah.com> - 2017-03-31 08:20 +0200
Re: [PATCH 02/16] fpga: add FPGA device framework Wu Hao <hao.wu@intel.com> - 2017-03-31 15:40 +0200
Re: [PATCH 02/16] fpga: add FPGA device framework Greg KH <greg@kroah.com> - 2017-03-31 16:20 +0200
Re: [PATCH 02/16] fpga: add FPGA device framework Wu Hao <hao.wu@intel.com> - 2017-04-01 13:50 +0200
[PATCH 03/16] fpga: intel: add FPGA PCIe device driver Wu Hao <hao.wu@intel.com> - 2017-03-30 14:20 +0200
Re: [PATCH 03/16] fpga: intel: add FPGA PCIe device driver Moritz Fischer <mdf@kernel.org> - 2017-04-04 04:20 +0200
RE: [PATCH 03/16] fpga: intel: add FPGA PCIe device driver "Wu, Hao" <hao.wu@intel.com> - 2017-04-05 15:20 +0200
[PATCH 15/16] fpga: intel: afu: add user afu sub feature support Wu Hao <hao.wu@intel.com> - 2017-03-30 14:20 +0200
[PATCH 12/16] fpga: intel: add FPGA Accelerated Function Unit driver basic framework Wu Hao <hao.wu@intel.com> - 2017-03-30 14:20 +0200
[PATCH 09/16] fpga: intel: fme: add header sub feature support Wu Hao <hao.wu@intel.com> - 2017-03-30 14:20 +0200
[PATCH 16/16] fpga: intel: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support Wu Hao <hao.wu@intel.com> - 2017-03-30 14:20 +0200
[PATCH 07/16] fpga: intel: add feature device infrastructure Wu Hao <hao.wu@intel.com> - 2017-03-30 14:20 +0200
[PATCH 05/16] fpga: intel: pcie: add chardev support for feature devices Wu Hao <hao.wu@intel.com> - 2017-03-30 14:20 +0200
[PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features. Wu Hao <hao.wu@intel.com> - 2017-03-30 14:20 +0200
Re: [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features. Alan Tull <atull@kernel.org> - 2017-04-03 23:50 +0200
Re: [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features. Wu Hao <hao.wu@intel.com> - 2017-04-05 14:10 +0200
Re: [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features. Alan Tull <atull@kernel.org> - 2017-04-05 00:20 +0200
Re: [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features. Wu Hao <hao.wu@intel.com> - 2017-04-05 16:20 +0200
[PATCH 10/16] fpga: intel: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support Wu Hao <hao.wu@intel.com> - 2017-03-30 14:20 +0200
[PATCH 14/16] fpga: intel: afu add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support Wu Hao <hao.wu@intel.com> - 2017-03-30 14:20 +0200
[PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview Wu Hao <hao.wu@intel.com> - 2017-03-30 14:20 +0200
Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview matthew.gerlach@linux.intel.com - 2017-03-31 20:30 +0200
Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview Alan Tull <atull@kernel.org> - 2017-03-31 20:40 +0200
Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview Wu Hao <hao.wu@intel.com> - 2017-04-01 13:30 +0200
Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview Moritz Fischer <mdf@kernel.org> - 2017-04-02 16:50 +0200
Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview Alan Tull <atull@kernel.org> - 2017-04-03 22:50 +0200
Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview Wu Hao <hao.wu@intel.com> - 2017-04-04 07:30 +0200
Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview Wu Hao <hao.wu@intel.com> - 2017-04-04 07:20 +0200
[PATCH 06/16] fpga: intel: pcie: adds fpga_for_each_port callback for fme device Wu Hao <hao.wu@intel.com> - 2017-03-30 14:20 +0200
[PATCH 13/16] fpga: intel: afu: add header sub feature support Wu Hao <hao.wu@intel.com> - 2017-03-30 14:20 +0200
[PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Wu Hao <hao.wu@intel.com> - 2017-03-30 14:20 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Alan Tull <atull@kernel.org> - 2017-03-30 22:40 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Xiao Guangrong <xiaoguangrong.eric@gmail.com> - 2017-03-31 06:20 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Wu Hao <hao.wu@intel.com> - 2017-03-31 11:00 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Alan Tull <atull@kernel.org> - 2017-04-03 22:30 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Wu Hao <hao.wu@intel.com> - 2017-04-04 07:40 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Alan Tull <atull@kernel.org> - 2017-03-31 21:20 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Wu Hao <hao.wu@intel.com> - 2017-04-01 13:20 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Alan Tull <atull@kernel.org> - 2017-04-03 18:40 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Wu Hao <hao.wu@intel.com> - 2017-04-04 08:20 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Alan Tull <atull@kernel.org> - 2017-04-05 00:40 +0200
RE: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support "Wu, Hao" <hao.wu@intel.com> - 2017-04-05 13:50 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Alan Tull <atull@kernel.org> - 2017-04-05 17:30 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Alan Tull <atull@kernel.org> - 2017-04-05 17:50 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Wu Hao <hao.wu@intel.com> - 2017-04-06 13:10 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Alan Tull <atull@kernel.org> - 2017-04-06 21:30 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Wu Hao <hao.wu@intel.com> - 2017-04-07 08:10 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Alan Tull <atull@kernel.org> - 2017-04-03 23:30 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support matthew.gerlach@linux.intel.com - 2017-04-04 00:50 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Wu Hao <hao.wu@intel.com> - 2017-04-04 09:00 +0200
Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Wu Hao <hao.wu@intel.com> - 2017-04-04 08:40 +0200
Re: [PATCH 00/16] Intel FPGA Device Drivers Moritz Fischer <mdf@kernel.org> - 2017-03-30 19:20 +0200
Re: [PATCH 00/16] Intel FPGA Device Drivers Jerome Glisse <jglisse@redhat.com> - 2017-04-06 22:30 +0200
Page 2 of 4 — ← Prev page 1 [2] 3 4 Next page →
| From | Wu Hao <hao.wu@intel.com> |
|---|---|
| Date | 2017-03-30 14:20 +0200 |
| Subject | [PATCH 05/16] fpga: intel: pcie: add chardev support for feature devices |
| Message-ID | <tqHdw-69w-31@gated-at.bofh.it> |
| In reply to | #1613003 |
From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
For feature devices drivers, both the FPGA Management Engine (FME) and
Accelerated Function Unit (AFU) driver need to expose user interfaces via
the device file, for example, mmap and ioctls.
This patch adds chardev support in the pcie driver for feature devices,
FME and AFU. It reserves the chardev regions for FME and AFU, and provide
interfaces for FME and AFU driver to register their device file operations.
Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
drivers/fpga/intel/feature-dev.c | 76 ++++++++++++++++++++++++++++++++++++++++
drivers/fpga/intel/feature-dev.h | 16 +++++++++
drivers/fpga/intel/pcie.c | 18 +++++++++-
3 files changed, 109 insertions(+), 1 deletion(-)
diff --git a/drivers/fpga/intel/feature-dev.c b/drivers/fpga/intel/feature-dev.c
index 6952566..ada6548 100644
--- a/drivers/fpga/intel/feature-dev.c
+++ b/drivers/fpga/intel/feature-dev.c
@@ -59,6 +59,82 @@ int port_feature_num(void)
return PORT_FEATURE_ID_MAX;
}
+struct fpga_chardev_info {
+ const char *name;
+ dev_t devt;
+};
+
+/* indexed by enum fpga_devt_type */
+struct fpga_chardev_info fpga_chrdevs[] = {
+ {.name = FPGA_FEATURE_DEV_FME}, /* FPGA_DEVT_FME */
+ {.name = FPGA_FEATURE_DEV_PORT}, /* FPGA_DEVT_AFU */
+};
+
+void fpga_chardev_uinit(void)
+{
+ int i;
+
+ for (i = 0; i < FPGA_DEVT_MAX; i++)
+ if (MAJOR(fpga_chrdevs[i].devt)) {
+ unregister_chrdev_region(fpga_chrdevs[i].devt,
+ MINORMASK);
+ fpga_chrdevs[i].devt = MKDEV(0, 0);
+ }
+}
+
+int fpga_chardev_init(void)
+{
+ int i, ret;
+
+ for (i = 0; i < FPGA_DEVT_MAX; i++) {
+ ret = alloc_chrdev_region(&fpga_chrdevs[i].devt, 0, MINORMASK,
+ fpga_chrdevs[i].name);
+ if (ret)
+ goto exit;
+ }
+
+ return 0;
+
+exit:
+ fpga_chardev_uinit();
+ return ret;
+}
+
+dev_t fpga_get_devt(enum fpga_devt_type type, int id)
+{
+ WARN_ON(type >= FPGA_DEVT_MAX);
+
+ return MKDEV(MAJOR(fpga_chrdevs[type].devt), id);
+}
+
+int fpga_register_dev_ops(struct platform_device *pdev,
+ const struct file_operations *fops,
+ struct module *owner)
+{
+ struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+ cdev_init(&pdata->cdev, fops);
+ pdata->cdev.owner = owner;
+
+ /*
+ * set parent to the feature device so that its refcount is
+ * decreased after the last refcount of cdev is gone, that
+ * makes sure the feature device is valid during device
+ * file's life-cycle.
+ */
+ pdata->cdev.kobj.parent = &pdev->dev.kobj;
+ return cdev_add(&pdata->cdev, pdev->dev.devt, 1);
+}
+EXPORT_SYMBOL_GPL(fpga_register_dev_ops);
+
+void fpga_unregister_dev_ops(struct platform_device *pdev)
+{
+ struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+ cdev_del(&pdata->cdev);
+}
+EXPORT_SYMBOL_GPL(fpga_unregister_dev_ops);
+
int fpga_port_id(struct platform_device *pdev)
{
struct feature_port_header *port_hdr;
diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
index a1e6e7d..d1723ff 100644
--- a/drivers/fpga/intel/feature-dev.h
+++ b/drivers/fpga/intel/feature-dev.h
@@ -19,6 +19,7 @@
#define __INTEL_FPGA_FEATURE_H
#include <linux/fs.h>
+#include <linux/cdev.h>
#include <linux/pci.h>
#include <linux/uuid.h>
#include <linux/delay.h>
@@ -216,6 +217,7 @@ struct feature_platform_data {
/* list the feature dev to cci_drvdata->port_dev_list. */
struct list_head node;
struct mutex lock;
+ struct cdev cdev;
struct platform_device *dev;
unsigned int disable_count; /* count for port disable */
@@ -256,6 +258,20 @@ int feature_platform_data_size(int num);
struct feature_platform_data *
feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
+enum fpga_devt_type {
+ FPGA_DEVT_FME,
+ FPGA_DEVT_PORT,
+ FPGA_DEVT_MAX,
+};
+
+void fpga_chardev_uinit(void);
+int fpga_chardev_init(void);
+dev_t fpga_get_devt(enum fpga_devt_type type, int id);
+int fpga_register_dev_ops(struct platform_device *pdev,
+ const struct file_operations *fops,
+ struct module *owner);
+void fpga_unregister_dev_ops(struct platform_device *pdev);
+
int fpga_port_id(struct platform_device *pdev);
static inline int fpga_port_check_id(struct platform_device *pdev,
diff --git a/drivers/fpga/intel/pcie.c b/drivers/fpga/intel/pcie.c
index 28df63e..e3440ca 100644
--- a/drivers/fpga/intel/pcie.c
+++ b/drivers/fpga/intel/pcie.c
@@ -276,8 +276,12 @@ build_info_create_dev(struct build_feature_devs_info *binfo,
struct platform_device *fdev;
struct resource *res;
struct feature_platform_data *pdata;
+ enum fpga_devt_type devt_type = FPGA_DEVT_FME;
int ret;
+ if (type == PORT_ID)
+ devt_type = FPGA_DEVT_PORT;
+
/* we will create a new device, commit current device first */
ret = build_info_commit_dev(binfo);
if (ret)
@@ -296,6 +300,7 @@ build_info_create_dev(struct build_feature_devs_info *binfo,
return fdev->id;
fdev->dev.parent = &binfo->parent_dev->dev;
+ fdev->dev.devt = fpga_get_devt(devt_type, fdev->id);
/*
* we need not care the memory which is associated with the
@@ -945,16 +950,27 @@ static int __init ccidrv_init(void)
fpga_ids_init();
+ ret = fpga_chardev_init();
+ if (ret)
+ goto exit_ids;
+
ret = pci_register_driver(&cci_pci_driver);
if (ret)
- fpga_ids_destroy();
+ goto exit_chardev;
+ return 0;
+
+exit_chardev:
+ fpga_chardev_uinit();
+exit_ids:
+ fpga_ids_destroy();
return ret;
}
static void __exit ccidrv_exit(void)
{
pci_unregister_driver(&cci_pci_driver);
+ fpga_chardev_uinit();
fpga_ids_destroy();
}
--
2.7.4
[toc] | [prev] | [next] | [standalone]
| From | Wu Hao <hao.wu@intel.com> |
|---|---|
| Date | 2017-03-30 14:20 +0200 |
| Subject | [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features. |
| Message-ID | <tqHdw-69w-17@gated-at.bofh.it> |
| In reply to | #1613003 |
From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Device Featuer List structure creates a link list of feature headers
within the MMIO space to provide an extensiable way of adding features.
The Intel FPGA PCIe driver walks through the feature headers to enumerate
feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
Function Unit (AFU), and their private sub features. For feature devices,
it creates the platform devices and linked the private sub features into
their platform data.
Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Kang Luwei <luwei.kang@intel.com>
Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
drivers/fpga/intel/Makefile | 2 +-
drivers/fpga/intel/feature-dev.c | 139 +++++++
drivers/fpga/intel/feature-dev.h | 342 ++++++++++++++++
drivers/fpga/intel/pcie.c | 841 ++++++++++++++++++++++++++++++++++++++-
4 files changed, 1321 insertions(+), 3 deletions(-)
create mode 100644 drivers/fpga/intel/feature-dev.c
create mode 100644 drivers/fpga/intel/feature-dev.h
diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
index 61fd8ea..c029940 100644
--- a/drivers/fpga/intel/Makefile
+++ b/drivers/fpga/intel/Makefile
@@ -1,3 +1,3 @@
obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
-intel-fpga-pci-objs := pcie.o
+intel-fpga-pci-objs := pcie.o feature-dev.o
diff --git a/drivers/fpga/intel/feature-dev.c b/drivers/fpga/intel/feature-dev.c
new file mode 100644
index 0000000..6952566
--- /dev/null
+++ b/drivers/fpga/intel/feature-dev.c
@@ -0,0 +1,139 @@
+/*
+ * Intel FPGA Feature Device Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ * Kang Luwei <luwei.kang@intel.com>
+ * Zhang Yi <yi.z.zhang@intel.com>
+ * Wu Hao <hao.wu@intel.com>
+ * Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under this directory for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+
+#include "feature-dev.h"
+
+void feature_platform_data_add(struct feature_platform_data *pdata,
+ int index, const char *name,
+ int resource_index, void __iomem *ioaddr)
+{
+ WARN_ON(index >= pdata->num);
+
+ pdata->features[index].name = name;
+ pdata->features[index].resource_index = resource_index;
+ pdata->features[index].ioaddr = ioaddr;
+}
+
+int feature_platform_data_size(int num)
+{
+ return sizeof(struct feature_platform_data) +
+ num * sizeof(struct feature);
+}
+
+struct feature_platform_data *
+feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
+{
+ struct feature_platform_data *pdata;
+
+ pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
+ if (pdata) {
+ pdata->dev = dev;
+ pdata->num = num;
+ mutex_init(&pdata->lock);
+ }
+
+ return pdata;
+}
+
+int fme_feature_num(void)
+{
+ return FME_FEATURE_ID_MAX;
+}
+
+int port_feature_num(void)
+{
+ return PORT_FEATURE_ID_MAX;
+}
+
+int fpga_port_id(struct platform_device *pdev)
+{
+ struct feature_port_header *port_hdr;
+ struct feature_port_capability capability;
+
+ port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
+ PORT_FEATURE_ID_HEADER);
+ WARN_ON(!port_hdr);
+
+ capability.csr = readq(&port_hdr->capability);
+ return capability.port_number;
+}
+EXPORT_SYMBOL_GPL(fpga_port_id);
+
+/*
+ * Enable Port by clear the port soft reset bit, which is set by default.
+ * The User AFU is unable to respond to any MMIO access while in reset.
+ * __fpga_port_enable function should only be used after __fpga_port_disable
+ * function.
+ */
+void __fpga_port_enable(struct platform_device *pdev)
+{
+ struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+ struct feature_port_header *port_hdr;
+ struct feature_port_control control;
+
+ WARN_ON(!pdata->disable_count);
+
+ if (--pdata->disable_count != 0)
+ return;
+
+ port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
+ PORT_FEATURE_ID_HEADER);
+ WARN_ON(!port_hdr);
+
+ control.csr = readq(&port_hdr->control);
+ control.port_sftrst = 0x0;
+ writeq(control.csr, &port_hdr->control);
+}
+EXPORT_SYMBOL_GPL(__fpga_port_enable);
+
+#define RST_POLL_INVL 10 /* us */
+#define RST_POLL_TIMEOUT 1000 /* us */
+
+int __fpga_port_disable(struct platform_device *pdev)
+{
+ struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+ struct feature_port_header *port_hdr;
+ struct feature_port_control control;
+
+ if (pdata->disable_count++ != 0)
+ return 0;
+
+ port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
+ PORT_FEATURE_ID_HEADER);
+ WARN_ON(!port_hdr);
+
+ /* Set port soft reset */
+ control.csr = readq(&port_hdr->control);
+ control.port_sftrst = 0x1;
+ writeq(control.csr, &port_hdr->control);
+
+ /*
+ * HW sets ack bit to 1 when all outstanding requests have been drained
+ * on this port and minimum soft reset pulse width has elapsed.
+ * Driver polls port_soft_reset_ack to determine if reset done by HW.
+ */
+ control.port_sftrst_ack = 1;
+
+ if (fpga_wait_register_field(port_sftrst_ack, control,
+ &port_hdr->control, RST_POLL_TIMEOUT, RST_POLL_INVL)) {
+ dev_err(&pdev->dev, "timeout, fail to reset device\n");
+ return -ETIMEDOUT;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(__fpga_port_disable);
diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
new file mode 100644
index 0000000..a1e6e7d
--- /dev/null
+++ b/drivers/fpga/intel/feature-dev.h
@@ -0,0 +1,342 @@
+/*
+ * Intel FPGA Feature Device Driver Header File
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ * Kang Luwei <luwei.kang@intel.com>
+ * Zhang Yi <yi.z.zhang@intel.com>
+ * Wu Hao <hao.wu@intel.com>
+ * Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under this directory for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+
+#ifndef __INTEL_FPGA_FEATURE_H
+#define __INTEL_FPGA_FEATURE_H
+
+#include <linux/fs.h>
+#include <linux/pci.h>
+#include <linux/uuid.h>
+#include <linux/delay.h>
+#include <linux/platform_device.h>
+
+/* maximum supported number of ports */
+#define MAX_FPGA_PORT_NUM 4
+/* plus one for fme device */
+#define MAX_FEATURE_DEV_NUM (MAX_FPGA_PORT_NUM + 1)
+
+#define FME_FEATURE_HEADER "fme_hdr"
+#define FME_FEATURE_THERMAL_MGMT "fme_thermal"
+#define FME_FEATURE_POWER_MGMT "fme_power"
+#define FME_FEATURE_GLOBAL_PERF "fme_gperf"
+#define FME_FEATURE_GLOBAL_ERR "fme_error"
+#define FME_FEATURE_PR_MGMT "fme_pr"
+
+#define PORT_FEATURE_HEADER "port_hdr"
+#define PORT_FEATURE_UAFU "port_uafu"
+#define PORT_FEATURE_ERR "port_err"
+#define PORT_FEATURE_UMSG "port_umsg"
+#define PORT_FEATURE_PR "port_pr"
+#define PORT_FEATURE_STP "port_stp"
+
+/* All headers and structures must be byte-packed to match the spec. */
+#pragma pack(1)
+
+/* common header for all features */
+struct feature_header {
+ union {
+ u64 csr;
+ struct {
+ u16 id:12;
+ u8 revision:4;
+ u32 next_header_offset:24; /* offset to next header */
+ u32 rsvdz:20;
+ u8 type:4; /* feature type */
+#define FEATURE_TYPE_AFU 0x1
+#define FEATURE_TYPE_PRIVATE 0x3
+ };
+ };
+};
+
+/* common header for non-private features */
+struct feature_afu_header {
+ uuid_le guid;
+ union {
+ u64 csr;
+ struct {
+ u64 next_afu:24; /* pointer to next afu header */
+ u64 rsvdz:40;
+ };
+ };
+};
+
+/* FME Header Register Set */
+/* FME Capability Register */
+struct feature_fme_capability {
+ union {
+ u64 csr;
+ struct {
+ u8 fabric_verid; /* Fabric version ID */
+ u8 socket_id:1; /* Socket id */
+ u8 rsvdz1:3;
+ u8 pcie0_link_avl:1; /* PCIe0 link availability */
+ u8 pcie1_link_avl:1; /* PCIe1 link availability */
+ u8 coherent_link_avl:1;/* Coherent link availability */
+ u8 rsvdz2:1;
+ u8 iommu_support:1; /* IOMMU or VT-d supported */
+ u8 num_ports:3; /* Num of ports implemented */
+ u8 rsvdz3:4;
+ u8 addr_width_bits:6; /* Address width supported */
+ u8 rsvdz4:2;
+ u16 cache_size:12; /* Cache size in kb */
+ u8 cache_assoc:4; /* Cache Associativity */
+ u16 rsvdz5:15;
+ u8 lock_bit:1; /* Latched lock bit by BIOS */
+ };
+ };
+};
+
+/* FME Port Offset Register */
+struct feature_fme_port {
+ union {
+ u64 csr;
+ struct {
+ u32 port_offset:24; /* Offset to port header */
+ u8 rsvdz1;
+ u8 port_bar:3; /* Bar id */
+ u32 rsvdz2:20;
+ u8 afu_access_ctrl:1; /* AFU access type: PF/VF */
+ u8 rsvdz3:4;
+ u8 port_implemented:1; /* Port implemented or not */
+ u8 rsvdz4:3;
+ };
+ };
+};
+
+struct feature_fme_header {
+ struct feature_header header;
+ struct feature_afu_header afu_header;
+ u64 rsvd[2];
+ struct feature_fme_capability capability;
+ struct feature_fme_port port[MAX_FPGA_PORT_NUM];
+};
+
+/* FME Thermal Sub Feature Register Set */
+struct feature_fme_thermal {
+ struct feature_header header;
+};
+
+/* FME Power Sub Feature Register Set */
+struct feature_fme_power {
+ struct feature_header header;
+};
+
+/* FME Global Performance Sub Feature Register Set */
+struct feature_fme_gperf {
+ struct feature_header header;
+};
+
+/* FME Error Sub Feature Register Set */
+struct feature_fme_err {
+ struct feature_header header;
+};
+
+/* FME Partial Reconfiguration Sub Feature Register Set */
+struct feature_fme_pr {
+ struct feature_header header;
+};
+
+/* PORT Header Register Set */
+/* Port Capability Register */
+struct feature_port_capability {
+ union {
+ u64 csr;
+ struct {
+ u8 port_number:2; /* Port Number 0-3 */
+ u8 rsvdz1:6;
+ u16 mmio_size; /* User MMIO size in KB */
+ u8 rsvdz2;
+ u8 sp_intr_num:4; /* Supported interrupts num */
+ u32 rsvdz3:28;
+ };
+ };
+};
+
+/* Port Control Register */
+struct feature_port_control {
+ union {
+ u64 csr;
+ struct {
+ u8 port_sftrst:1; /* Port Soft Reset */
+ u8 rsvdz1:1;
+ u8 latency_tolerance:1;/* '1' >= 40us, '0' < 40us */
+ u8 rsvdz2:1;
+ u8 port_sftrst_ack:1; /* HW ACK for Soft Reset */
+ u64 rsvdz3:59;
+ };
+ };
+};
+
+struct feature_port_header {
+ struct feature_header header;
+ struct feature_afu_header afu_header;
+ u64 rsvd[2];
+ struct feature_port_capability capability;
+ struct feature_port_control control;
+};
+
+/* PORT Error Sub Feature Register Set */
+struct feature_port_error {
+ struct feature_header header;
+};
+
+/* PORT Unordered Message Sub Feature Register Set */
+struct feature_port_umsg {
+ struct feature_header header;
+};
+
+/* PORT SignalTap Sub Feature Register Set */
+struct feature_port_stp {
+ struct feature_header header;
+};
+
+#pragma pack()
+
+struct feature {
+ const char *name;
+ int resource_index;
+ void __iomem *ioaddr;
+};
+
+struct feature_platform_data {
+ /* list the feature dev to cci_drvdata->port_dev_list. */
+ struct list_head node;
+ struct mutex lock;
+ struct platform_device *dev;
+ unsigned int disable_count; /* count for port disable */
+
+ int num; /* number of features */
+ struct feature features[0];
+};
+
+enum fme_feature_id {
+ FME_FEATURE_ID_HEADER = 0x0,
+ FME_FEATURE_ID_THERMAL_MGMT = 0x1,
+ FME_FEATURE_ID_POWER_MGMT = 0x2,
+ FME_FEATURE_ID_GLOBAL_PERF = 0x3,
+ FME_FEATURE_ID_GLOBAL_ERR = 0x4,
+ FME_FEATURE_ID_PR_MGMT = 0x5,
+ FME_FEATURE_ID_MAX = 0x6,
+};
+
+enum port_feature_id {
+ PORT_FEATURE_ID_HEADER = 0x0,
+ PORT_FEATURE_ID_ERROR = 0x1,
+ PORT_FEATURE_ID_UMSG = 0x2,
+ PORT_FEATURE_ID_PR = 0x3,
+ PORT_FEATURE_ID_STP = 0x4,
+ PORT_FEATURE_ID_UAFU = 0x5,
+ PORT_FEATURE_ID_MAX = 0x6,
+};
+
+int fme_feature_num(void);
+int port_feature_num(void);
+
+#define FPGA_FEATURE_DEV_FME "intel-fpga-fme"
+#define FPGA_FEATURE_DEV_PORT "intel-fpga-port"
+
+void feature_platform_data_add(struct feature_platform_data *pdata,
+ int index, const char *name,
+ int resource_index, void __iomem *ioaddr);
+int feature_platform_data_size(int num);
+struct feature_platform_data *
+feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
+
+int fpga_port_id(struct platform_device *pdev);
+
+static inline int fpga_port_check_id(struct platform_device *pdev,
+ void *pport_id)
+{
+ return fpga_port_id(pdev) == *(int *)pport_id;
+}
+
+void __fpga_port_enable(struct platform_device *pdev);
+int __fpga_port_disable(struct platform_device *pdev);
+
+static inline void fpga_port_enable(struct platform_device *pdev)
+{
+ struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+ mutex_lock(&pdata->lock);
+ __fpga_port_enable(pdev);
+ mutex_unlock(&pdata->lock);
+}
+
+static inline int fpga_port_disable(struct platform_device *pdev)
+{
+ struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+ int ret;
+
+ mutex_lock(&pdata->lock);
+ ret = __fpga_port_disable(pdev);
+ mutex_unlock(&pdata->lock);
+
+ return ret;
+}
+
+static inline int __fpga_port_reset(struct platform_device *pdev)
+{
+ int ret;
+
+ ret = __fpga_port_disable(pdev);
+ if (ret)
+ return ret;
+
+ __fpga_port_enable(pdev);
+ return 0;
+}
+
+static inline int fpga_port_reset(struct platform_device *pdev)
+{
+ struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+ int ret;
+
+ mutex_lock(&pdata->lock);
+ ret = __fpga_port_reset(pdev);
+ mutex_unlock(&pdata->lock);
+ return ret;
+}
+
+static inline void __iomem *
+get_feature_ioaddr_by_index(struct device *dev, int index)
+{
+ struct feature_platform_data *pdata = dev_get_platdata(dev);
+
+ return pdata->features[index].ioaddr;
+}
+
+/*
+ * Wait register's _field to be changed to the given value (_expect's _field)
+ * by polling with given interval and timeout.
+ */
+#define fpga_wait_register_field(_field, _expect, _reg_addr, _timeout, _invl)\
+({ \
+ int wait = 0; \
+ int ret = -ETIMEDOUT; \
+ typeof(_expect) value; \
+ for (; wait <= _timeout; wait += _invl) { \
+ value.csr = readq(_reg_addr); \
+ if (_expect._field == value._field) { \
+ ret = 0; \
+ break; \
+ } \
+ udelay(_invl); \
+ } \
+ ret; \
+})
+
+#endif
diff --git a/drivers/fpga/intel/pcie.c b/drivers/fpga/intel/pcie.c
index 132d9da..28df63e 100644
--- a/drivers/fpga/intel/pcie.c
+++ b/drivers/fpga/intel/pcie.c
@@ -25,10 +25,827 @@
#include <linux/stddef.h>
#include <linux/errno.h>
#include <linux/aer.h>
+#include <linux/fpga/fpga-dev.h>
+
+#include "feature-dev.h"
#define DRV_VERSION "EXPERIMENTAL VERSION"
#define DRV_NAME "intel-fpga-pci"
+#define INTEL_FPGA_DEV "intel-fpga-dev"
+
+static DEFINE_MUTEX(fpga_id_mutex);
+
+enum fpga_id_type {
+ FME_ID, /* fme id allocation and mapping */
+ PORT_ID, /* port id allocation and mapping */
+ FPGA_ID_MAX,
+};
+
+/* it is protected by fpga_id_mutex */
+static struct idr fpga_ids[FPGA_ID_MAX];
+
+struct cci_drvdata {
+ struct device *fme_dev;
+
+ struct mutex lock;
+ struct list_head port_dev_list;
+
+ struct list_head regions; /* global list of pci bar mapping region */
+};
+
+/* pci bar mapping info */
+struct cci_pci_region {
+ int bar;
+ void __iomem *ioaddr; /* pointer to mapped bar region */
+ struct list_head node;
+};
+
+static void fpga_ids_init(void)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
+ idr_init(fpga_ids + i);
+}
+
+static void fpga_ids_destroy(void)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
+ idr_destroy(fpga_ids + i);
+}
+
+static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
+{
+ int id;
+
+ WARN_ON(type >= FPGA_ID_MAX);
+ mutex_lock(&fpga_id_mutex);
+ id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
+ mutex_unlock(&fpga_id_mutex);
+ return id;
+}
+
+static void free_fpga_id(enum fpga_id_type type, int id)
+{
+ WARN_ON(type >= FPGA_ID_MAX);
+ mutex_lock(&fpga_id_mutex);
+ idr_remove(fpga_ids + type, id);
+ mutex_unlock(&fpga_id_mutex);
+}
+
+static void cci_pci_add_port_dev(struct pci_dev *pdev,
+ struct platform_device *port_dev)
+{
+ struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
+ struct feature_platform_data *pdata = dev_get_platdata(&port_dev->dev);
+
+ mutex_lock(&drvdata->lock);
+ list_add(&pdata->node, &drvdata->port_dev_list);
+ get_device(&pdata->dev->dev);
+ mutex_unlock(&drvdata->lock);
+}
+
+static void cci_pci_remove_port_devs(struct pci_dev *pdev)
+{
+ struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
+ struct feature_platform_data *pdata, *ptmp;
+
+ mutex_lock(&drvdata->lock);
+ list_for_each_entry_safe(pdata, ptmp, &drvdata->port_dev_list, node) {
+ struct platform_device *port_dev = pdata->dev;
+
+ /* the port should be unregistered first. */
+ WARN_ON(device_is_registered(&port_dev->dev));
+ list_del(&pdata->node);
+ free_fpga_id(PORT_ID, port_dev->id);
+ put_device(&port_dev->dev);
+ }
+ mutex_unlock(&drvdata->lock);
+}
+
+/* info collection during feature dev build. */
+struct build_feature_devs_info {
+ struct pci_dev *pdev;
+
+ /*
+ * PCI BAR mapping info. Parsing feature list starts from
+ * BAR 0 and switch to different BARs to parse Port
+ */
+ void __iomem *ioaddr;
+ void __iomem *ioend;
+ int current_bar;
+
+ /* points to FME header where the port offset is figured out. */
+ void __iomem *pfme_hdr;
+
+ /* the container device for all feature devices */
+ struct fpga_dev *parent_dev;
+
+ /* current feature device */
+ struct platform_device *feature_dev;
+};
+
+static void cci_pci_release_regions(struct pci_dev *pdev)
+{
+ struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
+ struct cci_pci_region *tmp, *region;
+
+ list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
+ list_del(®ion->node);
+ if (region->ioaddr)
+ pci_iounmap(pdev, region->ioaddr);
+ devm_kfree(&pdev->dev, region);
+ }
+}
+
+static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pdev, int bar)
+{
+ struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
+ struct cci_pci_region *region;
+
+ list_for_each_entry(region, &drvdata->regions, node)
+ if (region->bar == bar) {
+ dev_dbg(&pdev->dev, "BAR %d region exists\n", bar);
+ return region->ioaddr;
+ }
+
+ region = devm_kzalloc(&pdev->dev, sizeof(*region), GFP_KERNEL);
+ if (!region)
+ return NULL;
+
+ region->bar = bar;
+ region->ioaddr = pci_ioremap_bar(pdev, bar);
+ if (!region->ioaddr) {
+ dev_err(&pdev->dev, "can't ioremap memory from BAR %d.\n", bar);
+ devm_kfree(&pdev->dev, region);
+ return NULL;
+ }
+
+ list_add(®ion->node, &drvdata->regions);
+ return region->ioaddr;
+}
+
+static int parse_start_from(struct build_feature_devs_info *binfo, int bar)
+{
+ binfo->ioaddr = cci_pci_ioremap_bar(binfo->pdev, bar);
+ if (!binfo->ioaddr)
+ return -ENOMEM;
+
+ binfo->current_bar = bar;
+ binfo->ioend = binfo->ioaddr + pci_resource_len(binfo->pdev, bar);
+ return 0;
+}
+
+static int parse_start(struct build_feature_devs_info *binfo)
+{
+ /* fpga feature list starts from BAR 0 */
+ return parse_start_from(binfo, 0);
+}
+
+/* switch the memory mapping to BAR# @bar */
+static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
+{
+ return parse_start_from(binfo, bar);
+}
+
+static struct build_feature_devs_info *
+build_info_alloc_and_init(struct pci_dev *pdev)
+{
+ struct build_feature_devs_info *binfo;
+
+ binfo = devm_kzalloc(&pdev->dev, sizeof(*binfo), GFP_KERNEL);
+ if (binfo)
+ binfo->pdev = pdev;
+
+ return binfo;
+}
+
+static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
+{
+ if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
+ return FME_ID;
+
+ if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
+ return PORT_ID;
+
+ WARN_ON(1);
+ return FPGA_ID_MAX;
+}
+
+/*
+ * register current feature device, it is called when we need to switch to
+ * another feature parsing or we have parsed all features
+ */
+static int build_info_commit_dev(struct build_feature_devs_info *binfo)
+{
+ int ret;
+
+ if (!binfo->feature_dev)
+ return 0;
+
+ ret = platform_device_add(binfo->feature_dev);
+ if (!ret) {
+ struct cci_drvdata *drvdata;
+
+ drvdata = dev_get_drvdata(&binfo->pdev->dev);
+ if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
+ cci_pci_add_port_dev(binfo->pdev, binfo->feature_dev);
+ else
+ drvdata->fme_dev = get_device(&binfo->feature_dev->dev);
+
+ /*
+ * reset it to avoid build_info_free() freeing their resource.
+ *
+ * The resource of successfully registered feature devices
+ * will be freed by platform_device_unregister(). See the
+ * comments in build_info_create_dev().
+ */
+ binfo->feature_dev = NULL;
+ }
+
+ return ret;
+}
+
+static int
+build_info_create_dev(struct build_feature_devs_info *binfo,
+ enum fpga_id_type type, int feature_nr, const char *name)
+{
+ struct platform_device *fdev;
+ struct resource *res;
+ struct feature_platform_data *pdata;
+ int ret;
+
+ /* we will create a new device, commit current device first */
+ ret = build_info_commit_dev(binfo);
+ if (ret)
+ return ret;
+
+ /*
+ * we use -ENODEV as the initialization indicator which indicates
+ * whether the id need to be reclaimed
+ */
+ fdev = binfo->feature_dev = platform_device_alloc(name, -ENODEV);
+ if (!fdev)
+ return -ENOMEM;
+
+ fdev->id = alloc_fpga_id(type, &fdev->dev);
+ if (fdev->id < 0)
+ return fdev->id;
+
+ fdev->dev.parent = &binfo->parent_dev->dev;
+
+ /*
+ * we need not care the memory which is associated with the
+ * platform device. After call platform_device_unregister(),
+ * it will be automatically freed by device's
+ * release() callback, platform_device_release().
+ */
+ pdata = feature_platform_data_alloc_and_init(fdev, feature_nr);
+ if (!pdata)
+ return -ENOMEM;
+
+ /*
+ * the count should be initialized to 0 to make sure
+ *__fpga_port_enable() following __fpga_port_disable()
+ * works properly for port device.
+ * and it should always be 0 for fme device.
+ */
+ WARN_ON(pdata->disable_count);
+
+ fdev->dev.platform_data = pdata;
+ fdev->num_resources = feature_nr;
+ fdev->resource = kcalloc(feature_nr, sizeof(*res), GFP_KERNEL);
+ if (!fdev->resource)
+ return -ENOMEM;
+
+ return 0;
+}
+
+static int remove_feature_dev(struct device *dev, void *data)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+
+ platform_device_unregister(pdev);
+ return 0;
+}
+
+static int remove_parent_dev(struct device *dev, void *data)
+{
+ /* remove platform devices attached in the parent device */
+ device_for_each_child(dev, NULL, remove_feature_dev);
+ fpga_dev_destroy(to_fpga_dev(dev));
+ return 0;
+}
+
+static void remove_all_devs(struct pci_dev *pdev)
+{
+ /* remove parent device and all its children. */
+ device_for_each_child(&pdev->dev, NULL, remove_parent_dev);
+}
+
+static void build_info_free(struct build_feature_devs_info *binfo)
+{
+ if (!IS_ERR_OR_NULL(binfo->parent_dev))
+ remove_all_devs(binfo->pdev);
+
+ /*
+ * it is a valid id, free it. See comments in
+ * build_info_create_dev()
+ */
+ if (binfo->feature_dev && binfo->feature_dev->id >= 0)
+ free_fpga_id(feature_dev_id_type(binfo->feature_dev),
+ binfo->feature_dev->id);
+
+ platform_device_put(binfo->feature_dev);
+
+ devm_kfree(&binfo->pdev->dev, binfo);
+}
+
+#define FEATURE_TYPE_AFU 0x1
+#define FEATURE_TYPE_PRIVATE 0x3
+
+/* FME and PORT GUID are fixed */
+#define FEATURE_FME_GUID "f9e17764-38f0-82fe-e346-524ae92aafbf"
+#define FEATURE_PORT_GUID "6b355b87-b06c-9642-eb42-8d139398b43a"
+
+static bool feature_is_fme(struct feature_afu_header *afu_hdr)
+{
+ uuid_le u;
+
+ uuid_le_to_bin(FEATURE_FME_GUID, &u);
+
+ return !uuid_le_cmp(u, afu_hdr->guid);
+}
+
+static bool feature_is_port(struct feature_afu_header *afu_hdr)
+{
+ uuid_le u;
+
+ uuid_le_to_bin(FEATURE_PORT_GUID, &u);
+
+ return !uuid_le_cmp(u, afu_hdr->guid);
+}
+
+/*
+ * UAFU GUID is dynamic as it can be changed after FME downloads different
+ * Green Bitstream to the port, so we treat the unknown GUIDs which are
+ * attached on port's feature list as UAFU.
+ */
+static bool feature_is_UAFU(struct build_feature_devs_info *binfo)
+{
+ if (!binfo->feature_dev ||
+ feature_dev_id_type(binfo->feature_dev) != PORT_ID)
+ return false;
+
+ return true;
+}
+
+static void
+build_info_add_sub_feature(struct build_feature_devs_info *binfo,
+ int feature_id, const char *feature_name,
+ resource_size_t resource_size, void __iomem *start)
+{
+
+ struct platform_device *fdev = binfo->feature_dev;
+ struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
+ struct resource *res = &fdev->resource[feature_id];
+
+ res->start = pci_resource_start(binfo->pdev, binfo->current_bar) +
+ start - binfo->ioaddr;
+ res->end = res->start + resource_size - 1;
+ res->flags = IORESOURCE_MEM;
+ res->name = feature_name;
+
+ feature_platform_data_add(pdata, feature_id,
+ feature_name, feature_id, start);
+}
+
+struct feature_info {
+ const char *name;
+ resource_size_t resource_size;
+ int feature_index;
+};
+
+/* indexed by fme feature IDs which are defined in 'enum fme_feature_id'. */
+static struct feature_info fme_features[] = {
+ {
+ .name = FME_FEATURE_HEADER,
+ .resource_size = sizeof(struct feature_fme_header),
+ .feature_index = FME_FEATURE_ID_HEADER,
+ },
+ {
+ .name = FME_FEATURE_THERMAL_MGMT,
+ .resource_size = sizeof(struct feature_fme_thermal),
+ .feature_index = FME_FEATURE_ID_THERMAL_MGMT,
+ },
+ {
+ .name = FME_FEATURE_POWER_MGMT,
+ .resource_size = sizeof(struct feature_fme_power),
+ .feature_index = FME_FEATURE_ID_POWER_MGMT,
+ },
+ {
+ .name = FME_FEATURE_GLOBAL_PERF,
+ .resource_size = sizeof(struct feature_fme_gperf),
+ .feature_index = FME_FEATURE_ID_GLOBAL_PERF,
+ },
+ {
+ .name = FME_FEATURE_GLOBAL_ERR,
+ .resource_size = sizeof(struct feature_fme_err),
+ .feature_index = FME_FEATURE_ID_GLOBAL_ERR,
+ },
+ {
+ .name = FME_FEATURE_PR_MGMT,
+ .resource_size = sizeof(struct feature_fme_pr),
+ .feature_index = FME_FEATURE_ID_PR_MGMT,
+ }
+};
+
+/* indexed by port feature IDs which are defined in 'enum port_feature_id'. */
+static struct feature_info port_features[] = {
+ {
+ .name = PORT_FEATURE_HEADER,
+ .resource_size = sizeof(struct feature_port_header),
+ .feature_index = PORT_FEATURE_ID_HEADER,
+ },
+ {
+ .name = PORT_FEATURE_ERR,
+ .resource_size = sizeof(struct feature_port_error),
+ .feature_index = PORT_FEATURE_ID_ERROR,
+ },
+ {
+ .name = PORT_FEATURE_UMSG,
+ .resource_size = sizeof(struct feature_port_umsg),
+ .feature_index = PORT_FEATURE_ID_UMSG,
+ },
+ {
+ /* This feature isn't available for now */
+ .name = PORT_FEATURE_PR,
+ .resource_size = 0,
+ .feature_index = PORT_FEATURE_ID_PR,
+ },
+ {
+ .name = PORT_FEATURE_STP,
+ .resource_size = sizeof(struct feature_port_stp),
+ .feature_index = PORT_FEATURE_ID_STP,
+ },
+ {
+ /*
+ * For User AFU feature, its region size is not fixed, but
+ * reported by register PortCapability.mmio_size. Resource
+ * size of UAFU will be set while parse port device.
+ */
+ .name = PORT_FEATURE_UAFU,
+ .resource_size = 0,
+ .feature_index = PORT_FEATURE_ID_UAFU,
+ },
+};
+
+static int
+create_feature_instance(struct build_feature_devs_info *binfo,
+ void __iomem *start, struct feature_info *finfo)
+{
+ if (binfo->ioend - start < finfo->resource_size)
+ return -EINVAL;
+
+ build_info_add_sub_feature(binfo, finfo->feature_index, finfo->name,
+ finfo->resource_size, start);
+ return 0;
+}
+
+static int parse_feature_fme(struct build_feature_devs_info *binfo,
+ void __iomem *start)
+{
+ struct cci_drvdata *drvdata = dev_get_drvdata(&binfo->pdev->dev);
+ int ret;
+
+ ret = build_info_create_dev(binfo, FME_ID, fme_feature_num(),
+ FPGA_FEATURE_DEV_FME);
+ if (ret)
+ return ret;
+
+ if (drvdata->fme_dev) {
+ dev_err(&binfo->pdev->dev, "Multiple FMEs are detected.\n");
+ return -EINVAL;
+ }
+
+ return create_feature_instance(binfo, start,
+ &fme_features[FME_FEATURE_ID_HEADER]);
+}
+
+static int parse_feature_fme_private(struct build_feature_devs_info *binfo,
+ struct feature_header *hdr)
+{
+ struct feature_header header;
+
+ header.csr = readq(hdr);
+
+ if (header.id >= ARRAY_SIZE(fme_features)) {
+ dev_info(&binfo->pdev->dev, "FME feature id %x is not supported yet.\n",
+ header.id);
+ return 0;
+ }
+
+ return create_feature_instance(binfo, hdr, &fme_features[header.id]);
+}
+
+static int parse_feature_port(struct build_feature_devs_info *binfo,
+ void __iomem *start)
+{
+ int ret;
+
+ ret = build_info_create_dev(binfo, PORT_ID, port_feature_num(),
+ FPGA_FEATURE_DEV_PORT);
+ if (ret)
+ return ret;
+
+ return create_feature_instance(binfo, start,
+ &port_features[PORT_FEATURE_ID_HEADER]);
+}
+
+static void enable_port_uafu(struct build_feature_devs_info *binfo,
+ void __iomem *start)
+{
+ enum port_feature_id id = PORT_FEATURE_ID_UAFU;
+ struct feature_port_header *port_hdr;
+ struct feature_port_capability capability;
+
+ port_hdr = (struct feature_port_header *)start;
+ capability.csr = readq(&port_hdr->capability);
+ port_features[id].resource_size = capability.mmio_size << 10;
+
+ /*
+ * To enable User AFU, driver needs to clear reset bit on related port,
+ * otherwise the mmio space of this user AFU will be invalid.
+ */
+ if (port_features[id].resource_size)
+ fpga_port_reset(binfo->feature_dev);
+}
+
+static int parse_feature_port_private(struct build_feature_devs_info *binfo,
+ struct feature_header *hdr)
+{
+ struct feature_header header;
+ enum port_feature_id id;
+
+ header.csr = readq(hdr);
+ /*
+ * the region of port feature id is [0x10, 0x13], + 1 to reserve 0
+ * which is dedicated for port-hdr.
+ */
+ id = (header.id & 0x000f) + 1;
+
+ if (id >= ARRAY_SIZE(port_features)) {
+ dev_info(&binfo->pdev->dev, "Port feature id %x is not supported yet.\n",
+ header.id);
+ return 0;
+ }
+
+ return create_feature_instance(binfo, hdr, &port_features[id]);
+}
+
+static int parse_feature_port_uafu(struct build_feature_devs_info *binfo,
+ struct feature_header *hdr)
+{
+ enum port_feature_id id = PORT_FEATURE_ID_UAFU;
+ int ret;
+
+ if (port_features[id].resource_size) {
+ ret = create_feature_instance(binfo, hdr, &port_features[id]);
+ port_features[id].resource_size = 0;
+ } else {
+ dev_err(&binfo->pdev->dev, "the uafu feature header is mis-configured.\n");
+ ret = -EINVAL;
+ }
+
+ return ret;
+}
+
+static int parse_feature_afus(struct build_feature_devs_info *binfo,
+ struct feature_header *hdr)
+{
+ int ret;
+ struct feature_afu_header *afu_hdr, header;
+ void __iomem *start;
+ void __iomem *end = binfo->ioend;
+
+ start = hdr;
+ for (; start < end; start += header.next_afu) {
+ if (end - start < (sizeof(*afu_hdr) + sizeof(*hdr)))
+ return -EINVAL;
+
+ hdr = start;
+ afu_hdr = (struct feature_afu_header *) (hdr + 1);
+ header.csr = readq(&afu_hdr->csr);
+
+ if (feature_is_fme(afu_hdr)) {
+ ret = parse_feature_fme(binfo, hdr);
+ binfo->pfme_hdr = hdr;
+ if (ret)
+ return ret;
+ } else if (feature_is_port(afu_hdr)) {
+ ret = parse_feature_port(binfo, hdr);
+ enable_port_uafu(binfo, hdr);
+ if (ret)
+ return ret;
+ } else if (feature_is_UAFU(binfo)) {
+ ret = parse_feature_port_uafu(binfo, hdr);
+ if (ret)
+ return ret;
+ } else
+ dev_info(&binfo->pdev->dev, "AFU GUID %pUl is not supported yet.\n",
+ afu_hdr->guid.b);
+
+ if (!header.next_afu)
+ break;
+ }
+
+ return 0;
+}
+
+static int parse_feature_private(struct build_feature_devs_info *binfo,
+ struct feature_header *hdr)
+{
+ struct feature_header header;
+
+ header.csr = readq(hdr);
+
+ if (!binfo->feature_dev) {
+ dev_err(&binfo->pdev->dev, "the private feature %x does not belong to any AFU.\n",
+ header.id);
+ return -EINVAL;
+ }
+
+ switch (feature_dev_id_type(binfo->feature_dev)) {
+ case FME_ID:
+ return parse_feature_fme_private(binfo, hdr);
+ case PORT_ID:
+ return parse_feature_port_private(binfo, hdr);
+ default:
+ dev_info(&binfo->pdev->dev, "private feature %x belonging to AFU %s is not supported yet.\n",
+ header.id, binfo->feature_dev->name);
+ }
+ return 0;
+}
+
+static int parse_feature(struct build_feature_devs_info *binfo,
+ struct feature_header *hdr)
+{
+ struct feature_header header;
+ int ret = 0;
+
+ header.csr = readq(hdr);
+
+ switch (header.type) {
+ case FEATURE_TYPE_AFU:
+ ret = parse_feature_afus(binfo, hdr);
+ break;
+ case FEATURE_TYPE_PRIVATE:
+ ret = parse_feature_private(binfo, hdr);
+ break;
+ default:
+ dev_info(&binfo->pdev->dev,
+ "Feature Type %x is not supported.\n", hdr->type);
+ };
+
+ return ret;
+}
+
+static int
+parse_feature_list(struct build_feature_devs_info *binfo, void __iomem *start)
+{
+ struct feature_header *hdr, header;
+ void __iomem *end = binfo->ioend;
+ int ret = 0;
+
+ for (; start < end; start += header.next_header_offset) {
+ if (end - start < sizeof(*hdr)) {
+ dev_err(&binfo->pdev->dev, "The region is too small to contain a feature.\n");
+ ret = -EINVAL;
+ break;
+ }
+
+ hdr = (struct feature_header *)start;
+ ret = parse_feature(binfo, hdr);
+ if (ret)
+ break;
+
+ header.csr = readq(hdr);
+ if (!header.next_header_offset)
+ break;
+ }
+
+ return ret;
+}
+
+static int parse_ports_from_fme(struct build_feature_devs_info *binfo)
+{
+ struct feature_fme_header *fme_hdr;
+ struct feature_fme_port port;
+ int i = 0, ret = 0;
+
+ if (binfo->pfme_hdr == NULL) {
+ dev_dbg(&binfo->pdev->dev, "VF is detected.\n");
+ return ret;
+ }
+
+ fme_hdr = binfo->pfme_hdr;
+
+ do {
+ port.csr = readq(&fme_hdr->port[i]);
+ if (!port.port_implemented)
+ break;
+
+ ret = parse_switch_to(binfo, port.port_bar);
+ if (ret)
+ break;
+
+ ret = parse_feature_list(binfo,
+ binfo->ioaddr + port.port_offset);
+ if (ret)
+ break;
+ } while (++i < MAX_FPGA_PORT_NUM);
+
+ return ret;
+}
+
+static int create_init_drvdata(struct pci_dev *pdev)
+{
+ struct cci_drvdata *drvdata;
+
+ drvdata = devm_kzalloc(&pdev->dev, sizeof(*drvdata), GFP_KERNEL);
+ if (!drvdata)
+ return -ENOMEM;
+
+ mutex_init(&drvdata->lock);
+ INIT_LIST_HEAD(&drvdata->port_dev_list);
+ INIT_LIST_HEAD(&drvdata->regions);
+
+ dev_set_drvdata(&pdev->dev, drvdata);
+ return 0;
+}
+
+static void destroy_drvdata(struct pci_dev *pdev)
+{
+ struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
+
+ if (drvdata->fme_dev) {
+ /* fme device should be unregistered first. */
+ WARN_ON(device_is_registered(drvdata->fme_dev));
+ free_fpga_id(FME_ID, to_platform_device(drvdata->fme_dev)->id);
+ put_device(drvdata->fme_dev);
+ }
+
+ cci_pci_remove_port_devs(pdev);
+ cci_pci_release_regions(pdev);
+ dev_set_drvdata(&pdev->dev, NULL);
+ devm_kfree(&pdev->dev, drvdata);
+}
+
+static int cci_pci_create_feature_devs(struct pci_dev *pdev)
+{
+ struct build_feature_devs_info *binfo;
+ int ret;
+
+ binfo = build_info_alloc_and_init(pdev);
+ if (!binfo)
+ return -ENOMEM;
+
+ binfo->parent_dev = fpga_dev_create(&pdev->dev, INTEL_FPGA_DEV);
+ if (IS_ERR(binfo->parent_dev)) {
+ ret = PTR_ERR(binfo->parent_dev);
+ goto free_binfo_exit;
+ }
+
+ ret = parse_start(binfo);
+ if (ret)
+ goto free_binfo_exit;
+
+ ret = parse_feature_list(binfo, binfo->ioaddr);
+ if (ret)
+ goto free_binfo_exit;
+
+ ret = parse_ports_from_fme(binfo);
+ if (ret)
+ goto free_binfo_exit;
+
+ ret = build_info_commit_dev(binfo);
+ if (ret)
+ goto free_binfo_exit;
+
+ /*
+ * everything is okay, reset ->parent_dev to stop it being
+ * freed by build_info_free()
+ */
+ binfo->parent_dev = NULL;
+
+free_binfo_exit:
+ build_info_free(binfo);
+ return ret;
+}
+
/* PCI Device ID */
#define PCIe_DEVICE_ID_PF_INT_5_X 0xBCBD
#define PCIe_DEVICE_ID_PF_INT_6_X 0xBCC0
@@ -83,9 +900,18 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
goto release_region_exit;
}
- /* TODO: create and add the platform device per feature list */
+ ret = create_init_drvdata(pcidev);
+ if (ret)
+ goto release_region_exit;
+
+ ret = cci_pci_create_feature_devs(pcidev);
+ if (ret)
+ goto destroy_drvdata_exit;
+
return 0;
+destroy_drvdata_exit:
+ destroy_drvdata(pcidev);
release_region_exit:
pci_release_regions(pcidev);
disable_error_report_exit:
@@ -97,6 +923,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
static void cci_pci_remove(struct pci_dev *pcidev)
{
+ remove_all_devs(pcidev);
+ destroy_drvdata(pcidev);
pci_release_regions(pcidev);
pci_disable_pcie_error_reporting(pcidev);
pci_disable_device(pcidev);
@@ -111,14 +939,23 @@ static struct pci_driver cci_pci_driver = {
static int __init ccidrv_init(void)
{
+ int ret;
+
pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
- return pci_register_driver(&cci_pci_driver);
+ fpga_ids_init();
+
+ ret = pci_register_driver(&cci_pci_driver);
+ if (ret)
+ fpga_ids_destroy();
+
+ return ret;
}
static void __exit ccidrv_exit(void)
{
pci_unregister_driver(&cci_pci_driver);
+ fpga_ids_destroy();
}
module_init(ccidrv_init);
--
2.7.4
[toc] | [prev] | [next] | [standalone]
| From | Alan Tull <atull@kernel.org> |
|---|---|
| Date | 2017-04-03 23:50 +0200 |
| Subject | Re: [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features. |
| Message-ID | <tsi1j-40v-11@gated-at.bofh.it> |
| In reply to | #1613013 |
On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
> From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>
> Device Featuer List structure creates a link list of feature headers
> within the MMIO space to provide an extensiable way of adding features.
>
> The Intel FPGA PCIe driver walks through the feature headers to enumerate
> feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
> Function Unit (AFU), and their private sub features. For feature devices,
> it creates the platform devices and linked the private sub features into
> their platform data.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
> drivers/fpga/intel/Makefile | 2 +-
> drivers/fpga/intel/feature-dev.c | 139 +++++++
> drivers/fpga/intel/feature-dev.h | 342 ++++++++++++++++
> drivers/fpga/intel/pcie.c | 841 ++++++++++++++++++++++++++++++++++++++-
> 4 files changed, 1321 insertions(+), 3 deletions(-)
> create mode 100644 drivers/fpga/intel/feature-dev.c
> create mode 100644 drivers/fpga/intel/feature-dev.h
>
> diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> index 61fd8ea..c029940 100644
> --- a/drivers/fpga/intel/Makefile
> +++ b/drivers/fpga/intel/Makefile
> @@ -1,3 +1,3 @@
> obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
>
> -intel-fpga-pci-objs := pcie.o
> +intel-fpga-pci-objs := pcie.o feature-dev.o
> diff --git a/drivers/fpga/intel/feature-dev.c b/drivers/fpga/intel/feature-dev.c
> new file mode 100644
> index 0000000..6952566
> --- /dev/null
> +++ b/drivers/fpga/intel/feature-dev.c
> @@ -0,0 +1,139 @@
> +/*
> + * Intel FPGA Feature Device Driver
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + * Kang Luwei <luwei.kang@intel.com>
> + * Zhang Yi <yi.z.zhang@intel.com>
> + * Wu Hao <hao.wu@intel.com>
> + * Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *
> + * This work is licensed under a dual BSD/GPLv2 license. When using or
> + * redistributing this file, you may do so under either license. See the
> + * LICENSE.BSD file under this directory for the BSD license and see
> + * the COPYING file in the top-level directory for the GPLv2 license.
> + */
> +
> +#include "feature-dev.h"
> +
> +void feature_platform_data_add(struct feature_platform_data *pdata,
> + int index, const char *name,
> + int resource_index, void __iomem *ioaddr)
> +{
> + WARN_ON(index >= pdata->num);
> +
> + pdata->features[index].name = name;
> + pdata->features[index].resource_index = resource_index;
> + pdata->features[index].ioaddr = ioaddr;
> +}
> +
> +int feature_platform_data_size(int num)
> +{
> + return sizeof(struct feature_platform_data) +
> + num * sizeof(struct feature);
> +}
> +
> +struct feature_platform_data *
> +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
> +{
> + struct feature_platform_data *pdata;
> +
> + pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
> + if (pdata) {
> + pdata->dev = dev;
> + pdata->num = num;
> + mutex_init(&pdata->lock);
> + }
> +
> + return pdata;
> +}
> +
> +int fme_feature_num(void)
> +{
> + return FME_FEATURE_ID_MAX;
> +}
> +
> +int port_feature_num(void)
> +{
> + return PORT_FEATURE_ID_MAX;
> +}
> +
> +int fpga_port_id(struct platform_device *pdev)
> +{
> + struct feature_port_header *port_hdr;
> + struct feature_port_capability capability;
> +
> + port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> + PORT_FEATURE_ID_HEADER);
> + WARN_ON(!port_hdr);
> +
> + capability.csr = readq(&port_hdr->capability);
> + return capability.port_number;
> +}
> +EXPORT_SYMBOL_GPL(fpga_port_id);
> +
> +/*
> + * Enable Port by clear the port soft reset bit, which is set by default.
> + * The User AFU is unable to respond to any MMIO access while in reset.
> + * __fpga_port_enable function should only be used after __fpga_port_disable
> + * function.
> + */
> +void __fpga_port_enable(struct platform_device *pdev)
> +{
feature-dev.c is handling enumeration and adding port
enable/disable/etc functions for a specific port device. I see the
port as a fpga-bridge. The enumeration code should be separate from
the bridge code. Especially separate from a very specific bridge low
level device driver implementation, otherwise this becomes obsolete as
soon as you have another port device with a different register
implementation. Even if you handle that, then this enumeration code
isn't useable by other people who are using fpga-bridge. The
fpga-bridge framework exists to separate low level things like how to
enable/disable a specific bridge device from upper level code that
knows when to enable/disable it (fpga-region).
Alan
[toc] | [prev] | [next] | [standalone]
| From | Wu Hao <hao.wu@intel.com> |
|---|---|
| Date | 2017-04-05 14:10 +0200 |
| Subject | Re: [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features. |
| Message-ID | <tsRV7-2sp-13@gated-at.bofh.it> |
| In reply to | #1615576 |
On Mon, Apr 03, 2017 at 04:44:15PM -0500, Alan Tull wrote:
> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
> > From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >
> > Device Featuer List structure creates a link list of feature headers
> > within the MMIO space to provide an extensiable way of adding features.
> >
> > The Intel FPGA PCIe driver walks through the feature headers to enumerate
> > feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
> > Function Unit (AFU), and their private sub features. For feature devices,
> > it creates the platform devices and linked the private sub features into
> > their platform data.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> > Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > ---
> > drivers/fpga/intel/Makefile | 2 +-
> > drivers/fpga/intel/feature-dev.c | 139 +++++++
> > drivers/fpga/intel/feature-dev.h | 342 ++++++++++++++++
> > drivers/fpga/intel/pcie.c | 841 ++++++++++++++++++++++++++++++++++++++-
> > 4 files changed, 1321 insertions(+), 3 deletions(-)
> > create mode 100644 drivers/fpga/intel/feature-dev.c
> > create mode 100644 drivers/fpga/intel/feature-dev.h
> >
> > diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> > index 61fd8ea..c029940 100644
> > --- a/drivers/fpga/intel/Makefile
> > +++ b/drivers/fpga/intel/Makefile
> > @@ -1,3 +1,3 @@
> > obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
> >
> > -intel-fpga-pci-objs := pcie.o
> > +intel-fpga-pci-objs := pcie.o feature-dev.o
> > diff --git a/drivers/fpga/intel/feature-dev.c b/drivers/fpga/intel/feature-dev.c
> > new file mode 100644
> > index 0000000..6952566
> > --- /dev/null
> > +++ b/drivers/fpga/intel/feature-dev.c
> > @@ -0,0 +1,139 @@
> > +/*
> > + * Intel FPGA Feature Device Driver
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + * Kang Luwei <luwei.kang@intel.com>
> > + * Zhang Yi <yi.z.zhang@intel.com>
> > + * Wu Hao <hao.wu@intel.com>
> > + * Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > + *
> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
> > + * redistributing this file, you may do so under either license. See the
> > + * LICENSE.BSD file under this directory for the BSD license and see
> > + * the COPYING file in the top-level directory for the GPLv2 license.
> > + */
> > +
> > +#include "feature-dev.h"
> > +
> > +void feature_platform_data_add(struct feature_platform_data *pdata,
> > + int index, const char *name,
> > + int resource_index, void __iomem *ioaddr)
> > +{
> > + WARN_ON(index >= pdata->num);
> > +
> > + pdata->features[index].name = name;
> > + pdata->features[index].resource_index = resource_index;
> > + pdata->features[index].ioaddr = ioaddr;
> > +}
> > +
> > +int feature_platform_data_size(int num)
> > +{
> > + return sizeof(struct feature_platform_data) +
> > + num * sizeof(struct feature);
> > +}
> > +
> > +struct feature_platform_data *
> > +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
> > +{
> > + struct feature_platform_data *pdata;
> > +
> > + pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
> > + if (pdata) {
> > + pdata->dev = dev;
> > + pdata->num = num;
> > + mutex_init(&pdata->lock);
> > + }
> > +
> > + return pdata;
> > +}
> > +
> > +int fme_feature_num(void)
> > +{
> > + return FME_FEATURE_ID_MAX;
> > +}
> > +
> > +int port_feature_num(void)
> > +{
> > + return PORT_FEATURE_ID_MAX;
> > +}
> > +
> > +int fpga_port_id(struct platform_device *pdev)
> > +{
> > + struct feature_port_header *port_hdr;
> > + struct feature_port_capability capability;
> > +
> > + port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > + PORT_FEATURE_ID_HEADER);
> > + WARN_ON(!port_hdr);
> > +
> > + capability.csr = readq(&port_hdr->capability);
> > + return capability.port_number;
> > +}
> > +EXPORT_SYMBOL_GPL(fpga_port_id);
> > +
> > +/*
> > + * Enable Port by clear the port soft reset bit, which is set by default.
> > + * The User AFU is unable to respond to any MMIO access while in reset.
> > + * __fpga_port_enable function should only be used after __fpga_port_disable
> > + * function.
> > + */
> > +void __fpga_port_enable(struct platform_device *pdev)
> > +{
>
> feature-dev.c is handling enumeration and adding port
> enable/disable/etc functions for a specific port device. I see the
> port as a fpga-bridge. The enumeration code should be separate from
> the bridge code. Especially separate from a very specific bridge low
> level device driver implementation, otherwise this becomes obsolete as
> soon as you have another port device with a different register
> implementation. Even if you handle that, then this enumeration code
> isn't useable by other people who are using fpga-bridge. The
> fpga-bridge framework exists to separate low level things like how to
> enable/disable a specific bridge device from upper level code that
> knows when to enable/disable it (fpga-region).
Hi Alan
The major purpose of feature-dev is to create infrastructure for feature
device. Please refer to patch 7. It abstracts feature device common
data structures and functions in feature-dev.c for FME and AFU now (but
may be more in the future). The reason we add port enable/disable/etc
fuctions there for code reuse. e.g FME driver needs port enable/disable
for PR (in the future, to implement the fpga bridge enable_set function),
AFU driver needs similar code to implement reset interface for user space
application, PCIe driver needs port enable to make AFU MMIO region valid,
then it can access Device Feature Header inside this AFU MMIO during
enumeration. Other fpga_port_* are all for code reuse purpose too. So we
should not put these function in the same feature-dev.c but a seperated
file?
Thanks
Hao
>
> Alan
[toc] | [prev] | [next] | [standalone]
| From | Alan Tull <atull@kernel.org> |
|---|---|
| Date | 2017-04-05 00:20 +0200 |
| Subject | Re: [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features. |
| Message-ID | <tsEXT-2s7-19@gated-at.bofh.it> |
| In reply to | #1613013 |
On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote: > From: Xiao Guangrong <guangrong.xiao@linux.intel.com> > > Device Featuer List structure creates a link list of feature headers > within the MMIO space to provide an extensiable way of adding features. > > The Intel FPGA PCIe driver walks through the feature headers to enumerate > feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated > Function Unit (AFU), and their private sub features. For feature devices, > it creates the platform devices and linked the private sub features into > their platform data. > I'm still looking at this code and it's pretty new to me, but I think it would be desirable and not really hard to separate the code that enumerates from all the fixed feature code. So pcie.c could see that there is a pci device out there that it knows has these memory mapped enumeration structs. So it goes and reads the structs, parses them, and knows what drivers to probe. The FME and AFU and other fpga device drivers could register their guids with the framework and be discoverable in that way. That way if you need to implement a different FME or anything else, it could be added with a new guid to this framework and would get enumerated. I'm thinking of the future and of more general usability of this code. Then the enumeration code wouldn't have to be 'intel' code or even code dedicated to FME's and AFU's. Any FPGA with a PCIe port that has the right id's could have this struct and use this enumeration method. Actually if the parse* enumeration code could be in a separate file as helper functions for the pcie code, this stuff would be structured for future support this of the same framework on embedded FPGA devices. Alan
[toc] | [prev] | [next] | [standalone]
| From | Wu Hao <hao.wu@intel.com> |
|---|---|
| Date | 2017-04-05 16:20 +0200 |
| Subject | Re: [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features. |
| Message-ID | <tsTWW-3H6-17@gated-at.bofh.it> |
| In reply to | #1616414 |
On Tue, Apr 04, 2017 at 05:09:23PM -0500, Alan Tull wrote: > On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote: > > From: Xiao Guangrong <guangrong.xiao@linux.intel.com> > > > > Device Featuer List structure creates a link list of feature headers > > within the MMIO space to provide an extensiable way of adding features. > > > > The Intel FPGA PCIe driver walks through the feature headers to enumerate > > feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated > > Function Unit (AFU), and their private sub features. For feature devices, > > it creates the platform devices and linked the private sub features into > > their platform data. > > > > I'm still looking at this code and it's pretty new to me, but I think > it would be desirable and not really hard to separate the code > that enumerates from all the fixed feature code. So pcie.c could see > that there is a pci device out there that it knows has these memory > mapped enumeration structs. So it goes and reads the structs, parses > them, and knows what drivers to probe. The FME and AFU and other > fpga device drivers could register their guids with the framework > and be discoverable in that way. > > That way if you need to implement a different FME or anything else, it > could be added with a new guid to this framework and would get > enumerated. I'm thinking of the future and of more general usability > of this code. > > Then the enumeration code wouldn't have to be 'intel' code or even > code dedicated to FME's and AFU's. Any FPGA with a PCIe > port that has the right id's could have this struct and use this > enumeration method. Actually if the parse* enumeration code could be in a > separate file as helper functions for the pcie code, this stuff would > be structured for future support this of the same framework on > embedded FPGA devices. > Hi Alan Thank you very much for the review and comments. Actually I am not sure if the 'Device Feature List' is designed for common usage or not, but per current implementation of Port/AFU and FME, they did not use the exact same way for enumeration. e.g PCIe driver reads register under Port to know the AFU MMIO region size. So for each module, it has its own method to enumerate and prepare the resource for its platform device. Other developers may not be able to use them for a new module. From the whole device's point of view, do enumeration for all modules is still a device specific thing. e.g 'Device Feature List' of the FME is in PCI BAR0, but the location of Port/AFU's 'Device Feature List' is not linked to FME's Device Feature List, but given (PCI BAR + offset) by a FME register. So the process of enumeration may be totally different in another device with different module. I don't expect FME can be used without PCIE or Port/AFU now, as it ties to them so closely. e.g give PCI Bar info for port, registers to support PCI SRIOV function. And so does PCIe and AFU driver, e.g PCIe driver is not only handling the enumeration, but also manage ports and access FME registers for SRIOV function support (sriov related code is not included in this patch set). If we have any PCIE FPGA has FME, then we can reuse this Intel FPGA driver directly, but if no FME, then most code can't be reused. I fully understand your point for code reuse and agree with you that parse* enumeration code could be in a common place as helper function. But for others, I still have concern due to current hardware implementation. Anyway I will continue consideration on this. Thanks Hao > Alan
[toc] | [prev] | [next] | [standalone]
| From | Wu Hao <hao.wu@intel.com> |
|---|---|
| Date | 2017-03-30 14:20 +0200 |
| Subject | [PATCH 10/16] fpga: intel: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support |
| Message-ID | <tqHdw-69w-27@gated-at.bofh.it> |
| In reply to | #1613003 |
FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
need to be supported by all feature devices drivers including FME and AFU.
Userspace application can use these ioctl interfaces to get the API info
and check if specific extension is supported or not in current driver.
This patch implements above 2 ioctls in Intel FPGA Management Engine (FME)
driver.
Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
Documentation/ioctl/ioctl-number.txt | 1 +
drivers/fpga/intel/fme-main.c | 12 +++++++++
include/uapi/linux/intel-fpga.h | 52 ++++++++++++++++++++++++++++++++++++
3 files changed, 65 insertions(+)
create mode 100644 include/uapi/linux/intel-fpga.h
diff --git a/Documentation/ioctl/ioctl-number.txt b/Documentation/ioctl/ioctl-number.txt
index 08244be..462f4a5 100644
--- a/Documentation/ioctl/ioctl-number.txt
+++ b/Documentation/ioctl/ioctl-number.txt
@@ -322,6 +322,7 @@ Code Seq#(hex) Include File Comments
0xB3 00 linux/mmc/ioctl.h
0xB4 00-0F linux/gpio.h <mailto:linux-gpio@vger.kernel.org>
0xB5 00-0F uapi/linux/rpmsg.h <mailto:linux-remoteproc@vger.kernel.org>
+0xB6 all linux/intel-fpga.h
0xC0 00-0F linux/usb/iowarrior.h
0xCA 00-0F uapi/misc/cxl.h
0xCA 80-8F uapi/scsi/cxlflash_ioctl.h
diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
index a7c69fc..36d0c4c 100644
--- a/drivers/fpga/intel/fme-main.c
+++ b/drivers/fpga/intel/fme-main.c
@@ -20,6 +20,7 @@
#include <linux/kernel.h>
#include <linux/module.h>
+#include <linux/intel-fpga.h>
#include "feature-dev.h"
@@ -104,6 +105,13 @@ static struct feature_driver fme_feature_drvs[] = {
},
};
+static long fme_ioctl_check_extension(struct feature_platform_data *pdata,
+ unsigned long arg)
+{
+ /* No extension support for now */
+ return 0;
+}
+
static int fme_open(struct inode *inode, struct file *filp)
{
struct platform_device *fdev = fpga_inode_to_feature_dev(inode);
@@ -142,6 +150,10 @@ static long fme_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
switch (cmd) {
+ case FPGA_GET_API_VERSION:
+ return FPGA_API_VERSION;
+ case FPGA_CHECK_EXTENSION:
+ return fme_ioctl_check_extension(pdata, arg);
default:
/*
* Let sub-feature's ioctl function to handle the cmd
diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
new file mode 100644
index 0000000..992e556
--- /dev/null
+++ b/include/uapi/linux/intel-fpga.h
@@ -0,0 +1,52 @@
+/*
+ * Header File for Intel FPGA User API
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ * Kang Luwei <luwei.kang@intel.com>
+ * Zhang Yi <yi.z.zhang@intel.com>
+ * Wu Hao <hao.wu@intel.com>
+ * Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under drivers/fpga/intel for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+
+#ifndef _UAPI_LINUX_INTEL_FPGA_H
+#define _UAPI_LINUX_INTEL_FPGA_H
+
+#define FPGA_API_VERSION 0
+
+/*
+ * The IOCTL interface for Intel FPGA is designed for extensibility by
+ * embedding the structure length (argsz) and flags into structures passed
+ * between kernel and userspace. This design referenced the VFIO IOCTL
+ * interface (include/uapi/linux/vfio.h).
+ */
+
+#define FPGA_MAGIC 0xB6
+
+#define FPGA_BASE 0
+
+/**
+ * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
+ *
+ * Report the version of the driver API.
+ * Return: Driver API Version.
+ */
+
+#define FPGA_GET_API_VERSION _IO(FPGA_MAGIC, FPGA_BASE + 0)
+
+/**
+ * FPGA_CHECK_EXTENSION - _IO(FPGA_MAGIC, FPGA_BASE + 1)
+ *
+ * Check whether an extension is supported.
+ * Return: 0 if not supported, otherwise the extension is supported.
+ */
+
+#define FPGA_CHECK_EXTENSION _IO(FPGA_MAGIC, FPGA_BASE + 1)
+
+#endif /* _UAPI_INTEL_FPGA_H */
--
2.7.4
[toc] | [prev] | [next] | [standalone]
| From | Wu Hao <hao.wu@intel.com> |
|---|---|
| Date | 2017-03-30 14:20 +0200 |
| Subject | [PATCH 14/16] fpga: intel: afu add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support |
| Message-ID | <tqHdw-69w-29@gated-at.bofh.it> |
| In reply to | #1613003 |
FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
need to be supported by all feature devices drivers including FME and AFU.
This patch implements above 2 ioctls in Intel FPGA Accelerated Function
Unit (AFU) driver.
Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
drivers/fpga/intel/afu-main.c | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/drivers/fpga/intel/afu-main.c b/drivers/fpga/intel/afu-main.c
index 7166d5c..89d4b2f 100644
--- a/drivers/fpga/intel/afu-main.c
+++ b/drivers/fpga/intel/afu-main.c
@@ -124,6 +124,13 @@ static int afu_release(struct inode *inode, struct file *filp)
return 0;
}
+static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
+ unsigned long arg)
+{
+ /* No extension support for now */
+ return 0;
+}
+
static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
{
struct platform_device *pdev = filp->private_data;
@@ -134,6 +141,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
switch (cmd) {
+ case FPGA_GET_API_VERSION:
+ return FPGA_API_VERSION;
+ case FPGA_CHECK_EXTENSION:
+ return afu_ioctl_check_extension(pdata, arg);
default:
/*
* Let sub-feature's ioctl function to handle the cmd
--
2.7.4
[toc] | [prev] | [next] | [standalone]
| From | Wu Hao <hao.wu@intel.com> |
|---|---|
| Date | 2017-03-30 14:20 +0200 |
| Subject | [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview |
| Message-ID | <tqHdw-69w-33@gated-at.bofh.it> |
| In reply to | #1613003 |
Add a document for Intel FPGA driver overview. Signed-off-by: Enno Luebbers <enno.luebbers@intel.com> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com> Signed-off-by: Wu Hao <hao.wu@intel.com> --- Documentation/fpga/intel-fpga.txt | 259 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 259 insertions(+) create mode 100644 Documentation/fpga/intel-fpga.txt diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt new file mode 100644 index 0000000..9396cea --- /dev/null +++ b/Documentation/fpga/intel-fpga.txt @@ -0,0 +1,259 @@ +=============================================================================== + Intel FPGA driver Overview +------------------------------------------------------------------------------- + Enno Luebbers <enno.luebbers@intel.com> + Xiao Guangrong <guangrong.xiao@linux.intel.com> + Wu Hao <hao.wu@intel.com> + +The Intel FPGA driver provides interfaces for userspace applications to +configure, enumerate, open, and access FPGA accelerators on platforms equipped +with Intel(R) FPGA solutions and enables system level management functions such +as FPGA reconfiguration, power management, and virtualization. + +HW Architecture +=============== +From the OS's point of view, the FPGA hardware appears as a regular PCIe device. +The FPGA device memory is organized using a predefined data structure (Device +Feature List). Features supported by the particular FPGA device are exposed +through these data structures, as illustrated below: + + +-------------------------------+ +-------------+ + | PF | | VF | + +-------------------------------+ +-------------+ + ^ ^ ^ ^ + | | | | ++-----|------------|---------|--------------|-------+ +| | | | | | +| +-----+ +-------+ +-------+ +-------+ | +| | FME | | Port0 | | Port1 | | Port2 | | +| +-----+ +-------+ +-------+ +-------+ | +| ^ ^ ^ | +| | | | | +| +-------+ +------+ +-------+ | +| | AFU | | AFU | | AFU | | +| +-------+ +------+ +-------+ | +| | +| FPGA PCIe Device | ++---------------------------------------------------+ + +The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be +used to assign individual accelerators to virtual machines . + +FME (FPGA Management Engine) +============================ +The FPGA Management Enging performs power and thermal management, error +reporting, reconfiguration, performance reporting, and other infrastructure +functions. Each FPGA has one FME, which is always accessed through the physical +function (PF). + +User-space applications can acquire exclusive access to the FME using open(), +and release it using close(). + +The following functions are exposed through ioctls: + + Get driver API version (FPGA_GET_API_VERSION) + Check for extensions (FPGA_CHECK_EXTENSION) + Assign port to PF (FPGA_FME_PORT_ASSIGN) + Release port from PF (FPGA_FME_PORT_RELEASE) + Program bitstream (FPGA_FME_PORT_PR) + +More functions are exposed through sysfs +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/): + + Read bitstream ID (bitstream_id) + Read bitstream metadata (bitstream_metadata) + Read number of ports (ports_num) + Read socket ID (socket_id) + Read performance counters (perf/) + Power management (power_mgmt/) + Thermal management (thermal_mgmt/) + Error reporting (errors/) + +PORT +==== +A port represents the interface between the static FPGA fabric (the "blue +bitstream") and a partially reconfigurable region containing an AFU (the "green +bitstream"). It controls the communication from SW to the accelerator and +exposes features such as reset and debug. + +A PCIe device may have several ports and each port can be released from PF by +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov +sysfs interface. + +AFU +=== +An AFU is attached to a port and exposes a 256k MMIO region to be used for +accelerator-specific control registers. + +User-space applications can acquire exclusive access to an AFU attached to a +port by using open() on the port device node, and release it using close(). + +The following functions are exposed through ioctls: + + Get driver API version (FPGA_GET_API_VERSION) + Check for extensions (FPGA_CHECK_EXTENSION) + Get port info (FPGA_PORT_GET_INFO) + Get MMIO region info (FPGA_PORT_GET_REGION_INFO) + Map DMA buffer (FPGA_PORT_DMA_MAP) + Unmap DMA buffer (FPGA_PORT_DMA_UNMAP) + Reset AFU (FPGA_PORT_RESET) + Enable UMsg (FPGA_PORT_UMSG_ENABLE) + Disable UMsg (FPGA_PORT_UMSG_DISABLE) + Set UMsg mode (FPGA_PORT_UMSG_SET_MODE) + Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR) + +User-space applications can also mmap() accelerator MMIO regions. + +More functions are exposed through sysfs: +(/sys/class/fpga/fpga.n/intel-fpga-port.m/): + + Read Accelerator GUID (afu_id) + Error reporting (errors/) + +Partial Reconfiguration +======================= +As mentioned above, accelerators can be reconfigured through partial +reconfiguration of a green bitstream file (GBS). The green bitstream must have +been generated for the exact blue bitstream and targeted reconfigurable region +(port) of the FPGA; otherwise, the reconfiguration operation will fail and +possibly cause system instability. This compatibility can be checked by +comparing the interface ID noted in the GBS header against the interface ID +exposed by the FME through sysfs (see above). This check is usually done by +user-space before calling the reconfiguration IOCTL. + +FPGA virtualization +=================== +To enable accessing an accelerator from applications running in a VM, the +respective AFU's port needs to be assigned to a VF using the following steps: + + a) The PF owns all AFU ports by default. Any port that needs to be reassigned + to a VF must be released from PF firstly through the FPGA_FME_PORT_RELEASE + ioctl on the FME device. + + b) Once N ports are released from PF, then user can use below command to + enable SRIOV and VFs. Each VF owns only one Port with AFU. + + echo N > $PCI_DEVICE_PATH/sriov_numvfs + + c) Pass through the VFs to VMs + + d) The AFU under VF is accessiable from applications in VM (using the same + driver inside the VF). + +Note the an FME can't be assigned to a VF, thus PR and other management +functions are only available via the PF. + + +Driver organization +=================== + + +------------------+ +---------+ | +---------+ + | +-------+ | | | | | | + | | FPGA | FME | | AFU | | | AFU | + | |Manager| Module | | Module | | | Module | + | +-------+ | | | | | | + +------------------+ +---------+ | +---------+ + +-----------------------+ | +-----------------------+ + | FPGA Container Device | | | FPGA Container Device | + +-----------------------+ | +-----------------------+ + +------------------+ | +------------------+ + | FPGA PCIE Module | | Virtual | FPGA PCIE Module | + +------------------+ Host | Machine +------------------+ + ------------------------------------ | ------------------------------ + +---------------+ | +---------------+ + | PCI PF Device | | | PCI VF Device | + +---------------+ | +---------------+ + +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe device +driver is always loaded first once a FPGA PCIE PF or VF device is detected. This +driver plays an infrastructural role in the driver architecuture. It: + + a) creates FPGA container device as parent of the feature devices. + b) walks through the Device Feature List, which is implemented in PCIE + device BAR memory, to discover feature devices and their sub features + and create platform device for them under the container device. + c) supports SRIOV. + d) introduces the feature device infrastructure, which abstracts + operations for sub features and exposes common functions to feature + device drivers. + +The FPGA Management Engine (FME) driver is a platform driver which is loaded +automatically after FME platform device creation from the PCIE driver. It +provides the key features for FPGA management, including: + + a) Power and thermal management, error reporting, performance reporting + and other infrastructure functions. Users can access these functions + via sysfs interfaces exposed by FME driver. + b) Paritial Reconfiguration. The FME driver registers a FPGA Manager + during PR sub feature initialization; once it receives an + FPGA_FME_PORT_PR ioctl from user, it invokes the common interface + function from FPGA Manager to complete the partial reconfiguration of + the bitstream to the given port. + c) Port management for virtualization. The FME driver introduces two + ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and + FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is + released from the PF, it can be assigned to the VF through the SRIOV + interfaces provided by PCIE driver. (Refer to "FPGA virtualization" + for more details). + +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver +is probed once the AFU platform device is created. The main function of this +module is to provide an interface for userspace applications to access the +individual accelerators, including basic reset control on port, AFU MMIO region +export, dma buffer mapping service, UMsg notification, and remote debug +functions (see above). + + +Device enumeration +================== +This section introduces how applications enumerate the fpga device from +the sysfs hierarchy under /sys/class/fpga. + +In the example below, two Intel(R) FPGA devices are installed in the host. Each +fpga device has one FME and two ports (AFUs). + +For each FPGA device, a device director is created under /sys/class/fpga/: + + /sys/class/fpga/fpga.0 + /sys/class/fpga/fpga.1 + +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's name. +Application can retrieve name information via the sysfs interface: + + /sys/class/fpga/fpga.0/name + +Each node has one FME and two ports (AFUs) as child devices: + + /sys/class/fpga/fpga.0/intel-fpga-fme.0 + /sys/class/fpga/fpga.0/intel-fpga-port.0 + /sys/class/fpga/fpga.0/intel-fpga-port.1 + + /sys/class/fpga/fpga.1/intel-fpga-fme.1 + /sys/class/fpga/fpga.1/intel-fpga-port.2 + /sys/class/fpga/fpga.1/intel-fpga-port.3 + +In general, the FME/AFU sysfs interfaces are named as follows: + + /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/ + /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/ + +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all +ports. + +The device nodes used for ioctl() or mmap() can be referenced through: + + /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev + /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev + + +Open discussions +================ +The current FME driver does not provide user space access to the FME MMIO +region, but exposes access through sysfs and ioctls. It also provides an FPGA +manger interface for partial reconfiguration (PR), but does not make use of +fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled inside +the FME, and fpga-region depends on device tree which is not used at all. There +are patches from Alan Tull to separate the device tree specific code and +introduce a sysfs interface for PR. We plan to add fpga-regions support in the +driver once the related patches get merged. Then the FME driver should create +one fpga-region for each Port/AFU. -- 2.7.4
[toc] | [prev] | [next] | [standalone]
| From | matthew.gerlach@linux.intel.com |
|---|---|
| Date | 2017-03-31 20:30 +0200 |
| Subject | Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview |
| Message-ID | <tr9t8-8dB-13@gated-at.bofh.it> |
| In reply to | #1613016 |
On Thu, 30 Mar 2017, Wu Hao wrote: Hi Wu Hao, Great documentation. I'm looking forward to diving into the rest of the patches. Please see my comments inline. Matthew Gerlach > Add a document for Intel FPGA driver overview. > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com> > Signed-off-by: Wu Hao <hao.wu@intel.com> > --- > Documentation/fpga/intel-fpga.txt | 259 ++++++++++++++++++++++++++++++++++++++ > 1 file changed, 259 insertions(+) > create mode 100644 Documentation/fpga/intel-fpga.txt > > diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt > new file mode 100644 > index 0000000..9396cea > --- /dev/null > +++ b/Documentation/fpga/intel-fpga.txt > @@ -0,0 +1,259 @@ > +=============================================================================== > + Intel FPGA driver Overview > +------------------------------------------------------------------------------- > + Enno Luebbers <enno.luebbers@intel.com> > + Xiao Guangrong <guangrong.xiao@linux.intel.com> > + Wu Hao <hao.wu@intel.com> > + > +The Intel FPGA driver provides interfaces for userspace applications to > +configure, enumerate, open, and access FPGA accelerators on platforms equipped > +with Intel(R) FPGA solutions and enables system level management functions such > +as FPGA reconfiguration, power management, and virtualization. > + From a Linux kernel perspective, I'm not sure this is the best name for this code. The name gives me the impression that it is a driver for all Intel FPGAs, but not all Intel FPGAs are connected to the processor over a PCIe bus. The processor could be directely connected like the Arria10 SOCFPGA. Such a processor could certainly benefit from this accelerator usage model. In an extreme case, couldn't a processor in the FPGA, running Linux, also benefit from this accelerator model? Is this code a "FPGA Accelerator Framework"? > +HW Architecture > +=============== > +From the OS's point of view, the FPGA hardware appears as a regular PCIe device. > +The FPGA device memory is organized using a predefined data structure (Device > +Feature List). Features supported by the particular FPGA device are exposed > +through these data structures, as illustrated below: > + > + +-------------------------------+ +-------------+ > + | PF | | VF | > + +-------------------------------+ +-------------+ > + ^ ^ ^ ^ > + | | | | > ++-----|------------|---------|--------------|-------+ > +| | | | | | > +| +-----+ +-------+ +-------+ +-------+ | > +| | FME | | Port0 | | Port1 | | Port2 | | > +| +-----+ +-------+ +-------+ +-------+ | > +| ^ ^ ^ | > +| | | | | > +| +-------+ +------+ +-------+ | > +| | AFU | | AFU | | AFU | | > +| +-------+ +------+ +-------+ | > +| | > +| FPGA PCIe Device | > ++---------------------------------------------------+ > + > +The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be > +used to assign individual accelerators to virtual machines . Does this HW Architecture require an Intel FPGA? Couldn't any vendors FPGA be used as long as it presented itself the PCIe bus the same and contained an appropriate Device Feature List? > + > +FME (FPGA Management Engine) > +============================ > +The FPGA Management Enging performs power and thermal management, error > +reporting, reconfiguration, performance reporting, and other infrastructure > +functions. Each FPGA has one FME, which is always accessed through the physical > +function (PF). > + > +User-space applications can acquire exclusive access to the FME using open(), > +and release it using close(). > + > +The following functions are exposed through ioctls: > + > + Get driver API version (FPGA_GET_API_VERSION) > + Check for extensions (FPGA_CHECK_EXTENSION) > + Assign port to PF (FPGA_FME_PORT_ASSIGN) > + Release port from PF (FPGA_FME_PORT_RELEASE) > + Program bitstream (FPGA_FME_PORT_PR) > + > +More functions are exposed through sysfs > +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/): > + > + Read bitstream ID (bitstream_id) > + Read bitstream metadata (bitstream_metadata) > + Read number of ports (ports_num) > + Read socket ID (socket_id) > + Read performance counters (perf/) > + Power management (power_mgmt/) > + Thermal management (thermal_mgmt/) > + Error reporting (errors/) > + > +PORT > +==== > +A port represents the interface between the static FPGA fabric (the "blue > +bitstream") and a partially reconfigurable region containing an AFU (the "green > +bitstream"). It controls the communication from SW to the accelerator and > +exposes features such as reset and debug. > + > +A PCIe device may have several ports and each port can be released from PF by > +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov > +sysfs interface. > + > +AFU > +=== > +An AFU is attached to a port and exposes a 256k MMIO region to be used for > +accelerator-specific control registers. > + > +User-space applications can acquire exclusive access to an AFU attached to a > +port by using open() on the port device node, and release it using close(). > + > +The following functions are exposed through ioctls: > + > + Get driver API version (FPGA_GET_API_VERSION) > + Check for extensions (FPGA_CHECK_EXTENSION) > + Get port info (FPGA_PORT_GET_INFO) > + Get MMIO region info (FPGA_PORT_GET_REGION_INFO) > + Map DMA buffer (FPGA_PORT_DMA_MAP) > + Unmap DMA buffer (FPGA_PORT_DMA_UNMAP) > + Reset AFU (FPGA_PORT_RESET) > + Enable UMsg (FPGA_PORT_UMSG_ENABLE) > + Disable UMsg (FPGA_PORT_UMSG_DISABLE) > + Set UMsg mode (FPGA_PORT_UMSG_SET_MODE) > + Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR) > + > +User-space applications can also mmap() accelerator MMIO regions. > + > +More functions are exposed through sysfs: > +(/sys/class/fpga/fpga.n/intel-fpga-port.m/): > + > + Read Accelerator GUID (afu_id) > + Error reporting (errors/) > + > +Partial Reconfiguration > +======================= > +As mentioned above, accelerators can be reconfigured through partial > +reconfiguration of a green bitstream file (GBS). The green bitstream must have > +been generated for the exact blue bitstream and targeted reconfigurable region > +(port) of the FPGA; otherwise, the reconfiguration operation will fail and > +possibly cause system instability. This compatibility can be checked by > +comparing the interface ID noted in the GBS header against the interface ID > +exposed by the FME through sysfs (see above). This check is usually done by > +user-space before calling the reconfiguration IOCTL. > + > +FPGA virtualization > +=================== > +To enable accessing an accelerator from applications running in a VM, the > +respective AFU's port needs to be assigned to a VF using the following steps: > + > + a) The PF owns all AFU ports by default. Any port that needs to be reassigned > + to a VF must be released from PF firstly through the FPGA_FME_PORT_RELEASE > + ioctl on the FME device. > + > + b) Once N ports are released from PF, then user can use below command to > + enable SRIOV and VFs. Each VF owns only one Port with AFU. > + > + echo N > $PCI_DEVICE_PATH/sriov_numvfs > + > + c) Pass through the VFs to VMs > + > + d) The AFU under VF is accessiable from applications in VM (using the same > + driver inside the VF). > + > +Note the an FME can't be assigned to a VF, thus PR and other management > +functions are only available via the PF. > + > + > +Driver organization > +=================== > + > + +------------------+ +---------+ | +---------+ > + | +-------+ | | | | | | > + | | FPGA | FME | | AFU | | | AFU | > + | |Manager| Module | | Module | | | Module | > + | +-------+ | | | | | | > + +------------------+ +---------+ | +---------+ > + +-----------------------+ | +-----------------------+ > + | FPGA Container Device | | | FPGA Container Device | > + +-----------------------+ | +-----------------------+ > + +------------------+ | +------------------+ > + | FPGA PCIE Module | | Virtual | FPGA PCIE Module | > + +------------------+ Host | Machine +------------------+ > + ------------------------------------ | ------------------------------ > + +---------------+ | +---------------+ > + | PCI PF Device | | | PCI VF Device | > + +---------------+ | +---------------+ > + > +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe device > +driver is always loaded first once a FPGA PCIE PF or VF device is detected. This > +driver plays an infrastructural role in the driver architecuture. It: > + > + a) creates FPGA container device as parent of the feature devices. > + b) walks through the Device Feature List, which is implemented in PCIE > + device BAR memory, to discover feature devices and their sub features > + and create platform device for them under the container device. I really like the idea of creating platform devices for the sub features. It is in line with other FPGA use cases. Platform devices are at the heart of device trees used by processors directly connected FPGAs and processors inside FPGAs. > + c) supports SRIOV. > + d) introduces the feature device infrastructure, which abstracts > + operations for sub features and exposes common functions to feature > + device drivers. > + > +The FPGA Management Engine (FME) driver is a platform driver which is loaded > +automatically after FME platform device creation from the PCIE driver. It > +provides the key features for FPGA management, including: > + > + a) Power and thermal management, error reporting, performance reporting > + and other infrastructure functions. Users can access these functions > + via sysfs interfaces exposed by FME driver. > + b) Paritial Reconfiguration. The FME driver registers a FPGA Manager > + during PR sub feature initialization; once it receives an > + FPGA_FME_PORT_PR ioctl from user, it invokes the common interface > + function from FPGA Manager to complete the partial reconfiguration of > + the bitstream to the given port. > + c) Port management for virtualization. The FME driver introduces two > + ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and > + FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is > + released from the PF, it can be assigned to the VF through the SRIOV > + interfaces provided by PCIE driver. (Refer to "FPGA virtualization" > + for more details). > + > +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver > +is probed once the AFU platform device is created. The main function of this > +module is to provide an interface for userspace applications to access the > +individual accelerators, including basic reset control on port, AFU MMIO region > +export, dma buffer mapping service, UMsg notification, and remote debug > +functions (see above). > + > + > +Device enumeration > +================== > +This section introduces how applications enumerate the fpga device from > +the sysfs hierarchy under /sys/class/fpga. > + > +In the example below, two Intel(R) FPGA devices are installed in the host. Each > +fpga device has one FME and two ports (AFUs). > + > +For each FPGA device, a device director is created under /sys/class/fpga/: > + > + /sys/class/fpga/fpga.0 > + /sys/class/fpga/fpga.1 > + > +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's name. > +Application can retrieve name information via the sysfs interface: > + > + /sys/class/fpga/fpga.0/name > + > +Each node has one FME and two ports (AFUs) as child devices: > + > + /sys/class/fpga/fpga.0/intel-fpga-fme.0 > + /sys/class/fpga/fpga.0/intel-fpga-port.0 > + /sys/class/fpga/fpga.0/intel-fpga-port.1 > + > + /sys/class/fpga/fpga.1/intel-fpga-fme.1 > + /sys/class/fpga/fpga.1/intel-fpga-port.2 > + /sys/class/fpga/fpga.1/intel-fpga-port.3 > + > +In general, the FME/AFU sysfs interfaces are named as follows: > + > + /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/ > + /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/ > + > +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all > +ports. > + > +The device nodes used for ioctl() or mmap() can be referenced through: > + > + /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev > + /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev > + > + > +Open discussions > +================ > +The current FME driver does not provide user space access to the FME MMIO > +region, but exposes access through sysfs and ioctls. It also provides an FPGA > +manger interface for partial reconfiguration (PR), but does not make use of > +fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled inside > +the FME, and fpga-region depends on device tree which is not used at all. There > +are patches from Alan Tull to separate the device tree specific code and I am currently trying to use those patches in a different driver. They've compiled cleanly in my out of tree pcie module driver against the 3.10 kernel. I need to actually write the code to create and register the region, but Alan's platform driver code should be a good guide for me. Just need to find the time. > +introduce a sysfs interface for PR. We plan to add fpga-regions support in the > +driver once the related patches get merged. Then the FME driver should create > +one fpga-region for each Port/AFU. Does the FME driver create the fpga-region, or is each region described as an entry in the Device Feature List and therefore created by the code that enumerates the Device Feature List? > -- > 2.7.4 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-fpga" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html >
[toc] | [prev] | [next] | [standalone]
| From | Alan Tull <atull@kernel.org> |
|---|---|
| Date | 2017-03-31 20:40 +0200 |
| Subject | Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview |
| Message-ID | <tr9CO-8h2-9@gated-at.bofh.it> |
| In reply to | #1614241 |
On Fri, Mar 31, 2017 at 1:24 PM, <matthew.gerlach@linux.intel.com> wrote: > > > On Thu, 30 Mar 2017, Wu Hao wrote: > > > Hi Wu Hao, > > Great documentation. I'm looking forward to diving into the rest of the > patches. Please see my comments inline. > > Matthew Gerlach > > >> Add a document for Intel FPGA driver overview. >> >> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com> >> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com> >> Signed-off-by: Wu Hao <hao.wu@intel.com> >> --- >> Documentation/fpga/intel-fpga.txt | 259 >> ++++++++++++++++++++++++++++++++++++++ >> 1 file changed, 259 insertions(+) >> create mode 100644 Documentation/fpga/intel-fpga.txt >> >> diff --git a/Documentation/fpga/intel-fpga.txt >> b/Documentation/fpga/intel-fpga.txt >> new file mode 100644 >> index 0000000..9396cea >> --- /dev/null >> +++ b/Documentation/fpga/intel-fpga.txt >> @@ -0,0 +1,259 @@ >> >> +=============================================================================== >> + Intel FPGA driver Overview >> >> +------------------------------------------------------------------------------- >> + Enno Luebbers <enno.luebbers@intel.com> >> + Xiao Guangrong <guangrong.xiao@linux.intel.com> >> + Wu Hao <hao.wu@intel.com> >> + >> +The Intel FPGA driver provides interfaces for userspace applications to >> +configure, enumerate, open, and access FPGA accelerators on platforms >> equipped >> +with Intel(R) FPGA solutions and enables system level management >> functions such >> +as FPGA reconfiguration, power management, and virtualization. >> + > > > From a Linux kernel perspective, I'm not sure this is the best name for > this code. The name gives me the impression that it is a driver for all > Intel FPGAs, but not all Intel FPGAs are connected to the processor over a > PCIe bus. The processor could be directely connected like the Arria10 > SOCFPGA. Such a processor could certainly benefit from this accelerator > usage model. In an extreme case, couldn't a processor in the FPGA, > running Linux, also benefit from this accelerator model? Is this code a > "FPGA Accelerator Framework"? > >> +HW Architecture >> +=============== >> +From the OS's point of view, the FPGA hardware appears as a regular PCIe >> device. >> +The FPGA device memory is organized using a predefined data structure >> (Device >> +Feature List). Features supported by the particular FPGA device are >> exposed >> +through these data structures, as illustrated below: >> + >> + +-------------------------------+ +-------------+ >> + | PF | | VF | >> + +-------------------------------+ +-------------+ >> + ^ ^ ^ ^ >> + | | | | >> ++-----|------------|---------|--------------|-------+ >> +| | | | | | >> +| +-----+ +-------+ +-------+ +-------+ | >> +| | FME | | Port0 | | Port1 | | Port2 | | >> +| +-----+ +-------+ +-------+ +-------+ | >> +| ^ ^ ^ | >> +| | | | | >> +| +-------+ +------+ +-------+ | >> +| | AFU | | AFU | | AFU | | >> +| +-------+ +------+ +-------+ | >> +| | >> +| FPGA PCIe Device | >> ++---------------------------------------------------+ >> + >> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which >> can be >> +used to assign individual accelerators to virtual machines . > > > Does this HW Architecture require an Intel FPGA? Couldn't any vendors FPGA > be used as long as it presented itself the PCIe bus the same and contained > an appropriate Device Feature List? > >> + >> +FME (FPGA Management Engine) >> +============================ >> +The FPGA Management Enging performs power and thermal management, error >> +reporting, reconfiguration, performance reporting, and other >> infrastructure >> +functions. Each FPGA has one FME, which is always accessed through the >> physical >> +function (PF). >> + >> +User-space applications can acquire exclusive access to the FME using >> open(), >> +and release it using close(). >> + >> +The following functions are exposed through ioctls: >> + >> + Get driver API version (FPGA_GET_API_VERSION) >> + Check for extensions (FPGA_CHECK_EXTENSION) >> + Assign port to PF (FPGA_FME_PORT_ASSIGN) >> + Release port from PF (FPGA_FME_PORT_RELEASE) >> + Program bitstream (FPGA_FME_PORT_PR) >> + >> +More functions are exposed through sysfs >> +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/): >> + >> + Read bitstream ID (bitstream_id) >> + Read bitstream metadata (bitstream_metadata) >> + Read number of ports (ports_num) >> + Read socket ID (socket_id) >> + Read performance counters (perf/) >> + Power management (power_mgmt/) >> + Thermal management (thermal_mgmt/) >> + Error reporting (errors/) >> + >> +PORT >> +==== >> +A port represents the interface between the static FPGA fabric (the "blue >> +bitstream") and a partially reconfigurable region containing an AFU (the >> "green Is this an fpga bridge but with added features? >> +bitstream"). It controls the communication from SW to the accelerator and >> +exposes features such as reset and debug. >> + >> +A PCIe device may have several ports and each port can be released from >> PF by >> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe >> sriov >> +sysfs interface. >> + >> +AFU >> +=== >> +An AFU is attached to a port and exposes a 256k MMIO region to be used >> for >> +accelerator-specific control registers. >> + >> +User-space applications can acquire exclusive access to an AFU attached >> to a >> +port by using open() on the port device node, and release it using >> close(). >> + >> +The following functions are exposed through ioctls: >> + >> + Get driver API version (FPGA_GET_API_VERSION) >> + Check for extensions (FPGA_CHECK_EXTENSION) >> + Get port info (FPGA_PORT_GET_INFO) >> + Get MMIO region info (FPGA_PORT_GET_REGION_INFO) >> + Map DMA buffer (FPGA_PORT_DMA_MAP) >> + Unmap DMA buffer (FPGA_PORT_DMA_UNMAP) >> + Reset AFU (FPGA_PORT_RESET) >> + Enable UMsg (FPGA_PORT_UMSG_ENABLE) >> + Disable UMsg (FPGA_PORT_UMSG_DISABLE) >> + Set UMsg mode (FPGA_PORT_UMSG_SET_MODE) >> + Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR) >> + >> +User-space applications can also mmap() accelerator MMIO regions. >> + >> +More functions are exposed through sysfs: >> +(/sys/class/fpga/fpga.n/intel-fpga-port.m/): >> + >> + Read Accelerator GUID (afu_id) >> + Error reporting (errors/) >> + >> +Partial Reconfiguration >> +======================= >> +As mentioned above, accelerators can be reconfigured through partial >> +reconfiguration of a green bitstream file (GBS). The green bitstream must >> have >> +been generated for the exact blue bitstream and targeted reconfigurable >> region >> +(port) of the FPGA; otherwise, the reconfiguration operation will fail >> and >> +possibly cause system instability. This compatibility can be checked by >> +comparing the interface ID noted in the GBS header against the interface >> ID >> +exposed by the FME through sysfs (see above). This check is usually done >> by >> +user-space before calling the reconfiguration IOCTL. >> + >> +FPGA virtualization >> +=================== >> +To enable accessing an accelerator from applications running in a VM, the >> +respective AFU's port needs to be assigned to a VF using the following >> steps: >> + >> + a) The PF owns all AFU ports by default. Any port that needs to be >> reassigned >> + to a VF must be released from PF firstly through the >> FPGA_FME_PORT_RELEASE >> + ioctl on the FME device. >> + >> + b) Once N ports are released from PF, then user can use below command to >> + enable SRIOV and VFs. Each VF owns only one Port with AFU. >> + >> + echo N > $PCI_DEVICE_PATH/sriov_numvfs >> + >> + c) Pass through the VFs to VMs >> + >> + d) The AFU under VF is accessiable from applications in VM (using the >> same >> + driver inside the VF). >> + >> +Note the an FME can't be assigned to a VF, thus PR and other management >> +functions are only available via the PF. >> + >> + >> +Driver organization >> +=================== >> + >> + +------------------+ +---------+ | +---------+ >> + | +-------+ | | | | | | >> + | | FPGA | FME | | AFU | | | AFU | >> + | |Manager| Module | | Module | | | Module | >> + | +-------+ | | | | | | >> + +------------------+ +---------+ | +---------+ >> + +-----------------------+ | +-----------------------+ >> + | FPGA Container Device | | | FPGA Container Device | >> + +-----------------------+ | +-----------------------+ >> + +------------------+ | +------------------+ >> + | FPGA PCIE Module | | Virtual | FPGA PCIE Module | >> + +------------------+ Host | Machine +------------------+ >> + ------------------------------------ | ------------------------------ >> + +---------------+ | +---------------+ >> + | PCI PF Device | | | PCI VF Device | >> + +---------------+ | +---------------+ >> + >> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe >> device >> +driver is always loaded first once a FPGA PCIE PF or VF device is >> detected. This >> +driver plays an infrastructural role in the driver architecuture. It: >> + >> + a) creates FPGA container device as parent of the feature devices. >> + b) walks through the Device Feature List, which is implemented in >> PCIE >> + device BAR memory, to discover feature devices and their sub >> features >> + and create platform device for them under the container device. > > > I really like the idea of creating platform devices for the sub features. It > is in line with other FPGA use cases. Platform devices are at the heart of > device trees used by processors directly connected FPGAs and processors > inside FPGAs. > >> + c) supports SRIOV. >> + d) introduces the feature device infrastructure, which abstracts >> + operations for sub features and exposes common functions to >> feature >> + device drivers. >> + >> +The FPGA Management Engine (FME) driver is a platform driver which is >> loaded >> +automatically after FME platform device creation from the PCIE driver. It >> +provides the key features for FPGA management, including: >> + >> + a) Power and thermal management, error reporting, performance >> reporting >> + and other infrastructure functions. Users can access these >> functions >> + via sysfs interfaces exposed by FME driver. >> + b) Paritial Reconfiguration. The FME driver registers a FPGA >> Manager >> + during PR sub feature initialization; once it receives an >> + FPGA_FME_PORT_PR ioctl from user, it invokes the common >> interface >> + function from FPGA Manager to complete the partial >> reconfiguration of >> + the bitstream to the given port. >> + c) Port management for virtualization. The FME driver introduces >> two >> + ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and >> + FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the >> port is >> + released from the PF, it can be assigned to the VF through the >> SRIOV >> + interfaces provided by PCIE driver. (Refer to "FPGA >> virtualization" >> + for more details). >> + >> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) >> driver >> +is probed once the AFU platform device is created. The main function of >> this >> +module is to provide an interface for userspace applications to access >> the >> +individual accelerators, including basic reset control on port, AFU MMIO >> region >> +export, dma buffer mapping service, UMsg notification, and remote debug >> +functions (see above). >> + >> + >> +Device enumeration >> +================== >> +This section introduces how applications enumerate the fpga device from >> +the sysfs hierarchy under /sys/class/fpga. >> + >> +In the example below, two Intel(R) FPGA devices are installed in the >> host. Each >> +fpga device has one FME and two ports (AFUs). >> + >> +For each FPGA device, a device director is created under >> /sys/class/fpga/: >> + >> + /sys/class/fpga/fpga.0 >> + /sys/class/fpga/fpga.1 >> + >> +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's >> name. >> +Application can retrieve name information via the sysfs interface: >> + >> + /sys/class/fpga/fpga.0/name >> + >> +Each node has one FME and two ports (AFUs) as child devices: >> + >> + /sys/class/fpga/fpga.0/intel-fpga-fme.0 >> + /sys/class/fpga/fpga.0/intel-fpga-port.0 >> + /sys/class/fpga/fpga.0/intel-fpga-port.1 >> + >> + /sys/class/fpga/fpga.1/intel-fpga-fme.1 >> + /sys/class/fpga/fpga.1/intel-fpga-port.2 >> + /sys/class/fpga/fpga.1/intel-fpga-port.3 >> + >> +In general, the FME/AFU sysfs interfaces are named as follows: >> + >> + /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/ >> + /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/ >> + >> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering >> all >> +ports. >> + >> +The device nodes used for ioctl() or mmap() can be referenced through: >> + >> + /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev >> + /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev >> + >> + >> +Open discussions >> +================ >> +The current FME driver does not provide user space access to the FME MMIO >> +region, but exposes access through sysfs and ioctls. It also provides an >> FPGA >> +manger interface for partial reconfiguration (PR), but does not make use >> of >> +fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled >> inside >> +the FME, and fpga-region depends on device tree which is not used at all. >> There >> +are patches from Alan Tull to separate the device tree specific code and > > > I am currently trying to use those patches in a different driver. They've > compiled cleanly in my out of tree pcie module driver against the 3.10 > kernel. > I need to actually write the code to create and register the region, but > Alan's platform driver code should be a good guide for me. Just need to > find the time. > >> +introduce a sysfs interface for PR. We plan to add fpga-regions support >> in the >> +driver once the related patches get merged. Then the FME driver should >> create >> +one fpga-region for each Port/AFU. > > > Does the FME driver create the fpga-region, or is each region described as > an entry in the Device Feature List and therefore created by the code that > enumerates the Device Feature List? > >> -- >> 2.7.4 >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >
[toc] | [prev] | [next] | [standalone]
| From | Wu Hao <hao.wu@intel.com> |
|---|---|
| Date | 2017-04-01 13:30 +0200 |
| Subject | Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview |
| Message-ID | <trpod-1Uj-13@gated-at.bofh.it> |
| In reply to | #1614242 |
On Fri, Mar 31, 2017 at 01:38:06PM -0500, Alan Tull wrote: > On Fri, Mar 31, 2017 at 1:24 PM, <matthew.gerlach@linux.intel.com> wrote: > > > > > > On Thu, 30 Mar 2017, Wu Hao wrote: > > > > > > Hi Wu Hao, > > > > Great documentation. I'm looking forward to diving into the rest of the > > patches. Please see my comments inline. > > > > Matthew Gerlach > > > > > >> Add a document for Intel FPGA driver overview. > >> > >> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com> > >> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com> > >> Signed-off-by: Wu Hao <hao.wu@intel.com> > >> --- > >> Documentation/fpga/intel-fpga.txt | 259 > >> ++++++++++++++++++++++++++++++++++++++ > >> 1 file changed, 259 insertions(+) > >> create mode 100644 Documentation/fpga/intel-fpga.txt > >> > >> diff --git a/Documentation/fpga/intel-fpga.txt > >> b/Documentation/fpga/intel-fpga.txt > >> new file mode 100644 > >> index 0000000..9396cea > >> --- /dev/null > >> +++ b/Documentation/fpga/intel-fpga.txt > >> @@ -0,0 +1,259 @@ > >> > >> +=============================================================================== > >> + Intel FPGA driver Overview > >> > >> +------------------------------------------------------------------------------- > >> + Enno Luebbers <enno.luebbers@intel.com> > >> + Xiao Guangrong <guangrong.xiao@linux.intel.com> > >> + Wu Hao <hao.wu@intel.com> > >> + > >> +The Intel FPGA driver provides interfaces for userspace applications to > >> +configure, enumerate, open, and access FPGA accelerators on platforms > >> equipped > >> +with Intel(R) FPGA solutions and enables system level management > >> functions such > >> +as FPGA reconfiguration, power management, and virtualization. > >> + > > > > > > From a Linux kernel perspective, I'm not sure this is the best name for > > this code. The name gives me the impression that it is a driver for all > > Intel FPGAs, but not all Intel FPGAs are connected to the processor over a > > PCIe bus. The processor could be directely connected like the Arria10 > > SOCFPGA. Such a processor could certainly benefit from this accelerator > > usage model. In an extreme case, couldn't a processor in the FPGA, > > running Linux, also benefit from this accelerator model? Is this code a > > "FPGA Accelerator Framework"? > > > >> +HW Architecture > >> +=============== > >> +From the OS's point of view, the FPGA hardware appears as a regular PCIe > >> device. > >> +The FPGA device memory is organized using a predefined data structure > >> (Device > >> +Feature List). Features supported by the particular FPGA device are > >> exposed > >> +through these data structures, as illustrated below: > >> + > >> + +-------------------------------+ +-------------+ > >> + | PF | | VF | > >> + +-------------------------------+ +-------------+ > >> + ^ ^ ^ ^ > >> + | | | | > >> ++-----|------------|---------|--------------|-------+ > >> +| | | | | | > >> +| +-----+ +-------+ +-------+ +-------+ | > >> +| | FME | | Port0 | | Port1 | | Port2 | | > >> +| +-----+ +-------+ +-------+ +-------+ | > >> +| ^ ^ ^ | > >> +| | | | | > >> +| +-------+ +------+ +-------+ | > >> +| | AFU | | AFU | | AFU | | > >> +| +-------+ +------+ +-------+ | > >> +| | > >> +| FPGA PCIe Device | > >> ++---------------------------------------------------+ > >> + > >> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which > >> can be > >> +used to assign individual accelerators to virtual machines . > > > > > > Does this HW Architecture require an Intel FPGA? Couldn't any vendors FPGA > > be used as long as it presented itself the PCIe bus the same and contained > > an appropriate Device Feature List? > > > >> + > >> +FME (FPGA Management Engine) > >> +============================ > >> +The FPGA Management Enging performs power and thermal management, error > >> +reporting, reconfiguration, performance reporting, and other > >> infrastructure > >> +functions. Each FPGA has one FME, which is always accessed through the > >> physical > >> +function (PF). > >> + > >> +User-space applications can acquire exclusive access to the FME using > >> open(), > >> +and release it using close(). > >> + > >> +The following functions are exposed through ioctls: > >> + > >> + Get driver API version (FPGA_GET_API_VERSION) > >> + Check for extensions (FPGA_CHECK_EXTENSION) > >> + Assign port to PF (FPGA_FME_PORT_ASSIGN) > >> + Release port from PF (FPGA_FME_PORT_RELEASE) > >> + Program bitstream (FPGA_FME_PORT_PR) > >> + > >> +More functions are exposed through sysfs > >> +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/): > >> + > >> + Read bitstream ID (bitstream_id) > >> + Read bitstream metadata (bitstream_metadata) > >> + Read number of ports (ports_num) > >> + Read socket ID (socket_id) > >> + Read performance counters (perf/) > >> + Power management (power_mgmt/) > >> + Thermal management (thermal_mgmt/) > >> + Error reporting (errors/) > >> + > >> +PORT > >> +==== > >> +A port represents the interface between the static FPGA fabric (the "blue > >> +bitstream") and a partially reconfigurable region containing an AFU (the > >> "green > > Is this an fpga bridge but with added features? Yes, I think so. As you see the fme_pr function in patch 11, related port needs to be disabled firstly before fpga_mgr_buf_load for given accelerator. Thanks Hao > > >> +bitstream"). It controls the communication from SW to the accelerator and > >> +exposes features such as reset and debug. > >> + > >> +A PCIe device may have several ports and each port can be released from > >> PF by > >> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe > >> sriov > >> +sysfs interface. > >> + > >> +AFU > >> +=== > >> +An AFU is attached to a port and exposes a 256k MMIO region to be used > >> for > >> +accelerator-specific control registers. > >> + > >> +User-space applications can acquire exclusive access to an AFU attached > >> to a > >> +port by using open() on the port device node, and release it using > >> close(). > >> + > >> +The following functions are exposed through ioctls: > >> + > >> + Get driver API version (FPGA_GET_API_VERSION) > >> + Check for extensions (FPGA_CHECK_EXTENSION) > >> + Get port info (FPGA_PORT_GET_INFO) > >> + Get MMIO region info (FPGA_PORT_GET_REGION_INFO) > >> + Map DMA buffer (FPGA_PORT_DMA_MAP) > >> + Unmap DMA buffer (FPGA_PORT_DMA_UNMAP) > >> + Reset AFU (FPGA_PORT_RESET) > >> + Enable UMsg (FPGA_PORT_UMSG_ENABLE) > >> + Disable UMsg (FPGA_PORT_UMSG_DISABLE) > >> + Set UMsg mode (FPGA_PORT_UMSG_SET_MODE) > >> + Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR) > >> + > >> +User-space applications can also mmap() accelerator MMIO regions. > >> + > >> +More functions are exposed through sysfs: > >> +(/sys/class/fpga/fpga.n/intel-fpga-port.m/): > >> + > >> + Read Accelerator GUID (afu_id) > >> + Error reporting (errors/) > >> + > >> +Partial Reconfiguration > >> +======================= > >> +As mentioned above, accelerators can be reconfigured through partial > >> +reconfiguration of a green bitstream file (GBS). The green bitstream must > >> have > >> +been generated for the exact blue bitstream and targeted reconfigurable > >> region > >> +(port) of the FPGA; otherwise, the reconfiguration operation will fail > >> and > >> +possibly cause system instability. This compatibility can be checked by > >> +comparing the interface ID noted in the GBS header against the interface > >> ID > >> +exposed by the FME through sysfs (see above). This check is usually done > >> by > >> +user-space before calling the reconfiguration IOCTL. > >> + > >> +FPGA virtualization > >> +=================== > >> +To enable accessing an accelerator from applications running in a VM, the > >> +respective AFU's port needs to be assigned to a VF using the following > >> steps: > >> + > >> + a) The PF owns all AFU ports by default. Any port that needs to be > >> reassigned > >> + to a VF must be released from PF firstly through the > >> FPGA_FME_PORT_RELEASE > >> + ioctl on the FME device. > >> + > >> + b) Once N ports are released from PF, then user can use below command to > >> + enable SRIOV and VFs. Each VF owns only one Port with AFU. > >> + > >> + echo N > $PCI_DEVICE_PATH/sriov_numvfs > >> + > >> + c) Pass through the VFs to VMs > >> + > >> + d) The AFU under VF is accessiable from applications in VM (using the > >> same > >> + driver inside the VF). > >> + > >> +Note the an FME can't be assigned to a VF, thus PR and other management > >> +functions are only available via the PF. > >> + > >> + > >> +Driver organization > >> +=================== > >> + > >> + +------------------+ +---------+ | +---------+ > >> + | +-------+ | | | | | | > >> + | | FPGA | FME | | AFU | | | AFU | > >> + | |Manager| Module | | Module | | | Module | > >> + | +-------+ | | | | | | > >> + +------------------+ +---------+ | +---------+ > >> + +-----------------------+ | +-----------------------+ > >> + | FPGA Container Device | | | FPGA Container Device | > >> + +-----------------------+ | +-----------------------+ > >> + +------------------+ | +------------------+ > >> + | FPGA PCIE Module | | Virtual | FPGA PCIE Module | > >> + +------------------+ Host | Machine +------------------+ > >> + ------------------------------------ | ------------------------------ > >> + +---------------+ | +---------------+ > >> + | PCI PF Device | | | PCI VF Device | > >> + +---------------+ | +---------------+ > >> + > >> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe > >> device > >> +driver is always loaded first once a FPGA PCIE PF or VF device is > >> detected. This > >> +driver plays an infrastructural role in the driver architecuture. It: > >> + > >> + a) creates FPGA container device as parent of the feature devices. > >> + b) walks through the Device Feature List, which is implemented in > >> PCIE > >> + device BAR memory, to discover feature devices and their sub > >> features > >> + and create platform device for them under the container device. > > > > > > I really like the idea of creating platform devices for the sub features. It > > is in line with other FPGA use cases. Platform devices are at the heart of > > device trees used by processors directly connected FPGAs and processors > > inside FPGAs. > > > >> + c) supports SRIOV. > >> + d) introduces the feature device infrastructure, which abstracts > >> + operations for sub features and exposes common functions to > >> feature > >> + device drivers. > >> + > >> +The FPGA Management Engine (FME) driver is a platform driver which is > >> loaded > >> +automatically after FME platform device creation from the PCIE driver. It > >> +provides the key features for FPGA management, including: > >> + > >> + a) Power and thermal management, error reporting, performance > >> reporting > >> + and other infrastructure functions. Users can access these > >> functions > >> + via sysfs interfaces exposed by FME driver. > >> + b) Paritial Reconfiguration. The FME driver registers a FPGA > >> Manager > >> + during PR sub feature initialization; once it receives an > >> + FPGA_FME_PORT_PR ioctl from user, it invokes the common > >> interface > >> + function from FPGA Manager to complete the partial > >> reconfiguration of > >> + the bitstream to the given port. > >> + c) Port management for virtualization. The FME driver introduces > >> two > >> + ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and > >> + FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the > >> port is > >> + released from the PF, it can be assigned to the VF through the > >> SRIOV > >> + interfaces provided by PCIE driver. (Refer to "FPGA > >> virtualization" > >> + for more details). > >> + > >> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) > >> driver > >> +is probed once the AFU platform device is created. The main function of > >> this > >> +module is to provide an interface for userspace applications to access > >> the > >> +individual accelerators, including basic reset control on port, AFU MMIO > >> region > >> +export, dma buffer mapping service, UMsg notification, and remote debug > >> +functions (see above). > >> + > >> + > >> +Device enumeration > >> +================== > >> +This section introduces how applications enumerate the fpga device from > >> +the sysfs hierarchy under /sys/class/fpga. > >> + > >> +In the example below, two Intel(R) FPGA devices are installed in the > >> host. Each > >> +fpga device has one FME and two ports (AFUs). > >> + > >> +For each FPGA device, a device director is created under > >> /sys/class/fpga/: > >> + > >> + /sys/class/fpga/fpga.0 > >> + /sys/class/fpga/fpga.1 > >> + > >> +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's > >> name. > >> +Application can retrieve name information via the sysfs interface: > >> + > >> + /sys/class/fpga/fpga.0/name > >> + > >> +Each node has one FME and two ports (AFUs) as child devices: > >> + > >> + /sys/class/fpga/fpga.0/intel-fpga-fme.0 > >> + /sys/class/fpga/fpga.0/intel-fpga-port.0 > >> + /sys/class/fpga/fpga.0/intel-fpga-port.1 > >> + > >> + /sys/class/fpga/fpga.1/intel-fpga-fme.1 > >> + /sys/class/fpga/fpga.1/intel-fpga-port.2 > >> + /sys/class/fpga/fpga.1/intel-fpga-port.3 > >> + > >> +In general, the FME/AFU sysfs interfaces are named as follows: > >> + > >> + /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/ > >> + /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/ > >> + > >> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering > >> all > >> +ports. > >> + > >> +The device nodes used for ioctl() or mmap() can be referenced through: > >> + > >> + /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev > >> + /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev > >> + > >> + > >> +Open discussions > >> +================ > >> +The current FME driver does not provide user space access to the FME MMIO > >> +region, but exposes access through sysfs and ioctls. It also provides an > >> FPGA > >> +manger interface for partial reconfiguration (PR), but does not make use > >> of > >> +fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled > >> inside > >> +the FME, and fpga-region depends on device tree which is not used at all. > >> There > >> +are patches from Alan Tull to separate the device tree specific code and > > > > > > I am currently trying to use those patches in a different driver. They've > > compiled cleanly in my out of tree pcie module driver against the 3.10 > > kernel. > > I need to actually write the code to create and register the region, but > > Alan's platform driver code should be a good guide for me. Just need to > > find the time. > > > >> +introduce a sysfs interface for PR. We plan to add fpga-regions support > >> in the > >> +driver once the related patches get merged. Then the FME driver should > >> create > >> +one fpga-region for each Port/AFU. > > > > > > Does the FME driver create the fpga-region, or is each region described as > > an entry in the Device Feature List and therefore created by the code that > > enumerates the Device Feature List? > > > >> -- > >> 2.7.4 > >> > >> -- > >> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in > >> the body of a message to majordomo@vger.kernel.org > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > >> > >
[toc] | [prev] | [next] | [standalone]
| From | Moritz Fischer <mdf@kernel.org> |
|---|---|
| Date | 2017-04-02 16:50 +0200 |
| Subject | Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview |
| Message-ID | <trOZj-1KM-9@gated-at.bofh.it> |
| In reply to | #1614480 |
On Sat, Apr 01, 2017 at 07:16:19PM +0800, Wu Hao wrote: > On Fri, Mar 31, 2017 at 01:38:06PM -0500, Alan Tull wrote: > > On Fri, Mar 31, 2017 at 1:24 PM, <matthew.gerlach@linux.intel.com> wrote: > > > > > > > > > On Thu, 30 Mar 2017, Wu Hao wrote: > > > > > > > > > Hi Wu Hao, > > > > > > Great documentation. I'm looking forward to diving into the rest of the > > > patches. Please see my comments inline. > > > > > > Matthew Gerlach > > > > > > > > >> Add a document for Intel FPGA driver overview. > > >> > > >> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com> > > >> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com> > > >> Signed-off-by: Wu Hao <hao.wu@intel.com> > > >> --- > > >> Documentation/fpga/intel-fpga.txt | 259 > > >> ++++++++++++++++++++++++++++++++++++++ > > >> 1 file changed, 259 insertions(+) > > >> create mode 100644 Documentation/fpga/intel-fpga.txt > > >> > > >> diff --git a/Documentation/fpga/intel-fpga.txt > > >> b/Documentation/fpga/intel-fpga.txt > > >> new file mode 100644 > > >> index 0000000..9396cea > > >> --- /dev/null > > >> +++ b/Documentation/fpga/intel-fpga.txt > > >> @@ -0,0 +1,259 @@ > > >> > > >> +=============================================================================== > > >> + Intel FPGA driver Overview > > >> > > >> +------------------------------------------------------------------------------- > > >> + Enno Luebbers <enno.luebbers@intel.com> > > >> + Xiao Guangrong <guangrong.xiao@linux.intel.com> > > >> + Wu Hao <hao.wu@intel.com> > > >> + > > >> +The Intel FPGA driver provides interfaces for userspace applications to > > >> +configure, enumerate, open, and access FPGA accelerators on platforms > > >> equipped > > >> +with Intel(R) FPGA solutions and enables system level management > > >> functions such > > >> +as FPGA reconfiguration, power management, and virtualization. > > >> + > > > > > > > > > From a Linux kernel perspective, I'm not sure this is the best name for > > > this code. The name gives me the impression that it is a driver for all > > > Intel FPGAs, but not all Intel FPGAs are connected to the processor over a > > > PCIe bus. The processor could be directely connected like the Arria10 > > > SOCFPGA. Such a processor could certainly benefit from this accelerator > > > usage model. In an extreme case, couldn't a processor in the FPGA, > > > running Linux, also benefit from this accelerator model? Is this code a > > > "FPGA Accelerator Framework"? > > > > > >> +HW Architecture > > >> +=============== > > >> +From the OS's point of view, the FPGA hardware appears as a regular PCIe > > >> device. > > >> +The FPGA device memory is organized using a predefined data structure > > >> (Device > > >> +Feature List). Features supported by the particular FPGA device are > > >> exposed > > >> +through these data structures, as illustrated below: > > >> + > > >> + +-------------------------------+ +-------------+ > > >> + | PF | | VF | > > >> + +-------------------------------+ +-------------+ > > >> + ^ ^ ^ ^ > > >> + | | | | > > >> ++-----|------------|---------|--------------|-------+ > > >> +| | | | | | > > >> +| +-----+ +-------+ +-------+ +-------+ | > > >> +| | FME | | Port0 | | Port1 | | Port2 | | > > >> +| +-----+ +-------+ +-------+ +-------+ | > > >> +| ^ ^ ^ | > > >> +| | | | | > > >> +| +-------+ +------+ +-------+ | > > >> +| | AFU | | AFU | | AFU | | > > >> +| +-------+ +------+ +-------+ | > > >> +| | > > >> +| FPGA PCIe Device | > > >> ++---------------------------------------------------+ > > >> + > > >> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which > > >> can be > > >> +used to assign individual accelerators to virtual machines . > > > > > > > > > Does this HW Architecture require an Intel FPGA? Couldn't any vendors FPGA > > > be used as long as it presented itself the PCIe bus the same and contained > > > an appropriate Device Feature List? I think this is a good (and important) point. Especially when sysfs entries & ioctls constituting ABI depend on it. > > > > > >> + > > >> +FME (FPGA Management Engine) > > >> +============================ > > >> +The FPGA Management Enging performs power and thermal management, error Enging->Engine > > >> +reporting, reconfiguration, performance reporting, and other > > >> infrastructure > > >> +functions. Each FPGA has one FME, which is always accessed through the > > >> physical > > >> +function (PF). > > >> + > > >> +User-space applications can acquire exclusive access to the FME using > > >> open(), > > >> +and release it using close(). > > >> + > > >> +The following functions are exposed through ioctls: > > >> + > > >> + Get driver API version (FPGA_GET_API_VERSION) > > >> + Check for extensions (FPGA_CHECK_EXTENSION) > > >> + Assign port to PF (FPGA_FME_PORT_ASSIGN) > > >> + Release port from PF (FPGA_FME_PORT_RELEASE) > > >> + Program bitstream (FPGA_FME_PORT_PR) > > >> + > > >> +More functions are exposed through sysfs > > >> +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/): > > >> + > > >> + Read bitstream ID (bitstream_id) > > >> + Read bitstream metadata (bitstream_metadata) > > >> + Read number of ports (ports_num) > > >> + Read socket ID (socket_id) > > >> + Read performance counters (perf/) > > >> + Power management (power_mgmt/) > > >> + Thermal management (thermal_mgmt/) > > >> + Error reporting (errors/) > > >> + > > >> +PORT > > >> +==== > > >> +A port represents the interface between the static FPGA fabric (the "blue > > >> +bitstream") and a partially reconfigurable region containing an AFU (the > > >> "green > > > > Is this an fpga bridge but with added features? > > Yes, I think so. As you see the fme_pr function in patch 11, related port needs > to be disabled firstly before fpga_mgr_buf_load for given accelerator. Can we just extend the bridge to have the additional features, please? > > >> +bitstream"). It controls the communication from SW to the accelerator and > > >> +exposes features such as reset and debug. > > >> + > > >> +A PCIe device may have several ports and each port can be released from > > >> PF by > > >> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe > > >> sriov > > >> +sysfs interface. > > >> + > > >> +AFU > > >> +=== > > >> +An AFU is attached to a port and exposes a 256k MMIO region to be used > > >> for > > >> +accelerator-specific control registers. > > >> + > > >> +User-space applications can acquire exclusive access to an AFU attached > > >> to a > > >> +port by using open() on the port device node, and release it using > > >> close(). > > >> + > > >> +The following functions are exposed through ioctls: > > >> + > > >> + Get driver API version (FPGA_GET_API_VERSION) > > >> + Check for extensions (FPGA_CHECK_EXTENSION) > > >> + Get port info (FPGA_PORT_GET_INFO) > > >> + Get MMIO region info (FPGA_PORT_GET_REGION_INFO) > > >> + Map DMA buffer (FPGA_PORT_DMA_MAP) > > >> + Unmap DMA buffer (FPGA_PORT_DMA_UNMAP) > > >> + Reset AFU (FPGA_PORT_RESET) > > >> + Enable UMsg (FPGA_PORT_UMSG_ENABLE) > > >> + Disable UMsg (FPGA_PORT_UMSG_DISABLE) > > >> + Set UMsg mode (FPGA_PORT_UMSG_SET_MODE) > > >> + Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR) > > >> + > > >> +User-space applications can also mmap() accelerator MMIO regions. > > >> + > > >> +More functions are exposed through sysfs: > > >> +(/sys/class/fpga/fpga.n/intel-fpga-port.m/): > > >> + > > >> + Read Accelerator GUID (afu_id) > > >> + Error reporting (errors/) > > >> + > > >> +Partial Reconfiguration > > >> +======================= > > >> +As mentioned above, accelerators can be reconfigured through partial > > >> +reconfiguration of a green bitstream file (GBS). The green bitstream must > > >> have > > >> +been generated for the exact blue bitstream and targeted reconfigurable > > >> region > > >> +(port) of the FPGA; otherwise, the reconfiguration operation will fail > > >> and > > >> +possibly cause system instability. This compatibility can be checked by > > >> +comparing the interface ID noted in the GBS header against the interface > > >> ID > > >> +exposed by the FME through sysfs (see above). This check is usually done > > >> by > > >> +user-space before calling the reconfiguration IOCTL. > > >> + > > >> +FPGA virtualization > > >> +=================== > > >> +To enable accessing an accelerator from applications running in a VM, the > > >> +respective AFU's port needs to be assigned to a VF using the following > > >> steps: > > >> + > > >> + a) The PF owns all AFU ports by default. Any port that needs to be > > >> reassigned > > >> + to a VF must be released from PF firstly through the > > >> FPGA_FME_PORT_RELEASE > > >> + ioctl on the FME device. > > >> + > > >> + b) Once N ports are released from PF, then user can use below command to > > >> + enable SRIOV and VFs. Each VF owns only one Port with AFU. > > >> + > > >> + echo N > $PCI_DEVICE_PATH/sriov_numvfs > > >> + > > >> + c) Pass through the VFs to VMs > > >> + > > >> + d) The AFU under VF is accessiable from applications in VM (using the > > >> same > > >> + driver inside the VF). > > >> + > > >> +Note the an FME can't be assigned to a VF, thus PR and other management > > >> +functions are only available via the PF. > > >> + > > >> + > > >> +Driver organization > > >> +=================== > > >> + > > >> + +------------------+ +---------+ | +---------+ > > >> + | +-------+ | | | | | | > > >> + | | FPGA | FME | | AFU | | | AFU | > > >> + | |Manager| Module | | Module | | | Module | > > >> + | +-------+ | | | | | | > > >> + +------------------+ +---------+ | +---------+ > > >> + +-----------------------+ | +-----------------------+ > > >> + | FPGA Container Device | | | FPGA Container Device | > > >> + +-----------------------+ | +-----------------------+ > > >> + +------------------+ | +------------------+ > > >> + | FPGA PCIE Module | | Virtual | FPGA PCIE Module | > > >> + +------------------+ Host | Machine +------------------+ > > >> + ------------------------------------ | ------------------------------ > > >> + +---------------+ | +---------------+ > > >> + | PCI PF Device | | | PCI VF Device | > > >> + +---------------+ | +---------------+ > > >> + > > >> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe > > >> device > > >> +driver is always loaded first once a FPGA PCIE PF or VF device is > > >> detected. This > > >> +driver plays an infrastructural role in the driver architecuture. It: > > >> + > > >> + a) creates FPGA container device as parent of the feature devices. > > >> + b) walks through the Device Feature List, which is implemented in > > >> PCIE > > >> + device BAR memory, to discover feature devices and their sub > > >> features > > >> + and create platform device for them under the container device. > > > > > > > > > I really like the idea of creating platform devices for the sub features. It > > > is in line with other FPGA use cases. Platform devices are at the heart of > > > device trees used by processors directly connected FPGAs and processors > > > inside FPGAs. > > > > > >> + c) supports SRIOV. > > >> + d) introduces the feature device infrastructure, which abstracts > > >> + operations for sub features and exposes common functions to > > >> feature > > >> + device drivers. > > >> + > > >> +The FPGA Management Engine (FME) driver is a platform driver which is > > >> loaded > > >> +automatically after FME platform device creation from the PCIE driver. It > > >> +provides the key features for FPGA management, including: > > >> + > > >> + a) Power and thermal management, error reporting, performance > > >> reporting > > >> + and other infrastructure functions. Users can access these > > >> functions > > >> + via sysfs interfaces exposed by FME driver. > > >> + b) Paritial Reconfiguration. The FME driver registers a FPGA > > >> Manager > > >> + during PR sub feature initialization; once it receives an > > >> + FPGA_FME_PORT_PR ioctl from user, it invokes the common > > >> interface > > >> + function from FPGA Manager to complete the partial > > >> reconfiguration of > > >> + the bitstream to the given port. > > >> + c) Port management for virtualization. The FME driver introduces > > >> two > > >> + ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and > > >> + FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the > > >> port is > > >> + released from the PF, it can be assigned to the VF through the > > >> SRIOV > > >> + interfaces provided by PCIE driver. (Refer to "FPGA > > >> virtualization" > > >> + for more details). > > >> + > > >> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) > > >> driver > > >> +is probed once the AFU platform device is created. The main function of > > >> this > > >> +module is to provide an interface for userspace applications to access > > >> the > > >> +individual accelerators, including basic reset control on port, AFU MMIO > > >> region > > >> +export, dma buffer mapping service, UMsg notification, and remote debug > > >> +functions (see above). > > >> + > > >> + > > >> +Device enumeration > > >> +================== > > >> +This section introduces how applications enumerate the fpga device from > > >> +the sysfs hierarchy under /sys/class/fpga. > > >> + > > >> +In the example below, two Intel(R) FPGA devices are installed in the > > >> host. Each > > >> +fpga device has one FME and two ports (AFUs). > > >> + > > >> +For each FPGA device, a device director is created under > > >> /sys/class/fpga/: > > >> + > > >> + /sys/class/fpga/fpga.0 > > >> + /sys/class/fpga/fpga.1 > > >> + > > >> +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's > > >> name. > > >> +Application can retrieve name information via the sysfs interface: > > >> + > > >> + /sys/class/fpga/fpga.0/name > > >> + > > >> +Each node has one FME and two ports (AFUs) as child devices: > > >> + > > >> + /sys/class/fpga/fpga.0/intel-fpga-fme.0 > > >> + /sys/class/fpga/fpga.0/intel-fpga-port.0 > > >> + /sys/class/fpga/fpga.0/intel-fpga-port.1 > > >> + > > >> + /sys/class/fpga/fpga.1/intel-fpga-fme.1 > > >> + /sys/class/fpga/fpga.1/intel-fpga-port.2 > > >> + /sys/class/fpga/fpga.1/intel-fpga-port.3 > > >> + > > >> +In general, the FME/AFU sysfs interfaces are named as follows: > > >> + > > >> + /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/ > > >> + /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/ > > >> + > > >> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering > > >> all > > >> +ports. > > >> + > > >> +The device nodes used for ioctl() or mmap() can be referenced through: > > >> + > > >> + /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev > > >> + /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev > > >> + > > >> + > > >> +Open discussions > > >> +================ > > >> +The current FME driver does not provide user space access to the FME MMIO > > >> +region, but exposes access through sysfs and ioctls. It also provides an > > >> FPGA > > >> +manger interface for partial reconfiguration (PR), but does not make use > > >> of > > >> +fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled > > >> inside > > >> +the FME, and fpga-region depends on device tree which is not used at all. > > >> There > > >> +are patches from Alan Tull to separate the device tree specific code and > > > > > > > > > I am currently trying to use those patches in a different driver. They've > > > compiled cleanly in my out of tree pcie module driver against the 3.10 > > > kernel. > > > I need to actually write the code to create and register the region, but > > > Alan's platform driver code should be a good guide for me. Just need to > > > find the time. > > > > > >> +introduce a sysfs interface for PR. We plan to add fpga-regions support > > >> in the > > >> +driver once the related patches get merged. Then the FME driver should > > >> create > > >> +one fpga-region for each Port/AFU. > > > > > > > > > Does the FME driver create the fpga-region, or is each region described as > > > an entry in the Device Feature List and therefore created by the code that > > > enumerates the Device Feature List? > > > > > >> -- > > >> 2.7.4 > > >> > > >> -- > > >> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in > > >> the body of a message to majordomo@vger.kernel.org > > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > >> > > > Cheers, Moritz
[toc] | [prev] | [next] | [standalone]
| From | Alan Tull <atull@kernel.org> |
|---|---|
| Date | 2017-04-03 22:50 +0200 |
| Subject | Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview |
| Message-ID | <tsh5f-3nO-1@gated-at.bofh.it> |
| In reply to | #1614753 |
On Sun, Apr 2, 2017 at 9:41 AM, Moritz Fischer <mdf@kernel.org> wrote: > On Sat, Apr 01, 2017 at 07:16:19PM +0800, Wu Hao wrote: >> On Fri, Mar 31, 2017 at 01:38:06PM -0500, Alan Tull wrote: >> > On Fri, Mar 31, 2017 at 1:24 PM, <matthew.gerlach@linux.intel.com> wrote: >> > > >> > > >> > > On Thu, 30 Mar 2017, Wu Hao wrote: >> > > >> > > >> > > Hi Wu Hao, >> > > >> > > Great documentation. I'm looking forward to diving into the rest of the >> > > patches. Please see my comments inline. >> > > >> > > Matthew Gerlach >> > > >> > > >> > >> Add a document for Intel FPGA driver overview. >> > >> >> > >> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com> >> > >> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com> >> > >> Signed-off-by: Wu Hao <hao.wu@intel.com> >> > >> --- >> > >> Documentation/fpga/intel-fpga.txt | 259 >> > >> ++++++++++++++++++++++++++++++++++++++ >> > >> 1 file changed, 259 insertions(+) >> > >> create mode 100644 Documentation/fpga/intel-fpga.txt >> > >> >> > >> diff --git a/Documentation/fpga/intel-fpga.txt >> > >> b/Documentation/fpga/intel-fpga.txt >> > >> new file mode 100644 >> > >> index 0000000..9396cea >> > >> --- /dev/null >> > >> +++ b/Documentation/fpga/intel-fpga.txt >> > >> @@ -0,0 +1,259 @@ >> > >> >> > >> +=============================================================================== >> > >> + Intel FPGA driver Overview >> > >> >> > >> +------------------------------------------------------------------------------- >> > >> + Enno Luebbers <enno.luebbers@intel.com> >> > >> + Xiao Guangrong <guangrong.xiao@linux.intel.com> >> > >> + Wu Hao <hao.wu@intel.com> >> > >> + >> > >> +The Intel FPGA driver provides interfaces for userspace applications to >> > >> +configure, enumerate, open, and access FPGA accelerators on platforms >> > >> equipped >> > >> +with Intel(R) FPGA solutions and enables system level management >> > >> functions such >> > >> +as FPGA reconfiguration, power management, and virtualization. >> > >> + >> > > >> > > >> > > From a Linux kernel perspective, I'm not sure this is the best name for >> > > this code. The name gives me the impression that it is a driver for all >> > > Intel FPGAs, but not all Intel FPGAs are connected to the processor over a >> > > PCIe bus. The processor could be directely connected like the Arria10 >> > > SOCFPGA. Such a processor could certainly benefit from this accelerator >> > > usage model. In an extreme case, couldn't a processor in the FPGA, >> > > running Linux, also benefit from this accelerator model? Is this code a >> > > "FPGA Accelerator Framework"? >> > > >> > >> +HW Architecture >> > >> +=============== >> > >> +From the OS's point of view, the FPGA hardware appears as a regular PCIe >> > >> device. >> > >> +The FPGA device memory is organized using a predefined data structure >> > >> (Device >> > >> +Feature List). Features supported by the particular FPGA device are >> > >> exposed >> > >> +through these data structures, as illustrated below: >> > >> + >> > >> + +-------------------------------+ +-------------+ >> > >> + | PF | | VF | >> > >> + +-------------------------------+ +-------------+ >> > >> + ^ ^ ^ ^ >> > >> + | | | | >> > >> ++-----|------------|---------|--------------|-------+ >> > >> +| | | | | | >> > >> +| +-----+ +-------+ +-------+ +-------+ | >> > >> +| | FME | | Port0 | | Port1 | | Port2 | | >> > >> +| +-----+ +-------+ +-------+ +-------+ | >> > >> +| ^ ^ ^ | >> > >> +| | | | | >> > >> +| +-------+ +------+ +-------+ | >> > >> +| | AFU | | AFU | | AFU | | >> > >> +| +-------+ +------+ +-------+ | >> > >> +| | >> > >> +| FPGA PCIe Device | >> > >> ++---------------------------------------------------+ >> > >> + >> > >> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which >> > >> can be >> > >> +used to assign individual accelerators to virtual machines . >> > > >> > > >> > > Does this HW Architecture require an Intel FPGA? Couldn't any vendors FPGA >> > > be used as long as it presented itself the PCIe bus the same and contained >> > > an appropriate Device Feature List? > > I think this is a good (and important) point. Especially when sysfs > entries & ioctls constituting ABI depend on it. > >> > > >> > >> + >> > >> +FME (FPGA Management Engine) >> > >> +============================ >> > >> +The FPGA Management Enging performs power and thermal management, error > Enging->Engine >> > >> +reporting, reconfiguration, performance reporting, and other >> > >> infrastructure >> > >> +functions. Each FPGA has one FME, which is always accessed through the >> > >> physical >> > >> +function (PF). >> > >> + >> > >> +User-space applications can acquire exclusive access to the FME using >> > >> open(), >> > >> +and release it using close(). >> > >> + >> > >> +The following functions are exposed through ioctls: >> > >> + >> > >> + Get driver API version (FPGA_GET_API_VERSION) >> > >> + Check for extensions (FPGA_CHECK_EXTENSION) >> > >> + Assign port to PF (FPGA_FME_PORT_ASSIGN) >> > >> + Release port from PF (FPGA_FME_PORT_RELEASE) >> > >> + Program bitstream (FPGA_FME_PORT_PR) >> > >> + >> > >> +More functions are exposed through sysfs >> > >> +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/): >> > >> + >> > >> + Read bitstream ID (bitstream_id) >> > >> + Read bitstream metadata (bitstream_metadata) >> > >> + Read number of ports (ports_num) >> > >> + Read socket ID (socket_id) >> > >> + Read performance counters (perf/) >> > >> + Power management (power_mgmt/) >> > >> + Thermal management (thermal_mgmt/) >> > >> + Error reporting (errors/) >> > >> + >> > >> +PORT >> > >> +==== >> > >> +A port represents the interface between the static FPGA fabric (the "blue >> > >> +bitstream") and a partially reconfigurable region containing an AFU (the >> > >> "green >> > >> > Is this an fpga bridge but with added features? >> >> Yes, I think so. As you see the fme_pr function in patch 11, related port needs >> to be disabled firstly before fpga_mgr_buf_load for given accelerator. > > Can we just extend the bridge to have the additional features, please? OK then this code is taking place of a fpga-region that controls the bridge (port) and fpga-mgr during fpga programming. > >> > >> +bitstream"). It controls the communication from SW to the accelerator and >> > >> +exposes features such as reset and debug. >> > >> + >> > >> +A PCIe device may have several ports and each port can be released from >> > >> PF by >> > >> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe >> > >> sriov >> > >> +sysfs interface. >> > >> + >> > >> +AFU >> > >> +=== >> > >> +An AFU is attached to a port and exposes a 256k MMIO region to be used >> > >> for >> > >> +accelerator-specific control registers. >> > >> + >> > >> +User-space applications can acquire exclusive access to an AFU attached >> > >> to a >> > >> +port by using open() on the port device node, and release it using >> > >> close(). >> > >> + >> > >> +The following functions are exposed through ioctls: >> > >> + >> > >> + Get driver API version (FPGA_GET_API_VERSION) >> > >> + Check for extensions (FPGA_CHECK_EXTENSION) >> > >> + Get port info (FPGA_PORT_GET_INFO) >> > >> + Get MMIO region info (FPGA_PORT_GET_REGION_INFO) >> > >> + Map DMA buffer (FPGA_PORT_DMA_MAP) >> > >> + Unmap DMA buffer (FPGA_PORT_DMA_UNMAP) >> > >> + Reset AFU (FPGA_PORT_RESET) >> > >> + Enable UMsg (FPGA_PORT_UMSG_ENABLE) >> > >> + Disable UMsg (FPGA_PORT_UMSG_DISABLE) >> > >> + Set UMsg mode (FPGA_PORT_UMSG_SET_MODE) >> > >> + Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR) >> > >> + >> > >> +User-space applications can also mmap() accelerator MMIO regions. >> > >> + >> > >> +More functions are exposed through sysfs: >> > >> +(/sys/class/fpga/fpga.n/intel-fpga-port.m/): >> > >> + >> > >> + Read Accelerator GUID (afu_id) >> > >> + Error reporting (errors/) >> > >> + >> > >> +Partial Reconfiguration >> > >> +======================= >> > >> +As mentioned above, accelerators can be reconfigured through partial >> > >> +reconfiguration of a green bitstream file (GBS). The green bitstream must >> > >> have >> > >> +been generated for the exact blue bitstream and targeted reconfigurable >> > >> region >> > >> +(port) of the FPGA; otherwise, the reconfiguration operation will fail >> > >> and >> > >> +possibly cause system instability. This compatibility can be checked by >> > >> +comparing the interface ID noted in the GBS header against the interface >> > >> ID >> > >> +exposed by the FME through sysfs (see above). This check is usually done >> > >> by >> > >> +user-space before calling the reconfiguration IOCTL. >> > >> + >> > >> +FPGA virtualization >> > >> +=================== >> > >> +To enable accessing an accelerator from applications running in a VM, the >> > >> +respective AFU's port needs to be assigned to a VF using the following >> > >> steps: >> > >> + >> > >> + a) The PF owns all AFU ports by default. Any port that needs to be >> > >> reassigned >> > >> + to a VF must be released from PF firstly through the >> > >> FPGA_FME_PORT_RELEASE >> > >> + ioctl on the FME device. >> > >> + >> > >> + b) Once N ports are released from PF, then user can use below command to >> > >> + enable SRIOV and VFs. Each VF owns only one Port with AFU. >> > >> + >> > >> + echo N > $PCI_DEVICE_PATH/sriov_numvfs >> > >> + >> > >> + c) Pass through the VFs to VMs >> > >> + >> > >> + d) The AFU under VF is accessiable from applications in VM (using the >> > >> same >> > >> + driver inside the VF). >> > >> + >> > >> +Note the an FME can't be assigned to a VF, thus PR and other management >> > >> +functions are only available via the PF. >> > >> + >> > >> + >> > >> +Driver organization >> > >> +=================== >> > >> + >> > >> + +------------------+ +---------+ | +---------+ >> > >> + | +-------+ | | | | | | >> > >> + | | FPGA | FME | | AFU | | | AFU | >> > >> + | |Manager| Module | | Module | | | Module | >> > >> + | +-------+ | | | | | | >> > >> + +------------------+ +---------+ | +---------+ >> > >> + +-----------------------+ | +-----------------------+ >> > >> + | FPGA Container Device | | | FPGA Container Device | >> > >> + +-----------------------+ | +-----------------------+ >> > >> + +------------------+ | +------------------+ >> > >> + | FPGA PCIE Module | | Virtual | FPGA PCIE Module | >> > >> + +------------------+ Host | Machine +------------------+ >> > >> + ------------------------------------ | ------------------------------ >> > >> + +---------------+ | +---------------+ >> > >> + | PCI PF Device | | | PCI VF Device | >> > >> + +---------------+ | +---------------+ >> > >> + >> > >> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe >> > >> device >> > >> +driver is always loaded first once a FPGA PCIE PF or VF device is >> > >> detected. This >> > >> +driver plays an infrastructural role in the driver architecuture. It: >> > >> + >> > >> + a) creates FPGA container device as parent of the feature devices. >> > >> + b) walks through the Device Feature List, which is implemented in >> > >> PCIE >> > >> + device BAR memory, to discover feature devices and their sub >> > >> features >> > >> + and create platform device for them under the container device. >> > > >> > > >> > > I really like the idea of creating platform devices for the sub features. It >> > > is in line with other FPGA use cases. Platform devices are at the heart of >> > > device trees used by processors directly connected FPGAs and processors >> > > inside FPGAs. >> > > >> > >> + c) supports SRIOV. >> > >> + d) introduces the feature device infrastructure, which abstracts >> > >> + operations for sub features and exposes common functions to >> > >> feature >> > >> + device drivers. >> > >> + >> > >> +The FPGA Management Engine (FME) driver is a platform driver which is >> > >> loaded >> > >> +automatically after FME platform device creation from the PCIE driver. It >> > >> +provides the key features for FPGA management, including: >> > >> + >> > >> + a) Power and thermal management, error reporting, performance >> > >> reporting >> > >> + and other infrastructure functions. Users can access these >> > >> functions >> > >> + via sysfs interfaces exposed by FME driver. >> > >> + b) Paritial Reconfiguration. The FME driver registers a FPGA >> > >> Manager >> > >> + during PR sub feature initialization; once it receives an >> > >> + FPGA_FME_PORT_PR ioctl from user, it invokes the common >> > >> interface >> > >> + function from FPGA Manager to complete the partial >> > >> reconfiguration of >> > >> + the bitstream to the given port. >> > >> + c) Port management for virtualization. The FME driver introduces >> > >> two >> > >> + ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and >> > >> + FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the >> > >> port is >> > >> + released from the PF, it can be assigned to the VF through the >> > >> SRIOV >> > >> + interfaces provided by PCIE driver. (Refer to "FPGA >> > >> virtualization" >> > >> + for more details). >> > >> + >> > >> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) >> > >> driver >> > >> +is probed once the AFU platform device is created. The main function of >> > >> this >> > >> +module is to provide an interface for userspace applications to access >> > >> the >> > >> +individual accelerators, including basic reset control on port, AFU MMIO >> > >> region >> > >> +export, dma buffer mapping service, UMsg notification, and remote debug >> > >> +functions (see above). >> > >> + >> > >> + >> > >> +Device enumeration >> > >> +================== >> > >> +This section introduces how applications enumerate the fpga device from >> > >> +the sysfs hierarchy under /sys/class/fpga. >> > >> + >> > >> +In the example below, two Intel(R) FPGA devices are installed in the >> > >> host. Each >> > >> +fpga device has one FME and two ports (AFUs). >> > >> + >> > >> +For each FPGA device, a device director is created under >> > >> /sys/class/fpga/: >> > >> + >> > >> + /sys/class/fpga/fpga.0 >> > >> + /sys/class/fpga/fpga.1 >> > >> + >> > >> +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's >> > >> name. >> > >> +Application can retrieve name information via the sysfs interface: >> > >> + >> > >> + /sys/class/fpga/fpga.0/name >> > >> + >> > >> +Each node has one FME and two ports (AFUs) as child devices: >> > >> + >> > >> + /sys/class/fpga/fpga.0/intel-fpga-fme.0 >> > >> + /sys/class/fpga/fpga.0/intel-fpga-port.0 >> > >> + /sys/class/fpga/fpga.0/intel-fpga-port.1 >> > >> + >> > >> + /sys/class/fpga/fpga.1/intel-fpga-fme.1 >> > >> + /sys/class/fpga/fpga.1/intel-fpga-port.2 >> > >> + /sys/class/fpga/fpga.1/intel-fpga-port.3 >> > >> + >> > >> +In general, the FME/AFU sysfs interfaces are named as follows: >> > >> + >> > >> + /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/ >> > >> + /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/ >> > >> + >> > >> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering >> > >> all >> > >> +ports. >> > >> + >> > >> +The device nodes used for ioctl() or mmap() can be referenced through: >> > >> + >> > >> + /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev >> > >> + /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev >> > >> + >> > >> + >> > >> +Open discussions >> > >> +================ >> > >> +The current FME driver does not provide user space access to the FME MMIO >> > >> +region, but exposes access through sysfs and ioctls. It also provides an >> > >> FPGA >> > >> +manger interface for partial reconfiguration (PR), but does not make use >> > >> of >> > >> +fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled >> > >> inside >> > >> +the FME, and fpga-region depends on device tree which is not used at all. >> > >> There >> > >> +are patches from Alan Tull to separate the device tree specific code and >> > > >> > > >> > > I am currently trying to use those patches in a different driver. They've >> > > compiled cleanly in my out of tree pcie module driver against the 3.10 >> > > kernel. >> > > I need to actually write the code to create and register the region, but >> > > Alan's platform driver code should be a good guide for me. Just need to >> > > find the time. >> > > >> > >> +introduce a sysfs interface for PR. We plan to add fpga-regions support >> > >> in the >> > >> +driver once the related patches get merged. Then the FME driver should >> > >> create >> > >> +one fpga-region for each Port/AFU. >> > > >> > > >> > > Does the FME driver create the fpga-region, or is each region described as >> > > an entry in the Device Feature List and therefore created by the code that >> > > enumerates the Device Feature List? >> > > >> > >> -- >> > >> 2.7.4 >> > >> >> > >> -- >> > >> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in >> > >> the body of a message to majordomo@vger.kernel.org >> > >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > >> >> > > > > Cheers, > Moritz
[toc] | [prev] | [next] | [standalone]
| From | Wu Hao <hao.wu@intel.com> |
|---|---|
| Date | 2017-04-04 07:30 +0200 |
| Subject | Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview |
| Message-ID | <tspct-ug-1@gated-at.bofh.it> |
| In reply to | #1615550 |
On Mon, Apr 03, 2017 at 03:44:17PM -0500, Alan Tull wrote: > On Sun, Apr 2, 2017 at 9:41 AM, Moritz Fischer <mdf@kernel.org> wrote: > > On Sat, Apr 01, 2017 at 07:16:19PM +0800, Wu Hao wrote: > >> On Fri, Mar 31, 2017 at 01:38:06PM -0500, Alan Tull wrote: > >> > On Fri, Mar 31, 2017 at 1:24 PM, <matthew.gerlach@linux.intel.com> wrote: > >> > > > >> > > > >> > > On Thu, 30 Mar 2017, Wu Hao wrote: > >> > > > >> > > > >> > > Hi Wu Hao, > >> > > > >> > > Great documentation. I'm looking forward to diving into the rest of the > >> > > patches. Please see my comments inline. > >> > > > >> > > Matthew Gerlach > >> > > > >> > > > >> > >> Add a document for Intel FPGA driver overview. > >> > >> > >> > >> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com> > >> > >> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com> > >> > >> Signed-off-by: Wu Hao <hao.wu@intel.com> > >> > >> --- > >> > >> Documentation/fpga/intel-fpga.txt | 259 > >> > >> ++++++++++++++++++++++++++++++++++++++ > >> > >> 1 file changed, 259 insertions(+) > >> > >> create mode 100644 Documentation/fpga/intel-fpga.txt > >> > >> > >> > >> diff --git a/Documentation/fpga/intel-fpga.txt > >> > >> b/Documentation/fpga/intel-fpga.txt > >> > >> new file mode 100644 > >> > >> index 0000000..9396cea > >> > >> --- /dev/null > >> > >> +++ b/Documentation/fpga/intel-fpga.txt > >> > >> @@ -0,0 +1,259 @@ > >> > >> > >> > >> +=============================================================================== > >> > >> + Intel FPGA driver Overview > >> > >> > >> > >> +------------------------------------------------------------------------------- > >> > >> + Enno Luebbers <enno.luebbers@intel.com> > >> > >> + Xiao Guangrong <guangrong.xiao@linux.intel.com> > >> > >> + Wu Hao <hao.wu@intel.com> > >> > >> + > >> > >> +The Intel FPGA driver provides interfaces for userspace applications to > >> > >> +configure, enumerate, open, and access FPGA accelerators on platforms > >> > >> equipped > >> > >> +with Intel(R) FPGA solutions and enables system level management > >> > >> functions such > >> > >> +as FPGA reconfiguration, power management, and virtualization. > >> > >> + > >> > > > >> > > > >> > > From a Linux kernel perspective, I'm not sure this is the best name for > >> > > this code. The name gives me the impression that it is a driver for all > >> > > Intel FPGAs, but not all Intel FPGAs are connected to the processor over a > >> > > PCIe bus. The processor could be directely connected like the Arria10 > >> > > SOCFPGA. Such a processor could certainly benefit from this accelerator > >> > > usage model. In an extreme case, couldn't a processor in the FPGA, > >> > > running Linux, also benefit from this accelerator model? Is this code a > >> > > "FPGA Accelerator Framework"? > >> > > > >> > >> +HW Architecture > >> > >> +=============== > >> > >> +From the OS's point of view, the FPGA hardware appears as a regular PCIe > >> > >> device. > >> > >> +The FPGA device memory is organized using a predefined data structure > >> > >> (Device > >> > >> +Feature List). Features supported by the particular FPGA device are > >> > >> exposed > >> > >> +through these data structures, as illustrated below: > >> > >> + > >> > >> + +-------------------------------+ +-------------+ > >> > >> + | PF | | VF | > >> > >> + +-------------------------------+ +-------------+ > >> > >> + ^ ^ ^ ^ > >> > >> + | | | | > >> > >> ++-----|------------|---------|--------------|-------+ > >> > >> +| | | | | | > >> > >> +| +-----+ +-------+ +-------+ +-------+ | > >> > >> +| | FME | | Port0 | | Port1 | | Port2 | | > >> > >> +| +-----+ +-------+ +-------+ +-------+ | > >> > >> +| ^ ^ ^ | > >> > >> +| | | | | > >> > >> +| +-------+ +------+ +-------+ | > >> > >> +| | AFU | | AFU | | AFU | | > >> > >> +| +-------+ +------+ +-------+ | > >> > >> +| | > >> > >> +| FPGA PCIe Device | > >> > >> ++---------------------------------------------------+ > >> > >> + > >> > >> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which > >> > >> can be > >> > >> +used to assign individual accelerators to virtual machines . > >> > > > >> > > > >> > > Does this HW Architecture require an Intel FPGA? Couldn't any vendors FPGA > >> > > be used as long as it presented itself the PCIe bus the same and contained > >> > > an appropriate Device Feature List? > > > > I think this is a good (and important) point. Especially when sysfs > > entries & ioctls constituting ABI depend on it. > > > >> > > > >> > >> + > >> > >> +FME (FPGA Management Engine) > >> > >> +============================ > >> > >> +The FPGA Management Enging performs power and thermal management, error > > Enging->Engine > >> > >> +reporting, reconfiguration, performance reporting, and other > >> > >> infrastructure > >> > >> +functions. Each FPGA has one FME, which is always accessed through the > >> > >> physical > >> > >> +function (PF). > >> > >> + > >> > >> +User-space applications can acquire exclusive access to the FME using > >> > >> open(), > >> > >> +and release it using close(). > >> > >> + > >> > >> +The following functions are exposed through ioctls: > >> > >> + > >> > >> + Get driver API version (FPGA_GET_API_VERSION) > >> > >> + Check for extensions (FPGA_CHECK_EXTENSION) > >> > >> + Assign port to PF (FPGA_FME_PORT_ASSIGN) > >> > >> + Release port from PF (FPGA_FME_PORT_RELEASE) > >> > >> + Program bitstream (FPGA_FME_PORT_PR) > >> > >> + > >> > >> +More functions are exposed through sysfs > >> > >> +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/): > >> > >> + > >> > >> + Read bitstream ID (bitstream_id) > >> > >> + Read bitstream metadata (bitstream_metadata) > >> > >> + Read number of ports (ports_num) > >> > >> + Read socket ID (socket_id) > >> > >> + Read performance counters (perf/) > >> > >> + Power management (power_mgmt/) > >> > >> + Thermal management (thermal_mgmt/) > >> > >> + Error reporting (errors/) > >> > >> + > >> > >> +PORT > >> > >> +==== > >> > >> +A port represents the interface between the static FPGA fabric (the "blue > >> > >> +bitstream") and a partially reconfigurable region containing an AFU (the > >> > >> "green > >> > > >> > Is this an fpga bridge but with added features? > >> > >> Yes, I think so. As you see the fme_pr function in patch 11, related port needs > >> to be disabled firstly before fpga_mgr_buf_load for given accelerator. > > > > Can we just extend the bridge to have the additional features, please? > > OK then this code is taking place of a fpga-region that controls the > bridge (port) and fpga-mgr during fpga programming. > As mentioned in last email replied to Moritz, I prefer to have fpga-bridge in FME module together with fpga-region and fpga-manager, and reuse fpga region related function for PR. Other functions which required by user space applications when access the FPGA acclerator, should be covered in AFU driver. Please notice that In VF case (e.g in virtual machine), there is no FME at all, but only FPGA accelerators (AFUs). Create a duplciate fpga-bridge in AFU driver seems not useful. Thanks Hao
[toc] | [prev] | [next] | [standalone]
| From | Wu Hao <hao.wu@intel.com> |
|---|---|
| Date | 2017-04-04 07:20 +0200 |
| Subject | Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview |
| Message-ID | <tsp2O-qQ-11@gated-at.bofh.it> |
| In reply to | #1614753 |
On Sun, Apr 02, 2017 at 07:41:46AM -0700, Moritz Fischer wrote: > On Sat, Apr 01, 2017 at 07:16:19PM +0800, Wu Hao wrote: > > On Fri, Mar 31, 2017 at 01:38:06PM -0500, Alan Tull wrote: > > > On Fri, Mar 31, 2017 at 1:24 PM, <matthew.gerlach@linux.intel.com> wrote: > > > > > > > > > > > > On Thu, 30 Mar 2017, Wu Hao wrote: > > > > > > > > > > > > Hi Wu Hao, > > > > > > > > Great documentation. I'm looking forward to diving into the rest of the > > > > patches. Please see my comments inline. > > > > > > > > Matthew Gerlach > > > > > > > > > > > >> Add a document for Intel FPGA driver overview. > > > >> > > > >> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com> > > > >> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com> > > > >> Signed-off-by: Wu Hao <hao.wu@intel.com> > > > >> --- > > > >> Documentation/fpga/intel-fpga.txt | 259 > > > >> ++++++++++++++++++++++++++++++++++++++ > > > >> 1 file changed, 259 insertions(+) > > > >> create mode 100644 Documentation/fpga/intel-fpga.txt > > > >> > > > >> diff --git a/Documentation/fpga/intel-fpga.txt > > > >> b/Documentation/fpga/intel-fpga.txt > > > >> new file mode 100644 > > > >> index 0000000..9396cea > > > >> --- /dev/null > > > >> +++ b/Documentation/fpga/intel-fpga.txt > > > >> @@ -0,0 +1,259 @@ > > > >> > > > >> +=============================================================================== > > > >> + Intel FPGA driver Overview > > > >> > > > >> +------------------------------------------------------------------------------- > > > >> + Enno Luebbers <enno.luebbers@intel.com> > > > >> + Xiao Guangrong <guangrong.xiao@linux.intel.com> > > > >> + Wu Hao <hao.wu@intel.com> > > > >> + > > > >> +The Intel FPGA driver provides interfaces for userspace applications to > > > >> +configure, enumerate, open, and access FPGA accelerators on platforms > > > >> equipped > > > >> +with Intel(R) FPGA solutions and enables system level management > > > >> functions such > > > >> +as FPGA reconfiguration, power management, and virtualization. > > > >> + > > > > > > > > > > > > From a Linux kernel perspective, I'm not sure this is the best name for > > > > this code. The name gives me the impression that it is a driver for all > > > > Intel FPGAs, but not all Intel FPGAs are connected to the processor over a > > > > PCIe bus. The processor could be directely connected like the Arria10 > > > > SOCFPGA. Such a processor could certainly benefit from this accelerator > > > > usage model. In an extreme case, couldn't a processor in the FPGA, > > > > running Linux, also benefit from this accelerator model? Is this code a > > > > "FPGA Accelerator Framework"? > > > > > > > >> +HW Architecture > > > >> +=============== > > > >> +From the OS's point of view, the FPGA hardware appears as a regular PCIe > > > >> device. > > > >> +The FPGA device memory is organized using a predefined data structure > > > >> (Device > > > >> +Feature List). Features supported by the particular FPGA device are > > > >> exposed > > > >> +through these data structures, as illustrated below: > > > >> + > > > >> + +-------------------------------+ +-------------+ > > > >> + | PF | | VF | > > > >> + +-------------------------------+ +-------------+ > > > >> + ^ ^ ^ ^ > > > >> + | | | | > > > >> ++-----|------------|---------|--------------|-------+ > > > >> +| | | | | | > > > >> +| +-----+ +-------+ +-------+ +-------+ | > > > >> +| | FME | | Port0 | | Port1 | | Port2 | | > > > >> +| +-----+ +-------+ +-------+ +-------+ | > > > >> +| ^ ^ ^ | > > > >> +| | | | | > > > >> +| +-------+ +------+ +-------+ | > > > >> +| | AFU | | AFU | | AFU | | > > > >> +| +-------+ +------+ +-------+ | > > > >> +| | > > > >> +| FPGA PCIe Device | > > > >> ++---------------------------------------------------+ > > > >> + > > > >> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which > > > >> can be > > > >> +used to assign individual accelerators to virtual machines . > > > > > > > > > > > > Does this HW Architecture require an Intel FPGA? Couldn't any vendors FPGA > > > > be used as long as it presented itself the PCIe bus the same and contained > > > > an appropriate Device Feature List? > > I think this is a good (and important) point. Especially when sysfs > entries & ioctls constituting ABI depend on it. Thanks for your feedback and comments. I'm not sure if any vendors FPGA will resue the same. But if the same FME or Port/AFU module implemented in their hardwares, then they should be able to use our FME/AFU drivers. AFU driver creates interfaces for FPGA accelerators. And FME driver creates interfaces for FPGA partial reconfiguration and other management functions. As mentioned in 'Open discussions' below, we may need to switch to the ABI (sysfs) of fpga-region for PR. > > > > > > > > >> + > > > >> +FME (FPGA Management Engine) > > > >> +============================ > > > >> +The FPGA Management Enging performs power and thermal management, error > Enging->Engine > > > >> +reporting, reconfiguration, performance reporting, and other > > > >> infrastructure > > > >> +functions. Each FPGA has one FME, which is always accessed through the > > > >> physical > > > >> +function (PF). > > > >> + > > > >> +User-space applications can acquire exclusive access to the FME using > > > >> open(), > > > >> +and release it using close(). > > > >> + > > > >> +The following functions are exposed through ioctls: > > > >> + > > > >> + Get driver API version (FPGA_GET_API_VERSION) > > > >> + Check for extensions (FPGA_CHECK_EXTENSION) > > > >> + Assign port to PF (FPGA_FME_PORT_ASSIGN) > > > >> + Release port from PF (FPGA_FME_PORT_RELEASE) > > > >> + Program bitstream (FPGA_FME_PORT_PR) > > > >> + > > > >> +More functions are exposed through sysfs > > > >> +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/): > > > >> + > > > >> + Read bitstream ID (bitstream_id) > > > >> + Read bitstream metadata (bitstream_metadata) > > > >> + Read number of ports (ports_num) > > > >> + Read socket ID (socket_id) > > > >> + Read performance counters (perf/) > > > >> + Power management (power_mgmt/) > > > >> + Thermal management (thermal_mgmt/) > > > >> + Error reporting (errors/) > > > >> + > > > >> +PORT > > > >> +==== > > > >> +A port represents the interface between the static FPGA fabric (the "blue > > > >> +bitstream") and a partially reconfigurable region containing an AFU (the > > > >> "green > > > > > > Is this an fpga bridge but with added features? > > > > Yes, I think so. As you see the fme_pr function in patch 11, related port needs > > to be disabled firstly before fpga_mgr_buf_load for given accelerator. > > Can we just extend the bridge to have the additional features, please? As described in this document, in Intel FPGA device, there are two types of modudles, FME which provides management function including PR and AFU which provides the access to the FPGA accelerators, and other advanced features (e.g error reporting, debug, reset and etc) required by user space applications. I think FME needs to create fpga-bridge as well as fpga-manager and regions. Then enable/disable bridge action could be covered automatically by fpga-region function. (e.g fpga_region_program_fpga). Thanks Hao > > > > >> +bitstream"). It controls the communication from SW to the accelerator and > > > >> +exposes features such as reset and debug. > > > >> + > > > >> +A PCIe device may have several ports and each port can be released from > > > >> PF by > > > >> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe > > > >> sriov > > > >> +sysfs interface. > > > >> + > > > >> +AFU > > > >> +=== > > > >> +An AFU is attached to a port and exposes a 256k MMIO region to be used > > > >> for > > > >> +accelerator-specific control registers. > > > >> + > > > >> +User-space applications can acquire exclusive access to an AFU attached > > > >> to a > > > >> +port by using open() on the port device node, and release it using > > > >> close(). > > > >> + > > > >> +The following functions are exposed through ioctls: > > > >> + > > > >> + Get driver API version (FPGA_GET_API_VERSION) > > > >> + Check for extensions (FPGA_CHECK_EXTENSION) > > > >> + Get port info (FPGA_PORT_GET_INFO) > > > >> + Get MMIO region info (FPGA_PORT_GET_REGION_INFO) > > > >> + Map DMA buffer (FPGA_PORT_DMA_MAP) > > > >> + Unmap DMA buffer (FPGA_PORT_DMA_UNMAP) > > > >> + Reset AFU (FPGA_PORT_RESET) > > > >> + Enable UMsg (FPGA_PORT_UMSG_ENABLE) > > > >> + Disable UMsg (FPGA_PORT_UMSG_DISABLE) > > > >> + Set UMsg mode (FPGA_PORT_UMSG_SET_MODE) > > > >> + Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR) > > > >> + > > > >> +User-space applications can also mmap() accelerator MMIO regions. > > > >> + > > > >> +More functions are exposed through sysfs: > > > >> +(/sys/class/fpga/fpga.n/intel-fpga-port.m/): > > > >> + > > > >> + Read Accelerator GUID (afu_id) > > > >> + Error reporting (errors/) > > > >> + > > > >> +Partial Reconfiguration > > > >> +======================= > > > >> +As mentioned above, accelerators can be reconfigured through partial > > > >> +reconfiguration of a green bitstream file (GBS). The green bitstream must > > > >> have > > > >> +been generated for the exact blue bitstream and targeted reconfigurable > > > >> region > > > >> +(port) of the FPGA; otherwise, the reconfiguration operation will fail > > > >> and > > > >> +possibly cause system instability. This compatibility can be checked by > > > >> +comparing the interface ID noted in the GBS header against the interface > > > >> ID > > > >> +exposed by the FME through sysfs (see above). This check is usually done > > > >> by > > > >> +user-space before calling the reconfiguration IOCTL. > > > >> + > > > >> +FPGA virtualization > > > >> +=================== > > > >> +To enable accessing an accelerator from applications running in a VM, the > > > >> +respective AFU's port needs to be assigned to a VF using the following > > > >> steps: > > > >> + > > > >> + a) The PF owns all AFU ports by default. Any port that needs to be > > > >> reassigned > > > >> + to a VF must be released from PF firstly through the > > > >> FPGA_FME_PORT_RELEASE > > > >> + ioctl on the FME device. > > > >> + > > > >> + b) Once N ports are released from PF, then user can use below command to > > > >> + enable SRIOV and VFs. Each VF owns only one Port with AFU. > > > >> + > > > >> + echo N > $PCI_DEVICE_PATH/sriov_numvfs > > > >> + > > > >> + c) Pass through the VFs to VMs > > > >> + > > > >> + d) The AFU under VF is accessiable from applications in VM (using the > > > >> same > > > >> + driver inside the VF). > > > >> + > > > >> +Note the an FME can't be assigned to a VF, thus PR and other management > > > >> +functions are only available via the PF. > > > >> + > > > >> + > > > >> +Driver organization > > > >> +=================== > > > >> + > > > >> + +------------------+ +---------+ | +---------+ > > > >> + | +-------+ | | | | | | > > > >> + | | FPGA | FME | | AFU | | | AFU | > > > >> + | |Manager| Module | | Module | | | Module | > > > >> + | +-------+ | | | | | | > > > >> + +------------------+ +---------+ | +---------+ > > > >> + +-----------------------+ | +-----------------------+ > > > >> + | FPGA Container Device | | | FPGA Container Device | > > > >> + +-----------------------+ | +-----------------------+ > > > >> + +------------------+ | +------------------+ > > > >> + | FPGA PCIE Module | | Virtual | FPGA PCIE Module | > > > >> + +------------------+ Host | Machine +------------------+ > > > >> + ------------------------------------ | ------------------------------ > > > >> + +---------------+ | +---------------+ > > > >> + | PCI PF Device | | | PCI VF Device | > > > >> + +---------------+ | +---------------+ > > > >> + > > > >> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe > > > >> device > > > >> +driver is always loaded first once a FPGA PCIE PF or VF device is > > > >> detected. This > > > >> +driver plays an infrastructural role in the driver architecuture. It: > > > >> + > > > >> + a) creates FPGA container device as parent of the feature devices. > > > >> + b) walks through the Device Feature List, which is implemented in > > > >> PCIE > > > >> + device BAR memory, to discover feature devices and their sub > > > >> features > > > >> + and create platform device for them under the container device. > > > > > > > > > > > > I really like the idea of creating platform devices for the sub features. It > > > > is in line with other FPGA use cases. Platform devices are at the heart of > > > > device trees used by processors directly connected FPGAs and processors > > > > inside FPGAs. > > > > > > > >> + c) supports SRIOV. > > > >> + d) introduces the feature device infrastructure, which abstracts > > > >> + operations for sub features and exposes common functions to > > > >> feature > > > >> + device drivers. > > > >> + > > > >> +The FPGA Management Engine (FME) driver is a platform driver which is > > > >> loaded > > > >> +automatically after FME platform device creation from the PCIE driver. It > > > >> +provides the key features for FPGA management, including: > > > >> + > > > >> + a) Power and thermal management, error reporting, performance > > > >> reporting > > > >> + and other infrastructure functions. Users can access these > > > >> functions > > > >> + via sysfs interfaces exposed by FME driver. > > > >> + b) Paritial Reconfiguration. The FME driver registers a FPGA > > > >> Manager > > > >> + during PR sub feature initialization; once it receives an > > > >> + FPGA_FME_PORT_PR ioctl from user, it invokes the common > > > >> interface > > > >> + function from FPGA Manager to complete the partial > > > >> reconfiguration of > > > >> + the bitstream to the given port. > > > >> + c) Port management for virtualization. The FME driver introduces > > > >> two > > > >> + ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and > > > >> + FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the > > > >> port is > > > >> + released from the PF, it can be assigned to the VF through the > > > >> SRIOV > > > >> + interfaces provided by PCIE driver. (Refer to "FPGA > > > >> virtualization" > > > >> + for more details). > > > >> + > > > >> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) > > > >> driver > > > >> +is probed once the AFU platform device is created. The main function of > > > >> this > > > >> +module is to provide an interface for userspace applications to access > > > >> the > > > >> +individual accelerators, including basic reset control on port, AFU MMIO > > > >> region > > > >> +export, dma buffer mapping service, UMsg notification, and remote debug > > > >> +functions (see above). > > > >> + > > > >> + > > > >> +Device enumeration > > > >> +================== > > > >> +This section introduces how applications enumerate the fpga device from > > > >> +the sysfs hierarchy under /sys/class/fpga. > > > >> + > > > >> +In the example below, two Intel(R) FPGA devices are installed in the > > > >> host. Each > > > >> +fpga device has one FME and two ports (AFUs). > > > >> + > > > >> +For each FPGA device, a device director is created under > > > >> /sys/class/fpga/: > > > >> + > > > >> + /sys/class/fpga/fpga.0 > > > >> + /sys/class/fpga/fpga.1 > > > >> + > > > >> +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's > > > >> name. > > > >> +Application can retrieve name information via the sysfs interface: > > > >> + > > > >> + /sys/class/fpga/fpga.0/name > > > >> + > > > >> +Each node has one FME and two ports (AFUs) as child devices: > > > >> + > > > >> + /sys/class/fpga/fpga.0/intel-fpga-fme.0 > > > >> + /sys/class/fpga/fpga.0/intel-fpga-port.0 > > > >> + /sys/class/fpga/fpga.0/intel-fpga-port.1 > > > >> + > > > >> + /sys/class/fpga/fpga.1/intel-fpga-fme.1 > > > >> + /sys/class/fpga/fpga.1/intel-fpga-port.2 > > > >> + /sys/class/fpga/fpga.1/intel-fpga-port.3 > > > >> + > > > >> +In general, the FME/AFU sysfs interfaces are named as follows: > > > >> + > > > >> + /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/ > > > >> + /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/ > > > >> + > > > >> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering > > > >> all > > > >> +ports. > > > >> + > > > >> +The device nodes used for ioctl() or mmap() can be referenced through: > > > >> + > > > >> + /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev > > > >> + /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev > > > >> + > > > >> + > > > >> +Open discussions > > > >> +================ > > > >> +The current FME driver does not provide user space access to the FME MMIO > > > >> +region, but exposes access through sysfs and ioctls. It also provides an > > > >> FPGA > > > >> +manger interface for partial reconfiguration (PR), but does not make use > > > >> of > > > >> +fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled > > > >> inside > > > >> +the FME, and fpga-region depends on device tree which is not used at all. > > > >> There > > > >> +are patches from Alan Tull to separate the device tree specific code and > > > > > > > > > > > > I am currently trying to use those patches in a different driver. They've > > > > compiled cleanly in my out of tree pcie module driver against the 3.10 > > > > kernel. > > > > I need to actually write the code to create and register the region, but > > > > Alan's platform driver code should be a good guide for me. Just need to > > > > find the time. > > > > > > > >> +introduce a sysfs interface for PR. We plan to add fpga-regions support > > > >> in the > > > >> +driver once the related patches get merged. Then the FME driver should > > > >> create > > > >> +one fpga-region for each Port/AFU. > > > > > > > > > > > > Does the FME driver create the fpga-region, or is each region described as > > > > an entry in the Device Feature List and therefore created by the code that > > > > enumerates the Device Feature List? > > > > > > > >> -- > > > >> 2.7.4 > > > >> > > > >> -- > > > >> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in > > > >> the body of a message to majordomo@vger.kernel.org > > > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > > >> > > > > > > Cheers, > Moritz
[toc] | [prev] | [next] | [standalone]
| From | Wu Hao <hao.wu@intel.com> |
|---|---|
| Date | 2017-03-30 14:20 +0200 |
| Subject | [PATCH 06/16] fpga: intel: pcie: adds fpga_for_each_port callback for fme device |
| Message-ID | <tqHdw-69w-37@gated-at.bofh.it> |
| In reply to | #1613003 |
For FPGA Management Engine (FME), it requires fpga_for_each_port callback
for actions on ports, so export this function from PCIe driver by adding
the callback to the platform data.
Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
drivers/fpga/intel/feature-dev.h | 9 +++++++++
drivers/fpga/intel/pcie.c | 24 ++++++++++++++++++++++++
2 files changed, 33 insertions(+)
diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
index d1723ff..38531f8 100644
--- a/drivers/fpga/intel/feature-dev.h
+++ b/drivers/fpga/intel/feature-dev.h
@@ -221,6 +221,9 @@ struct feature_platform_data {
struct platform_device *dev;
unsigned int disable_count; /* count for port disable */
+ struct platform_device *(*fpga_for_each_port)(struct platform_device *,
+ void *, int (*match)(struct platform_device *, void *));
+
int num; /* number of features */
struct feature features[0];
};
@@ -335,6 +338,12 @@ get_feature_ioaddr_by_index(struct device *dev, int index)
return pdata->features[index].ioaddr;
}
+static inline struct device *
+fpga_feature_dev_to_pcidev(struct platform_device *dev)
+{
+ return dev->dev.parent->parent;
+}
+
/*
* Wait register's _field to be changed to the given value (_expect's _field)
* by polling with given interval and timeout.
diff --git a/drivers/fpga/intel/pcie.c b/drivers/fpga/intel/pcie.c
index e3440ca..f2b458d 100644
--- a/drivers/fpga/intel/pcie.c
+++ b/drivers/fpga/intel/pcie.c
@@ -211,6 +211,27 @@ static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
return parse_start_from(binfo, bar);
}
+static struct platform_device *fpga_for_each_port(struct platform_device *pdev,
+ void *data, int (*match)(struct platform_device *, void *))
+{
+ struct device *pci_dev = fpga_feature_dev_to_pcidev(pdev);
+ struct cci_drvdata *drvdata = dev_get_drvdata(pci_dev);
+ struct feature_platform_data *pdata;
+ struct platform_device *port_dev;
+
+ mutex_lock(&drvdata->lock);
+ list_for_each_entry(pdata, &drvdata->port_dev_list, node) {
+ port_dev = pdata->dev;
+
+ if (match(port_dev, data) && get_device(&port_dev->dev))
+ goto exit;
+ }
+ port_dev = NULL;
+exit:
+ mutex_unlock(&drvdata->lock);
+ return port_dev;
+}
+
static struct build_feature_devs_info *
build_info_alloc_and_init(struct pci_dev *pdev)
{
@@ -312,6 +333,9 @@ build_info_create_dev(struct build_feature_devs_info *binfo,
if (!pdata)
return -ENOMEM;
+ if (type == FME_ID)
+ pdata->fpga_for_each_port = fpga_for_each_port;
+
/*
* the count should be initialized to 0 to make sure
*__fpga_port_enable() following __fpga_port_disable()
--
2.7.4
[toc] | [prev] | [next] | [standalone]
| From | Wu Hao <hao.wu@intel.com> |
|---|---|
| Date | 2017-03-30 14:20 +0200 |
| Subject | [PATCH 13/16] fpga: intel: afu: add header sub feature support |
| Message-ID | <tqHdx-69w-47@gated-at.bofh.it> |
| In reply to | #1613003 |
The header register set is always present for the Port/AFU, it is mainly
for capability, control and status of the ports that AFU connected to.
This patch implements header sub feature support. Below user interfaces
are created by this patch.
Sysfs interface:
* /sys/class/fpga/<fpga.x>/<intel-fpga-port.x>/id
Read-only. Port ID.
Ioctl interface:
* FPGA_PORT_RESET
Reset the FPGA AFU Port.
Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
drivers/fpga/intel/afu-main.c | 44 ++++++++++++++++++++++++++++++++++++++++-
include/uapi/linux/intel-fpga.h | 14 +++++++++++++
2 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/drivers/fpga/intel/afu-main.c b/drivers/fpga/intel/afu-main.c
index 1c2035b..7166d5c 100644
--- a/drivers/fpga/intel/afu-main.c
+++ b/drivers/fpga/intel/afu-main.c
@@ -20,25 +20,66 @@
#include <linux/kernel.h>
#include <linux/module.h>
+#include <linux/intel-fpga.h>
#include "feature-dev.h"
+static ssize_t
+id_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ int id = fpga_port_id(to_platform_device(dev));
+
+ return scnprintf(buf, PAGE_SIZE, "%d\n", id);
+}
+static DEVICE_ATTR_RO(id);
+
+static const struct attribute *port_hdr_attrs[] = {
+ &dev_attr_id.attr,
+ NULL,
+};
+
static int port_hdr_init(struct platform_device *pdev, struct feature *feature)
{
dev_dbg(&pdev->dev, "PORT HDR Init.\n");
- return 0;
+ fpga_port_reset(pdev);
+
+ return sysfs_create_files(&pdev->dev.kobj, port_hdr_attrs);
}
static void port_hdr_uinit(struct platform_device *pdev,
struct feature *feature)
{
dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
+
+ sysfs_remove_files(&pdev->dev.kobj, port_hdr_attrs);
+}
+
+static long
+port_hdr_ioctl(struct platform_device *pdev, struct feature *feature,
+ unsigned int cmd, unsigned long arg)
+{
+ long ret;
+
+ switch (cmd) {
+ case FPGA_PORT_RESET:
+ if (!arg)
+ ret = fpga_port_reset(pdev);
+ else
+ ret = -EINVAL;
+ break;
+ default:
+ dev_dbg(&pdev->dev, "%x cmd not handled", cmd);
+ ret = -ENODEV;
+ }
+
+ return ret;
}
struct feature_ops port_hdr_ops = {
.init = port_hdr_init,
.uinit = port_hdr_uinit,
+ .ioctl = port_hdr_ioctl,
};
static struct feature_driver port_feature_drvs[] = {
@@ -78,6 +119,7 @@ static int afu_release(struct inode *inode, struct file *filp)
dev_dbg(&pdev->dev, "Device File Release\n");
+ fpga_port_reset(pdev);
feature_dev_use_end(pdata);
return 0;
}
diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
index 77658316..13b2e61 100644
--- a/include/uapi/linux/intel-fpga.h
+++ b/include/uapi/linux/intel-fpga.h
@@ -32,8 +32,11 @@
#define FPGA_MAGIC 0xB6
#define FPGA_BASE 0
+#define PORT_BASE 0x40
#define FME_BASE 0x80
+/* Common IOCTLs for both FME and AFU file descriptor */
+
/**
* FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
*
@@ -52,6 +55,17 @@
#define FPGA_CHECK_EXTENSION _IO(FPGA_MAGIC, FPGA_BASE + 1)
+/* IOCTLs for AFU file descriptor */
+
+/**
+ * FPGA_PORT_RESET - _IO(FPGA_MAGIC, PORT_BASE + 0)
+ *
+ * Reset the FPGA AFU Port. No parameters are supported.
+ * Return: 0 on success, -errno of failure
+ */
+
+#define FPGA_PORT_RESET _IO(FPGA_MAGIC, PORT_BASE + 0)
+
/* IOCTLs for FME file descriptor */
/**
--
2.7.4
[toc] | [prev] | [next] | [standalone]
| From | Wu Hao <hao.wu@intel.com> |
|---|---|
| Date | 2017-03-30 14:20 +0200 |
| Subject | [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support |
| Message-ID | <tqHdx-69w-43@gated-at.bofh.it> |
| In reply to | #1613003 |
From: Kang Luwei <luwei.kang@intel.com>
Partial Reconfiguration (PR) is the most important function for FME. It
allows reconfiguration for given Port/Accelerated Function Unit (AFU).
This patch adds support for PR sub feature. In this patch, it registers
a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
for PR operation once PR request received via ioctl. Below user space
interfaces are exposed by this sub feature.
Sysfs interface:
* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
Read-only. Indicate the hardware interface information. Userspace
applications need to check this interface to select correct green
bitstream format before PR.
Ioctl interface:
* FPGA_FME_PORT_PR
Do partial reconfiguration per information from userspace, including
target port(AFU), buffer size and address info. It returns the PR status
(PR error code if failed) to userspace.
Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Alan Tull <alan.tull@intel.com>
Signed-off-by: Kang Luwei <luwei.kang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
drivers/fpga/intel/Makefile | 2 +-
drivers/fpga/intel/feature-dev.h | 58 ++++++
drivers/fpga/intel/fme-main.c | 44 ++++-
drivers/fpga/intel/fme-pr.c | 400 +++++++++++++++++++++++++++++++++++++++
drivers/fpga/intel/fme.h | 32 ++++
include/uapi/linux/intel-fpga.h | 44 +++++
6 files changed, 578 insertions(+), 2 deletions(-)
create mode 100644 drivers/fpga/intel/fme-pr.c
create mode 100644 drivers/fpga/intel/fme.h
diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
index 546861d..0452cb6 100644
--- a/drivers/fpga/intel/Makefile
+++ b/drivers/fpga/intel/Makefile
@@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
intel-fpga-pci-objs := pcie.o feature-dev.o
-intel-fpga-fme-objs := fme-main.o
+intel-fpga-fme-objs := fme-main.o fme-pr.o
diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
index dccc283..5a25c915 100644
--- a/drivers/fpga/intel/feature-dev.h
+++ b/drivers/fpga/intel/feature-dev.h
@@ -150,8 +150,66 @@ struct feature_fme_err {
};
/* FME Partial Reconfiguration Sub Feature Register Set */
+/* FME PR Control Register */
+struct feature_fme_pr_ctl {
+ union {
+ u64 csr;
+ struct {
+ u8 pr_reset:1; /* Reset PR Engine */
+ u8 rsvdz1:3;
+ u8 pr_reset_ack:1; /* Reset PR Engine Ack */
+ u8 rsvdz2:3;
+ u8 pr_regionid:2; /* PR Region ID */
+ u8 rsvdz3:2;
+ u8 pr_start_req:1; /* PR Start Request */
+ u8 pr_push_complete:1; /* PR Data push complete */
+ u8 pr_kind:1; /* Load Customer or Intel GBS */
+ u32 rsvdz4:17;
+ u32 config_data;
+ };
+ };
+};
+
+/* FME PR Status Register */
+struct feature_fme_pr_status {
+ union {
+ u64 csr;
+ struct {
+ u16 pr_credit:9; /* Number of PR Credits */
+ u8 rsvdz1:7;
+ u8 pr_status:1; /* PR Operation status */
+ u8 rsvdz2:3;
+ u8 pr_ctrlr_status:3; /* Controller status */
+ u8 rsvdz3:1;
+ u8 pr_host_status:4; /* PR Host status */
+ u64 rsvdz4:36;
+ };
+ };
+};
+
+/* FME PR Data Register */
+struct feature_fme_pr_data {
+ union {
+ u64 csr;
+ struct {
+ /* PR data from the raw-binary file */
+ u32 pr_data_raw;
+ u32 rsvd;
+ };
+ };
+};
+
struct feature_fme_pr {
struct feature_header header;
+ struct feature_fme_pr_ctl control;
+ struct feature_fme_pr_status status;
+ struct feature_fme_pr_data data;
+ u64 error;
+
+ u64 rsvd[16];
+
+ u64 intfc_id_l; /* PR interface Id Low */
+ u64 intfc_id_h; /* PR interface Id High */
};
/* PORT Header Register Set */
diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
index 36d0c4c..0d9a7a6 100644
--- a/drivers/fpga/intel/fme-main.c
+++ b/drivers/fpga/intel/fme-main.c
@@ -23,6 +23,7 @@
#include <linux/intel-fpga.h>
#include "feature-dev.h"
+#include "fme.h"
static ssize_t ports_num_show(struct device *dev,
struct device_attribute *attr, char *buf)
@@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
.ops = &fme_hdr_ops,
},
{
+ .name = FME_FEATURE_PR_MGMT,
+ .ops = &pr_mgmt_ops,
+ },
+ {
.ops = NULL,
},
};
@@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
.unlocked_ioctl = fme_ioctl,
};
+static int fme_dev_init(struct platform_device *pdev)
+{
+ struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+ struct fpga_fme *fme;
+
+ fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
+ if (!fme)
+ return -ENOMEM;
+
+ fme->pdata = pdata;
+
+ mutex_lock(&pdata->lock);
+ fpga_pdata_set_private(pdata, fme);
+ mutex_unlock(&pdata->lock);
+ return 0;
+}
+
+static void fme_dev_destroy(struct platform_device *pdev)
+{
+ struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+ struct fpga_fme *fme;
+
+ mutex_lock(&pdata->lock);
+ fme = fpga_pdata_get_private(pdata);
+ fpga_pdata_set_private(pdata, NULL);
+ mutex_unlock(&pdata->lock);
+
+ devm_kfree(&pdev->dev, fme);
+}
+
static int fme_probe(struct platform_device *pdev)
{
int ret;
- ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+ ret = fme_dev_init(pdev);
if (ret)
goto exit;
+ ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+ if (ret)
+ goto dev_destroy;
+
ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
if (ret)
goto feature_uinit;
@@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
feature_uinit:
fpga_dev_feature_uinit(pdev);
+dev_destroy:
+ fme_dev_destroy(pdev);
exit:
return ret;
}
@@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
{
fpga_dev_feature_uinit(pdev);
fpga_unregister_dev_ops(pdev);
+ fme_dev_destroy(pdev);
return 0;
}
diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
new file mode 100644
index 0000000..3b44a3e
--- /dev/null
+++ b/drivers/fpga/intel/fme-pr.c
@@ -0,0 +1,400 @@
+/*
+ * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ * Kang Luwei <luwei.kang@intel.com>
+ * Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ * Joseph Grecco <joe.grecco@intel.com>
+ * Enno Luebbers <enno.luebbers@intel.com>
+ * Tim Whisonant <tim.whisonant@intel.com>
+ * Ananda Ravuri <ananda.ravuri@intel.com>
+ * Christopher Rauer <christopher.rauer@intel.com>
+ * Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under this directory for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/vmalloc.h>
+#include <linux/uaccess.h>
+#include <linux/fpga/fpga-mgr.h>
+#include <linux/intel-fpga.h>
+
+#include "feature-dev.h"
+#include "fme.h"
+
+#define PR_WAIT_TIMEOUT 8000000
+
+#define PR_HOST_STATUS_IDLE 0
+
+DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
+
+static ssize_t interface_id_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ u64 intfc_id_l, intfc_id_h;
+ struct feature_fme_pr *fme_pr
+ = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
+
+ intfc_id_l = readq(&fme_pr->intfc_id_l);
+ intfc_id_h = readq(&fme_pr->intfc_id_h);
+
+ return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
+ (unsigned long long)intfc_id_h,
+ (unsigned long long)intfc_id_l);
+}
+static DEVICE_ATTR_RO(interface_id);
+
+static struct attribute *pr_mgmt_attrs[] = {
+ &dev_attr_interface_id.attr,
+ NULL,
+};
+
+struct attribute_group pr_mgmt_attr_group = {
+ .attrs = pr_mgmt_attrs,
+ .name = "pr",
+};
+
+static u64
+pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
+{
+ struct feature_fme_pr_status fme_pr_status;
+ unsigned long err_code;
+ u64 fme_pr_error;
+ int i = 0;
+
+ fme_pr_status.csr = readq(&fme_pr->status);
+ if (!fme_pr_status.pr_status)
+ return 0;
+
+ err_code = fme_pr_error = readq(&fme_pr->error);
+ for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
+ dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
+ writeq(fme_pr_error, &fme_pr->error);
+ return fme_pr_error;
+}
+
+static int fme_pr_write_init(struct fpga_manager *mgr,
+ struct fpga_image_info *info, const char *buf, size_t count)
+{
+ struct fpga_fme *fme = mgr->priv;
+ struct platform_device *pdev;
+ struct feature_fme_pr *fme_pr;
+ struct feature_fme_pr_ctl fme_pr_ctl;
+ struct feature_fme_pr_status fme_pr_status;
+
+ pdev = fme->pdata->dev;
+ fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
+ FME_FEATURE_ID_PR_MGMT);
+ if (!fme_pr)
+ return -EINVAL;
+
+ if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
+ return -EINVAL;
+
+ dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
+
+ fme_pr_ctl.csr = readq(&fme_pr->control);
+ fme_pr_ctl.pr_reset = 1;
+ writeq(fme_pr_ctl.csr, &fme_pr->control);
+
+ fme_pr_ctl.pr_reset_ack = 1;
+
+ if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
+ &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
+ dev_err(&pdev->dev, "maximum PR timeout\n");
+ return -ETIMEDOUT;
+ }
+
+ fme_pr_ctl.csr = readq(&fme_pr->control);
+ fme_pr_ctl.pr_reset = 0;
+ writeq(fme_pr_ctl.csr, &fme_pr->control);
+
+ dev_dbg(&pdev->dev,
+ "waiting for PR resource in HW to be initialized and ready\n");
+
+ fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
+
+ if (fpga_wait_register_field(pr_host_status, fme_pr_status,
+ &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
+ dev_err(&pdev->dev, "maximum PR timeout\n");
+ return -ETIMEDOUT;
+ }
+
+ dev_dbg(&pdev->dev, "check if have any previous PR error\n");
+ pr_err_handle(pdev, fme_pr);
+ return 0;
+}
+
+static int fme_pr_write(struct fpga_manager *mgr,
+ const char *buf, size_t count)
+{
+ struct fpga_fme *fme = mgr->priv;
+ struct platform_device *pdev;
+ struct feature_fme_pr *fme_pr;
+ struct feature_fme_pr_ctl fme_pr_ctl;
+ struct feature_fme_pr_status fme_pr_status;
+ struct feature_fme_pr_data fme_pr_data;
+ int delay, pr_credit, i = 0;
+
+ pdev = fme->pdata->dev;
+ fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
+ FME_FEATURE_ID_PR_MGMT);
+
+ dev_dbg(&pdev->dev, "set PR port ID and start request\n");
+
+ fme_pr_ctl.csr = readq(&fme_pr->control);
+ fme_pr_ctl.pr_regionid = fme->port_id;
+ fme_pr_ctl.pr_start_req = 1;
+ writeq(fme_pr_ctl.csr, &fme_pr->control);
+
+ dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
+
+ fme_pr_status.csr = readq(&fme_pr->status);
+ pr_credit = fme_pr_status.pr_credit;
+
+ while (count > 0) {
+ delay = 0;
+ while (pr_credit <= 1) {
+ if (delay++ > PR_WAIT_TIMEOUT) {
+ dev_err(&pdev->dev, "maximum try\n");
+ return -ETIMEDOUT;
+ }
+ udelay(1);
+
+ fme_pr_status.csr = readq(&fme_pr->status);
+ pr_credit = fme_pr_status.pr_credit;
+ };
+
+ if (count >= 4) {
+ fme_pr_data.rsvd = 0;
+ fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
+ writeq(fme_pr_data.csr, &fme_pr->data);
+ count -= 4;
+ pr_credit--;
+ i++;
+ } else {
+ WARN_ON(1);
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
+static int fme_pr_write_complete(struct fpga_manager *mgr,
+ struct fpga_image_info *info)
+{
+ struct fpga_fme *fme = mgr->priv;
+ struct platform_device *pdev;
+ struct feature_fme_pr *fme_pr;
+ struct feature_fme_pr_ctl fme_pr_ctl;
+
+ pdev = fme->pdata->dev;
+ fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
+ FME_FEATURE_ID_PR_MGMT);
+
+ fme_pr_ctl.csr = readq(&fme_pr->control);
+ fme_pr_ctl.pr_push_complete = 1;
+ writeq(fme_pr_ctl.csr, &fme_pr->control);
+
+ dev_dbg(&pdev->dev, "green bitstream push complete\n");
+ dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
+
+ fme_pr_ctl.pr_start_req = 0;
+
+ if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
+ &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
+ dev_err(&pdev->dev, "maximum try.\n");
+ return -ETIMEDOUT;
+ }
+
+ dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
+ fme->pr_err = pr_err_handle(pdev, fme_pr);
+ if (fme->pr_err)
+ return -EIO;
+
+ dev_dbg(&pdev->dev, "PR done successfully\n");
+ return 0;
+}
+
+static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
+{
+ return FPGA_MGR_STATE_UNKNOWN;
+}
+
+static const struct fpga_manager_ops fme_pr_ops = {
+ .write_init = fme_pr_write_init,
+ .write = fme_pr_write,
+ .write_complete = fme_pr_write_complete,
+ .state = fme_pr_state,
+};
+
+static int fme_pr(struct platform_device *pdev, unsigned long arg)
+{
+ void __user *argp = (void __user *)arg;
+ struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+ struct fpga_fme *fme;
+ struct fpga_manager *mgr;
+ struct feature_fme_header *fme_hdr;
+ struct feature_fme_capability fme_capability;
+ struct fpga_image_info info;
+ struct fpga_fme_port_pr port_pr;
+ struct platform_device *port;
+ unsigned long minsz;
+ void *buf = NULL;
+ int ret = 0;
+
+ minsz = offsetofend(struct fpga_fme_port_pr, status);
+
+ if (copy_from_user(&port_pr, argp, minsz))
+ return -EFAULT;
+
+ if (port_pr.argsz < minsz || port_pr.flags)
+ return -EINVAL;
+
+ if (!IS_ALIGNED(port_pr.buffer_size, 4))
+ return -EINVAL;
+
+ /* get fme header region */
+ fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
+ FME_FEATURE_ID_HEADER);
+ if (WARN_ON(!fme_hdr))
+ return -EINVAL;
+
+ /* check port id */
+ fme_capability.csr = readq(&fme_hdr->capability);
+ if (port_pr.port_id >= fme_capability.num_ports) {
+ dev_dbg(&pdev->dev, "port number more than maximum\n");
+ return -EINVAL;
+ }
+
+ if (!access_ok(VERIFY_READ, port_pr.buffer_address,
+ port_pr.buffer_size))
+ return -EFAULT;
+
+ buf = vmalloc(port_pr.buffer_size);
+ if (!buf)
+ return -ENOMEM;
+
+ if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
+ port_pr.buffer_size)) {
+ ret = -EFAULT;
+ goto free_exit;
+ }
+
+ memset(&info, 0, sizeof(struct fpga_image_info));
+ info.flags = FPGA_MGR_PARTIAL_RECONFIG;
+
+ mgr = fpga_mgr_get(&pdev->dev);
+ if (IS_ERR(mgr)) {
+ ret = PTR_ERR(mgr);
+ goto free_exit;
+ }
+
+ mutex_lock(&pdata->lock);
+ fme = fpga_pdata_get_private(pdata);
+ /* fme device has been unregistered. */
+ if (!fme) {
+ ret = -EINVAL;
+ goto unlock_exit;
+ }
+
+ fme->pr_err = 0;
+ fme->port_id = port_pr.port_id;
+
+ /* Find and get port device by index */
+ port = pdata->fpga_for_each_port(pdev, &fme->port_id,
+ fpga_port_check_id);
+ WARN_ON(!port);
+
+ /* Disable Port before PR */
+ fpga_port_disable(port);
+
+ ret = fpga_mgr_buf_load(mgr, &info, buf, port_pr.buffer_size);
+ port_pr.status = fme->pr_err;
+
+ /* Re-enable Port after PR finished */
+ fpga_port_enable(port);
+
+ put_device(&port->dev);
+
+unlock_exit:
+ mutex_unlock(&pdata->lock);
+ fpga_mgr_put(mgr);
+free_exit:
+ vfree(buf);
+ if (copy_to_user((void __user *)arg, &port_pr, minsz))
+ return -EFAULT;
+ return ret;
+}
+
+static int fpga_fme_pr_probe(struct platform_device *pdev)
+{
+ struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+ struct fpga_fme *priv;
+ int ret;
+
+ mutex_lock(&pdata->lock);
+ priv = fpga_pdata_get_private(pdata);
+ ret = fpga_mgr_register(&pdata->dev->dev,
+ "Intel FPGA Manager", &fme_pr_ops, priv);
+ mutex_unlock(&pdata->lock);
+
+ return ret;
+}
+
+static int fpga_fme_pr_remove(struct platform_device *pdev)
+{
+ fpga_mgr_unregister(&pdev->dev);
+ return 0;
+}
+
+static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
+{
+ int ret;
+
+ ret = fpga_fme_pr_probe(pdev);
+ if (ret)
+ return ret;
+
+ ret = sysfs_create_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
+ if (ret)
+ fpga_fme_pr_remove(pdev);
+
+ return ret;
+}
+
+static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
+{
+ sysfs_remove_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
+ fpga_fme_pr_remove(pdev);
+}
+
+static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
+ unsigned int cmd, unsigned long arg)
+{
+ long ret;
+
+ switch (cmd) {
+ case FPGA_FME_PORT_PR:
+ ret = fme_pr(pdev, arg);
+ break;
+ default:
+ ret = -ENODEV;
+ }
+
+ return ret;
+}
+
+struct feature_ops pr_mgmt_ops = {
+ .init = pr_mgmt_init,
+ .uinit = pr_mgmt_uinit,
+ .ioctl = fme_pr_ioctl,
+};
diff --git a/drivers/fpga/intel/fme.h b/drivers/fpga/intel/fme.h
new file mode 100644
index 0000000..d6cb7ce
--- /dev/null
+++ b/drivers/fpga/intel/fme.h
@@ -0,0 +1,32 @@
+/*
+ * Header file for Intel FPGA Management Engine (FME) Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ * Kang Luwei <luwei.kang@intel.com>
+ * Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ * Joseph Grecco <joe.grecco@intel.com>
+ * Enno Luebbers <enno.luebbers@intel.com>
+ * Tim Whisonant <tim.whisonant@intel.com>
+ * Ananda Ravuri <ananda.ravuri@intel.com>
+ * Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under this directory for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+
+#ifndef __INTEL_FME_H
+#define __INTEL_FME_H
+
+struct fpga_fme {
+ u8 port_id;
+ u64 pr_err;
+ struct feature_platform_data *pdata;
+};
+
+extern struct feature_ops pr_mgmt_ops;
+
+#endif
diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
index 992e556..77658316 100644
--- a/include/uapi/linux/intel-fpga.h
+++ b/include/uapi/linux/intel-fpga.h
@@ -18,6 +18,8 @@
#ifndef _UAPI_LINUX_INTEL_FPGA_H
#define _UAPI_LINUX_INTEL_FPGA_H
+#include <linux/types.h>
+
#define FPGA_API_VERSION 0
/*
@@ -30,6 +32,7 @@
#define FPGA_MAGIC 0xB6
#define FPGA_BASE 0
+#define FME_BASE 0x80
/**
* FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
@@ -49,4 +52,45 @@
#define FPGA_CHECK_EXTENSION _IO(FPGA_MAGIC, FPGA_BASE + 1)
+/* IOCTLs for FME file descriptor */
+
+/**
+ * FPGA_FME_PORT_PR - _IOWR(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
+ *
+ * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
+ * provided by caller.
+ * Return: 0 on success, -errno on failure.
+ * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
+ * some errors during PR, under this case, the user can fetch HW error code
+ * from fpga_fme_port_pr.status. Each bit on the error code is used as the
+ * index for the array created by DEFINE_FPGA_PR_ERR_MSG().
+ * Otherwise, it is always zero.
+ */
+
+#define DEFINE_FPGA_PR_ERR_MSG(_name_) \
+static const char * const _name_[] = { \
+ "PR operation error detected", \
+ "PR CRC error detected", \
+ "PR incompatiable bitstream error detected", \
+ "PR IP protocol error detected", \
+ "PR FIFO overflow error detected", \
+ "Reserved", \
+ "PR secure load error detected", \
+}
+
+#define PR_MAX_ERR_NUM 7
+
+struct fpga_fme_port_pr {
+ /* Input */
+ __u32 argsz; /* Structure length */
+ __u32 flags; /* Zero for now */
+ __u32 port_id;
+ __u32 buffer_size;
+ __u64 buffer_address; /* Userspace address to the buffer for PR */
+ /* Output */
+ __u64 status; /* HW error code if ioctl returns -EIO */
+};
+
+#define FPGA_FME_PORT_PR _IO(FPGA_MAGIC, FME_BASE + 0)
+
#endif /* _UAPI_INTEL_FPGA_H */
--
2.7.4
[toc] | [prev] | [next] | [standalone]
| From | Alan Tull <atull@kernel.org> |
|---|---|
| Date | 2017-03-30 22:40 +0200 |
| Subject | Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support |
| Message-ID | <tqP1n-3lj-7@gated-at.bofh.it> |
| In reply to | #1613019 |
On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote: > From: Kang Luwei <luwei.kang@intel.com> > > Partial Reconfiguration (PR) is the most important function for FME. It > allows reconfiguration for given Port/Accelerated Function Unit (AFU). > > This patch adds support for PR sub feature. In this patch, it registers > a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load > for PR operation once PR request received via ioctl. Below user space > interfaces are exposed by this sub feature. > > Sysfs interface: > * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id > Read-only. Indicate the hardware interface information. Userspace > applications need to check this interface to select correct green > bitstream format before PR. > > Ioctl interface: > * FPGA_FME_PORT_PR > Do partial reconfiguration per information from userspace, including > target port(AFU), buffer size and address info. It returns the PR status > (PR error code if failed) to userspace. > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com> > Signed-off-by: Shiva Rao <shiva.rao@intel.com> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com> > Signed-off-by: Alan Tull <alan.tull@intel.com> Hi Wu Hao, Thanks for submitting your patches. I think there's been a misunderstanding of the meaning of 'Signed-off-by' [1]. I have not signed off on this code or had a hand in its development. But I'm happy to get to review it now. It will take a bit of time; I expect to be replying next week. Alan Tull [1] linux/Documentation/process/5.Posting.rst > Signed-off-by: Kang Luwei <luwei.kang@intel.com> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com> > Signed-off-by: Wu Hao <hao.wu@intel.com>
[toc] | [prev] | [next] | [standalone]
Page 2 of 4 — ← Prev page 1 [2] 3 4 Next page →
Back to top | Article view | linux.kernel
csiph-web