Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1445526 > unrolled thread

[RFC PATCH 00/30] Kernel NET policy

Back to article view | Back to linux.kernel

Contents

  [RFC PATCH 00/30] Kernel NET policy kan.liang@intel.com - 2016-07-18 16:30 +0200
    [RFC PATCH 10/30] net/netpolicy: introduce netpolicy object kan.liang@intel.com - 2016-07-18 16:30 +0200
    [RFC PATCH 20/30] net/netpolicy: introduce per socket netpolicy kan.liang@intel.com - 2016-07-18 16:30 +0200
    [RFC PATCH 28/30] net/netpolicy: optimize for queue pair kan.liang@intel.com - 2016-07-18 16:30 +0200
    [RFC PATCH 29/30] net/netpolicy: limit the total record number kan.liang@intel.com - 2016-07-18 16:30 +0200
    [RFC PATCH 30/30] Documentation/networking: Document net policy kan.liang@intel.com - 2016-07-18 16:30 +0200
      Re: [RFC PATCH 30/30] Documentation/networking: Document net policy Randy Dunlap <rdunlap@infradead.org> - 2016-07-18 19:10 +0200
    [RFC PATCH 24/30] net/netpolicy: set rx queues according to policy kan.liang@intel.com - 2016-07-18 16:30 +0200
    [RFC PATCH 23/30] i40e/ethtool: support RX_CLS_LOC_ANY kan.liang@intel.com - 2016-07-18 16:30 +0200
      Re: [RFC PATCH 23/30] i40e/ethtool: support RX_CLS_LOC_ANY Alexander Duyck <alexander.duyck@gmail.com> - 2016-07-18 18:30 +0200
    [RFC PATCH 14/30] net/netpolicy: add MIX policy kan.liang@intel.com - 2016-07-18 16:30 +0200
    [RFC PATCH 19/30] net/netpolicy: implement netpolicy register kan.liang@intel.com - 2016-07-18 16:30 +0200
    [RFC PATCH 27/30] net/netpolicy: fast path for finding the queues kan.liang@intel.com - 2016-07-18 16:30 +0200
    [RFC PATCH 09/30] net/netpolicy: enable and disable net policy kan.liang@intel.com - 2016-07-18 16:40 +0200
    [RFC PATCH 18/30] net/netpolicy: handle channel changes kan.liang@intel.com - 2016-07-18 16:40 +0200
    [RFC PATCH 08/30] net/netpolicy: set and remove irq affinity kan.liang@intel.com - 2016-07-18 16:40 +0200
    [RFC PATCH 16/30] net/netpolicy: net device hotplug kan.liang@intel.com - 2016-07-18 16:40 +0200
    [RFC PATCH 04/30] net/netpolicy: get driver information kan.liang@intel.com - 2016-07-18 16:40 +0200
    [RFC PATCH 15/30] i40e/netpolicy: add MIX policy support kan.liang@intel.com - 2016-07-18 16:40 +0200
    [RFC PATCH 05/30] i40e/netpolicy: implement ndo_get_irq_info kan.liang@intel.com - 2016-07-18 16:40 +0200
    [RFC PATCH 11/30] net/netpolicy: set net policy by policy name kan.liang@intel.com - 2016-07-18 16:40 +0200
    [RFC PATCH 01/30] net: introduce NET policy kan.liang@intel.com - 2016-07-18 16:40 +0200
    [RFC PATCH 03/30] i40e/netpolicy: Implement ndo_netpolicy_init kan.liang@intel.com - 2016-07-18 16:40 +0200
    [RFC PATCH 17/30] net/netpolicy: support CPU hotplug kan.liang@intel.com - 2016-07-18 16:40 +0200
    [RFC PATCH 02/30] net/netpolicy: init NET policy kan.liang@intel.com - 2016-07-18 16:40 +0200
    [RFC PATCH 06/30] net/netpolicy: get CPU information kan.liang@intel.com - 2016-07-18 16:40 +0200
    [RFC PATCH 07/30] net/netpolicy: create CPU and queue mapping kan.liang@intel.com - 2016-07-18 16:40 +0200
    Re: [RFC PATCH 00/30] Kernel NET policy Florian Westphal <fw@strlen.de> - 2016-07-18 17:20 +0200
      Re: [RFC PATCH 00/30] Kernel NET policy Andi Kleen <andi@firstfloor.org> - 2016-07-18 17:50 +0200
        Re: [RFC PATCH 00/30] Kernel NET policy Cong Wang <xiyou.wangcong@gmail.com> - 2016-07-18 20:00 +0200
          Re: [RFC PATCH 00/30] Kernel NET policy Cong Wang <xiyou.wangcong@gmail.com> - 2016-07-18 22:20 +0200
            RE: [RFC PATCH 00/30] Kernel NET policy "Liang, Kan" <kan.liang@intel.com> - 2016-07-18 22:30 +0200
          RE: [RFC PATCH 00/30] Kernel NET policy "Liang, Kan" <kan.liang@intel.com> - 2016-07-18 22:20 +0200
        Re: [RFC PATCH 00/30] Kernel NET policy Hannes Frederic Sowa <hannes@stressinduktion.org> - 2016-07-18 21:10 +0200
          Re: [RFC PATCH 00/30] Kernel NET policy Andi Kleen <andi@firstfloor.org> - 2016-07-18 21:50 +0200
            Re: [RFC PATCH 00/30] Kernel NET policy Hannes Frederic Sowa <hannes@stressinduktion.org> - 2016-07-19 00:00 +0200
              RE: [RFC PATCH 00/30] Kernel NET policy "Liang, Kan" <kan.liang@intel.com> - 2016-07-19 03:50 +0200
                Re: [RFC PATCH 00/30] Kernel NET policy David Miller <davem@davemloft.net> - 2016-07-19 07:10 +0200
                  RE: [RFC PATCH 00/30] Kernel NET policy "Liang, Kan" <kan.liang@intel.com> - 2016-07-19 15:50 +0200
      RE: [RFC PATCH 00/30] Kernel NET policy "Liang, Kan" <kan.liang@intel.com> - 2016-07-18 18:00 +0200
        Re: [RFC PATCH 00/30] Kernel NET policy Florian Westphal <fw@strlen.de> - 2016-07-18 18:20 +0200
          RE: [RFC PATCH 00/30] Kernel NET policy "Liang, Kan" <kan.liang@intel.com> - 2016-07-18 19:50 +0200
        Re: [RFC PATCH 00/30] Kernel NET policy Tom Herbert <tom@herbertland.com> - 2016-07-18 18:40 +0200
          RE: [RFC PATCH 00/30] Kernel NET policy "Liang, Kan" <kan.liang@intel.com> - 2016-07-18 20:00 +0200
    Re: [RFC PATCH 00/30] Kernel NET policy Daniel Borkmann <daniel@iogearbox.net> - 2016-07-18 18:30 +0200
      RE: [RFC PATCH 00/30] Kernel NET policy "Liang, Kan" <kan.liang@intel.com> - 2016-07-18 20:40 +0200
        Re: [RFC PATCH 00/30] Kernel NET policy Daniel Borkmann <daniel@iogearbox.net> - 2016-07-18 23:00 +0200
    Re: [RFC PATCH 00/30] Kernel NET policy Alexander Duyck <alexander.duyck@gmail.com> - 2016-07-18 19:10 +0200
      RE: [RFC PATCH 00/30] Kernel NET policy "Liang, Kan" <kan.liang@intel.com> - 2016-07-18 21:50 +0200
        Re: [RFC PATCH 00/30] Kernel NET policy Andi Kleen <andi@firstfloor.org> - 2016-07-18 22:00 +0200

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#1445552 — [RFC PATCH 11/30] net/netpolicy: set net policy by policy name

From: Kan Liang <kan.liang@intel.com>

User can write policy name to /proc/net/netpolicy/$DEV/policy to enable
net policy for specific device.

When the policy is enabled, the module automatically disables irq
balance and set irq affinity. The object list is also generated
accordingly.

np_lock will be used to protect the state.

Signed-off-by: Kan Liang <kan.liang@intel.com>
---
 include/linux/netdevice.h |  5 +++
 include/linux/netpolicy.h |  1 +
 net/core/netpolicy.c      | 95 +++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 101 insertions(+)

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index e60c30f..45cb589 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -1094,6 +1094,9 @@ struct tc_to_netdev {
  * int (*ndo_get_irq_info)(struct net_device *dev,
  * 			   struct netpolicy_dev_info *info);
  * 	This function is used to get irq information of rx and tx queues
+ * int (*ndo_set_net_policy)(struct net_device *dev,
+ * 			     enum netpolicy_name name);
+ * 	This function is used to set global net policy by name
  *
  */
 struct net_device_ops {
@@ -1283,6 +1286,8 @@ struct net_device_ops {
 						      struct netpolicy_info *info);
 	int			(*ndo_get_irq_info)(struct net_device *dev,
 						    struct netpolicy_dev_info *info);
+	int			(*ndo_set_net_policy)(struct net_device *dev,
+						      enum netpolicy_name name);
 #endif /* CONFIG_NETPOLICY */
 };
 
diff --git a/include/linux/netpolicy.h b/include/linux/netpolicy.h
index 73a5fa6..b1d9277 100644
--- a/include/linux/netpolicy.h
+++ b/include/linux/netpolicy.h
@@ -27,6 +27,7 @@ enum netpolicy_traffic {
 	NETPOLICY_RXTX,
 };
 
+#define POLICY_NAME_LEN_MAX	64
 extern const char *policy_name[];
 
 struct netpolicy_dev_info {
diff --git a/net/core/netpolicy.c b/net/core/netpolicy.c
index 0f8ff16..8112839 100644
--- a/net/core/netpolicy.c
+++ b/net/core/netpolicy.c
@@ -36,6 +36,7 @@
 #include <linux/netdevice.h>
 #include <net/net_namespace.h>
 #include <linux/sort.h>
+#include <linux/ctype.h>
 
 static int netpolicy_get_dev_info(struct net_device *dev,
 				  struct netpolicy_dev_info *d_info)
@@ -430,6 +431,69 @@ err:
 	return ret;
 }
 
+static int net_policy_set_by_name(char *name, struct net_device *dev)
+{
+	int i, ret;
+
+	spin_lock(&dev->np_lock);
+	ret = 0;
+
+	if (!dev->netpolicy ||
+	    !dev->netdev_ops->ndo_set_net_policy) {
+		ret = -ENOTSUPP;
+		goto unlock;
+	}
+
+	for (i = 0; i < NET_POLICY_MAX; i++) {
+		if (!strncmp(name, policy_name[i], strlen(policy_name[i])))
+		break;
+	}
+
+	if (!test_bit(i, dev->netpolicy->avail_policy)) {
+		ret = -ENOTSUPP;
+		goto unlock;
+	}
+
+	if (i == dev->netpolicy->cur_policy)
+		goto unlock;
+
+	/* If there is no policy applied yet, need to do enable first . */
+	if (dev->netpolicy->cur_policy == NET_POLICY_NONE) {
+		ret = netpolicy_enable(dev);
+		if (ret)
+			goto unlock;
+	}
+
+	netpolicy_free_obj_list(dev);
+
+	/* Generate object list according to policy name */
+	ret = netpolicy_gen_obj_list(dev, i);
+	if (ret)
+		goto err;
+
+	/* set policy */
+	ret = dev->netdev_ops->ndo_set_net_policy(dev, i);
+	if (ret)
+		goto err;
+
+	/* If removing policy, need to do disable. */
+	if (i == NET_POLICY_NONE)
+		netpolicy_disable(dev);
+
+	dev->netpolicy->cur_policy = i;
+
+	spin_unlock(&dev->np_lock);
+	return 0;
+
+err:
+	netpolicy_free_obj_list(dev);
+	if (dev->netpolicy->cur_policy == NET_POLICY_NONE)
+		netpolicy_disable(dev);
+unlock:
+	spin_unlock(&dev->np_lock);
+	return ret;
+}
+
 #ifdef CONFIG_PROC_FS
 
 static int net_policy_proc_show(struct seq_file *m, void *v)
@@ -459,11 +523,40 @@ static int net_policy_proc_open(struct inode *inode, struct file *file)
 	return single_open(file, net_policy_proc_show, PDE_DATA(inode));
 }
 
+static ssize_t net_policy_proc_write(struct file *file, const char __user *buf,
+				     size_t count, loff_t *pos)
+{
+	struct seq_file *m = file->private_data;
+	struct net_device *dev = (struct net_device *)m->private;
+	char name[POLICY_NAME_LEN_MAX];
+	int i, ret;
+
+	if (!dev->netpolicy)
+		return -ENOTSUPP;
+
+	if (count > POLICY_NAME_LEN_MAX)
+		return -EINVAL;
+
+	if (copy_from_user(name, buf, count))
+		return -EINVAL;
+
+	for (i = 0; i < count - 1; i++)
+		name[i] = toupper(name[i]);
+	name[POLICY_NAME_LEN_MAX - 1] = 0;
+
+	ret = net_policy_set_by_name(name, dev);
+	if (ret)
+		return ret;
+
+	return count;
+}
+
 static const struct file_operations proc_net_policy_operations = {
 	.open		= net_policy_proc_open,
 	.read		= seq_read,
 	.llseek		= seq_lseek,
 	.release	= seq_release,
+	.write		= net_policy_proc_write,
 	.owner		= THIS_MODULE,
 };
 
@@ -527,6 +620,8 @@ void uninit_netpolicy(struct net_device *dev)
 {
 	spin_lock(&dev->np_lock);
 	if (dev->netpolicy) {
+		if (dev->netpolicy->cur_policy > NET_POLICY_NONE)
+			netpolicy_disable(dev);
 		kfree(dev->netpolicy);
 		dev->netpolicy = NULL;
 	}
-- 
2.5.5

[toc] | [prev] | [next] | [standalone]

#1445555 — [RFC PATCH 01/30] net: introduce NET policy

From: Kan Liang <kan.liang@intel.com>

This patch introduce NET policy subsystem. If proc is supported in the
system, it creates netpolicy node in proc system.

Signed-off-by: Kan Liang <kan.liang@intel.com>
---
 include/linux/netdevice.h   |   7 +++
 include/net/net_namespace.h |   3 ++
 net/Kconfig                 |   7 +++
 net/core/Makefile           |   1 +
 net/core/netpolicy.c        | 128 ++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 146 insertions(+)
 create mode 100644 net/core/netpolicy.c

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 49736a3..9e30a31 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -1584,6 +1584,8 @@ enum netdev_priv_flags {
  *			switch driver and used to set the phys state of the
  *			switch port.
  *
+ *	@proc_dev:	device node in proc to configure device net policy
+ *
  *	FIXME: cleanup struct net_device such that network protocol info
  *	moves out.
  */
@@ -1851,6 +1853,11 @@ struct net_device {
 	struct lock_class_key	*qdisc_tx_busylock;
 	struct lock_class_key	*qdisc_running_key;
 	bool			proto_down;
+#ifdef CONFIG_NETPOLICY
+#ifdef CONFIG_PROC_FS
+	struct proc_dir_entry	*proc_dev;
+#endif /* CONFIG_PROC_FS */
+#endif /* CONFIG_NETPOLICY */
 };
 #define to_net_dev(d) container_of(d, struct net_device, dev)
 
diff --git a/include/net/net_namespace.h b/include/net/net_namespace.h
index 4089abc..d2ff6c4 100644
--- a/include/net/net_namespace.h
+++ b/include/net/net_namespace.h
@@ -142,6 +142,9 @@ struct net {
 #endif
 	struct sock		*diag_nlsk;
 	atomic_t		fnhe_genid;
+#ifdef CONFIG_NETPOLICY
+	struct proc_dir_entry	*proc_netpolicy;
+#endif /* CONFIG_NETPOLICY */
 };
 
 #include <linux/seq_file_net.h>
diff --git a/net/Kconfig b/net/Kconfig
index ff40562..c3ed726 100644
--- a/net/Kconfig
+++ b/net/Kconfig
@@ -205,6 +205,13 @@ source "net/bridge/netfilter/Kconfig"
 
 endif
 
+config NETPOLICY
+	depends on NET
+	bool "Net policy support"
+	default y
+	---help---
+	Net policy support
+
 source "net/dccp/Kconfig"
 source "net/sctp/Kconfig"
 source "net/rds/Kconfig"
diff --git a/net/core/Makefile b/net/core/Makefile
index d6508c2..0be7092 100644
--- a/net/core/Makefile
+++ b/net/core/Makefile
@@ -27,3 +27,4 @@ obj-$(CONFIG_LWTUNNEL) += lwtunnel.o
 obj-$(CONFIG_DST_CACHE) += dst_cache.o
 obj-$(CONFIG_HWBM) += hwbm.o
 obj-$(CONFIG_NET_DEVLINK) += devlink.o
+obj-$(CONFIG_NETPOLICY) += netpolicy.o
diff --git a/net/core/netpolicy.c b/net/core/netpolicy.c
new file mode 100644
index 0000000..faabfe7
--- /dev/null
+++ b/net/core/netpolicy.c
@@ -0,0 +1,128 @@
+/*
+ * netpolicy.c: Net policy support
+ * Copyright (c) 2016, Intel Corporation.
+ * Author: Kan Liang (kan.liang@intel.com)
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * NET policy intends to simplify the network configuration and get a good
+ * network performance according to the hints(policy) which is applied by user.
+ *
+ * Motivation
+ * 	- The network performance is not good with default system settings.
+ *	- It is too difficult to do automatic tuning for all possible
+ *	  workloads, since workloads have different requirements. Some
+ *	  workloads may want high throughput. Some may need low latency.
+ *	- There are lots of manual configurations. Fine grained configuration
+ *	  is too difficult for users.
+ * 	So, it is a big challenge to get good network performance.
+ *
+ */
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/errno.h>
+#include <linux/init.h>
+#include <linux/seq_file.h>
+#include <linux/proc_fs.h>
+#include <linux/uaccess.h>
+#include <linux/netdevice.h>
+#include <net/net_namespace.h>
+
+#ifdef CONFIG_PROC_FS
+
+static int net_policy_proc_show(struct seq_file *m, void *v)
+{
+	struct net_device *dev = (struct net_device *)m->private;
+
+	seq_printf(m, "%s doesn't support net policy manager\n", dev->name);
+
+	return 0;
+}
+
+static int net_policy_proc_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, net_policy_proc_show, PDE_DATA(inode));
+}
+
+static const struct file_operations proc_net_policy_operations = {
+	.open		= net_policy_proc_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= seq_release,
+	.owner		= THIS_MODULE,
+};
+
+static int netpolicy_proc_dev_init(struct net *net, struct net_device *dev)
+{
+	dev->proc_dev = proc_net_mkdir(net, dev->name, net->proc_netpolicy);
+	if (!dev->proc_dev)
+		return -ENOMEM;
+
+	if (!proc_create_data("policy", S_IWUSR | S_IRUGO,
+			      dev->proc_dev, &proc_net_policy_operations,
+			      (void *)dev)) {
+		remove_proc_subtree(dev->name, net->proc_netpolicy);
+		return -ENOMEM;
+	}
+	return 0;
+}
+
+static int __net_init netpolicy_net_init(struct net *net)
+{
+	struct net_device *dev, *aux;
+
+	net->proc_netpolicy = proc_net_mkdir(net, "netpolicy",
+					     net->proc_net);
+	if (!net->proc_netpolicy)
+		return -ENOMEM;
+
+	for_each_netdev_safe(net, dev, aux) {
+		netpolicy_proc_dev_init(net, dev);
+	}
+
+	return 0;
+}
+
+#else /* CONFIG_PROC_FS */
+
+static int __net_init netpolicy_net_init(struct net *net)
+{
+	return 0;
+}
+#endif /* CONFIG_PROC_FS */
+
+static void __net_exit netpolicy_net_exit(struct net *net)
+{
+#ifdef CONFIG_PROC_FS
+	remove_proc_subtree("netpolicy", net->proc_net);
+#endif /* CONFIG_PROC_FS */
+}
+
+static struct pernet_operations netpolicy_net_ops = {
+	.init = netpolicy_net_init,
+	.exit = netpolicy_net_exit,
+};
+
+static int __init netpolicy_init(void)
+{
+	int ret;
+
+	ret = register_pernet_subsys(&netpolicy_net_ops);
+
+	return ret;
+}
+
+static void __exit netpolicy_exit(void)
+{
+	unregister_pernet_subsys(&netpolicy_net_ops);
+}
+
+subsys_initcall(netpolicy_init);
+module_exit(netpolicy_exit);
-- 
2.5.5

[toc] | [prev] | [next] | [standalone]

#1445556 — [RFC PATCH 03/30] i40e/netpolicy: Implement ndo_netpolicy_init

From: Kan Liang <kan.liang@intel.com>

Support ndo_netpolicy_init in i40e driver.
For i40e driver, there is no extra initialization work to do. It only
needs to update the available policy bitmap.
policy_param will be filled according to different policies later.

Signed-off-by: Kan Liang <kan.liang@intel.com>
---
 drivers/net/ethernet/intel/i40e/i40e_main.c | 41 +++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c
index 2b11405..ee1f0b2 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_main.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_main.c
@@ -8960,6 +8960,44 @@ static netdev_features_t i40e_features_check(struct sk_buff *skb,
 	return features;
 }
 
+#ifdef CONFIG_NETPOLICY
+
+#define NET_POLICY_NOT_SUPPORT	-2
+#define NET_POLICY_END		-3
+static int policy_param[NET_POLICY_MAX + 1][2] = {
+	/* rx-usec, tx-usec */
+	{0, 0},
+
+	{NET_POLICY_END, NET_POLICY_END},
+};
+
+/**
+ * i40e_ndo_netpolicy_init
+ * @dev: the net device pointer
+ * @info: netpolicy info which need to be updated
+ *
+ * Init and update available policy on i40e driver
+ * Returns 0 on success, negative on failure
+ */
+static int i40e_ndo_netpolicy_init(struct net_device *dev,
+				   struct netpolicy_info *info)
+{
+	int i;
+
+	for (i = 0; i < NET_POLICY_MAX; i++) {
+		if ((policy_param[i][0] == NET_POLICY_END) &&
+		    (policy_param[i][1] == NET_POLICY_END))
+			break;
+
+		if ((policy_param[i][0] != NET_POLICY_NOT_SUPPORT) &&
+		    (policy_param[i][1] != NET_POLICY_NOT_SUPPORT))
+			set_bit(i, info->avail_policy);
+	}
+
+	return 0;
+}
+#endif /* CONFIG_NETPOLICY */
+
 static const struct net_device_ops i40e_netdev_ops = {
 	.ndo_open		= i40e_open,
 	.ndo_stop		= i40e_close,
@@ -8996,6 +9034,9 @@ static const struct net_device_ops i40e_netdev_ops = {
 	.ndo_features_check	= i40e_features_check,
 	.ndo_bridge_getlink	= i40e_ndo_bridge_getlink,
 	.ndo_bridge_setlink	= i40e_ndo_bridge_setlink,
+#ifdef CONFIG_NETPOLICY
+	.ndo_netpolicy_init	= i40e_ndo_netpolicy_init,
+#endif /* CONFIG_NETPOLICY */
 };
 
 /**
-- 
2.5.5

[toc] | [prev] | [next] | [standalone]

#1445557 — [RFC PATCH 17/30] net/netpolicy: support CPU hotplug

From: Kan Liang <kan.liang@intel.com>

For CPU hotplug, the net policy module will rebuild the sys map and
object list.

Signed-off-by: Kan Liang <kan.liang@intel.com>
---
 net/core/netpolicy.c | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)

diff --git a/net/core/netpolicy.c b/net/core/netpolicy.c
index 2a04fcf..46af407 100644
--- a/net/core/netpolicy.c
+++ b/net/core/netpolicy.c
@@ -37,6 +37,7 @@
 #include <net/net_namespace.h>
 #include <linux/sort.h>
 #include <linux/ctype.h>
+#include <linux/cpu.h>
 
 static int netpolicy_get_dev_info(struct net_device *dev,
 				  struct netpolicy_dev_info *d_info)
@@ -838,6 +839,75 @@ static struct notifier_block netpolicy_dev_notf = {
 	.notifier_call = netpolicy_notify,
 };
 
+/**
+ * update_netpolicy_sys_map() - rebuild the sys map and object list
+ *
+ * This function go through all the available net policy supported device,
+ * and rebuild sys map and object list.
+ *
+ */
+void update_netpolicy_sys_map(void)
+{
+	struct net *net;
+	struct net_device *dev, *aux;
+	enum netpolicy_name cur_policy;
+
+	for_each_net(net) {
+		for_each_netdev_safe(net, dev, aux) {
+			spin_lock(&dev->np_lock);
+			if (!dev->netpolicy)
+				goto unlock;
+			cur_policy = dev->netpolicy->cur_policy;
+			if (cur_policy == NET_POLICY_NONE)
+				goto unlock;
+
+			dev->netpolicy->cur_policy = NET_POLICY_NONE;
+
+			/* rebuild everything */
+			netpolicy_disable(dev);
+			netpolicy_enable(dev);
+			if (netpolicy_gen_obj_list(dev, cur_policy)) {
+				pr_warn("NETPOLICY: Failed to generate "
+					"netpolicy object list for dev %s\n",
+					dev->name);
+				netpolicy_disable(dev);
+				goto unlock;
+			}
+			if (dev->netdev_ops->ndo_set_net_policy(dev, cur_policy)) {
+				pr_warn("NETPOLICY: Failed to set "
+					"netpolicy for dev %s\n",
+					dev->name);
+				netpolicy_disable(dev);
+				goto unlock;
+			}
+
+			dev->netpolicy->cur_policy = cur_policy;
+unlock:
+			spin_unlock(&dev->np_lock);
+		}
+	}
+}
+
+static int netpolicy_cpu_callback(struct notifier_block *nfb,
+				  unsigned long action, void *hcpu)
+{
+	switch (action & ~CPU_TASKS_FROZEN) {
+	case CPU_ONLINE:
+		update_netpolicy_sys_map();
+		break;
+	case CPU_DYING:
+		update_netpolicy_sys_map();
+		break;
+	}
+	return NOTIFY_OK;
+}
+
+static struct notifier_block netpolicy_cpu_notifier = {
+	&netpolicy_cpu_callback,
+	NULL,
+	0
+};
+
 static int __init netpolicy_init(void)
 {
 	int ret;
@@ -846,6 +916,10 @@ static int __init netpolicy_init(void)
 	if (!ret)
 		register_netdevice_notifier(&netpolicy_dev_notf);
 
+	cpu_notifier_register_begin();
+	__register_cpu_notifier(&netpolicy_cpu_notifier);
+	cpu_notifier_register_done();
+
 	return ret;
 }
 
@@ -853,6 +927,10 @@ static void __exit netpolicy_exit(void)
 {
 	unregister_netdevice_notifier(&netpolicy_dev_notf);
 	unregister_pernet_subsys(&netpolicy_net_ops);
+
+	cpu_notifier_register_begin();
+	__unregister_cpu_notifier(&netpolicy_cpu_notifier);
+	cpu_notifier_register_done();
 }
 
 subsys_initcall(netpolicy_init);
-- 
2.5.5

[toc] | [prev] | [next] | [standalone]

#1445560 — [RFC PATCH 02/30] net/netpolicy: init NET policy

From: Kan Liang <kan.liang@intel.com>

This patch tries to initialize NET policy for all the devices in the
system. However, not all device drivers have NET policy support. For
those drivers who does not have NET policy support, the node will not be
showed in /proc/net/netpolicy/.
The device driver who has NET policy support must implement the
interface ndo_netpolicy_init, which is used to do necessory
initialization and collect information (E.g. supported policies) from
driver.
The user can check /proc/netpolicy/ and /proc/net/netpolicy/$DEV/policy
to know the available device and its supported policy.
np_lock is also introduced to protect the state of NET policy.
Device hotplug will be handled later in this series.

Signed-off-by: Kan Liang <kan.liang@intel.com>
---
 include/linux/netdevice.h | 12 +++++++
 include/linux/netpolicy.h | 31 +++++++++++++++++
 net/core/netpolicy.c      | 86 +++++++++++++++++++++++++++++++++++++++++------
 3 files changed, 118 insertions(+), 11 deletions(-)
 create mode 100644 include/linux/netpolicy.h

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 9e30a31..ef45dfe 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -52,6 +52,7 @@
 #include <uapi/linux/netdevice.h>
 #include <uapi/linux/if_bonding.h>
 #include <uapi/linux/pkt_cls.h>
+#include <linux/netpolicy.h>
 
 struct netpoll_info;
 struct device;
@@ -1087,6 +1088,9 @@ struct tc_to_netdev {
  *	appropriate rx headroom value allows avoiding skb head copy on
  *	forward. Setting a negative value resets the rx headroom to the
  *	default value.
+ * int (*ndo_netpolicy_init)(struct net_device *dev,
+ * 			     struct netpolicy_info *info);
+ * 	This function is used to init and get supported policy.
  *
  */
 struct net_device_ops {
@@ -1271,6 +1275,10 @@ struct net_device_ops {
 						       struct sk_buff *skb);
 	void			(*ndo_set_rx_headroom)(struct net_device *dev,
 						       int needed_headroom);
+#ifdef CONFIG_NETPOLICY
+	int			(*ndo_netpolicy_init)(struct net_device *dev,
+						      struct netpolicy_info *info);
+#endif /* CONFIG_NETPOLICY */
 };
 
 /**
@@ -1585,6 +1593,8 @@ enum netdev_priv_flags {
  *			switch port.
  *
  *	@proc_dev:	device node in proc to configure device net policy
+ *	@netpolicy:	NET policy related information of net device
+ *	@np_lock:	protect the state of NET policy
  *
  *	FIXME: cleanup struct net_device such that network protocol info
  *	moves out.
@@ -1857,6 +1867,8 @@ struct net_device {
 #ifdef CONFIG_PROC_FS
 	struct proc_dir_entry	*proc_dev;
 #endif /* CONFIG_PROC_FS */
+	struct netpolicy_info	*netpolicy;
+	spinlock_t		np_lock;
 #endif /* CONFIG_NETPOLICY */
 };
 #define to_net_dev(d) container_of(d, struct net_device, dev)
diff --git a/include/linux/netpolicy.h b/include/linux/netpolicy.h
new file mode 100644
index 0000000..ca1f131
--- /dev/null
+++ b/include/linux/netpolicy.h
@@ -0,0 +1,31 @@
+/*
+ * netpolicy.h: Net policy support
+ * Copyright (c) 2016, Intel Corporation.
+ * Author: Kan Liang (kan.liang@intel.com)
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ */
+#ifndef __LINUX_NETPOLICY_H
+#define __LINUX_NETPOLICY_H
+
+enum netpolicy_name {
+	NET_POLICY_NONE		= 0,
+	NET_POLICY_MAX,
+};
+
+extern const char *policy_name[];
+
+struct netpolicy_info {
+	enum netpolicy_name	cur_policy;
+	unsigned long avail_policy[BITS_TO_LONGS(NET_POLICY_MAX)];
+};
+
+#endif /*__LINUX_NETPOLICY_H*/
diff --git a/net/core/netpolicy.c b/net/core/netpolicy.c
index faabfe7..5f304d5 100644
--- a/net/core/netpolicy.c
+++ b/net/core/netpolicy.c
@@ -35,13 +35,29 @@
 #include <linux/netdevice.h>
 #include <net/net_namespace.h>
 
+const char *policy_name[NET_POLICY_MAX] = {
+	"NONE"
+};
 #ifdef CONFIG_PROC_FS
 
 static int net_policy_proc_show(struct seq_file *m, void *v)
 {
 	struct net_device *dev = (struct net_device *)m->private;
-
-	seq_printf(m, "%s doesn't support net policy manager\n", dev->name);
+	int i;
+
+	if (WARN_ON(!dev->netpolicy))
+		return -EINVAL;
+
+	if (dev->netpolicy->cur_policy == NET_POLICY_NONE) {
+		seq_printf(m, "%s: There is no policy applied\n", dev->name);
+		seq_printf(m, "%s: The available policy include:", dev->name);
+		for_each_set_bit(i, dev->netpolicy->avail_policy, NET_POLICY_MAX)
+			seq_printf(m, " %s", policy_name[i]);
+		seq_printf(m, "\n");
+	} else {
+		seq_printf(m, "%s: POLICY %s is running on the system\n",
+			   dev->name, policy_name[dev->netpolicy->cur_policy]);
+	}
 
 	return 0;
 }
@@ -73,33 +89,81 @@ static int netpolicy_proc_dev_init(struct net *net, struct net_device *dev)
 	}
 	return 0;
 }
+#endif /* CONFIG_PROC_FS */
+
+int init_netpolicy(struct net_device *dev)
+{
+	int ret;
+
+	spin_lock(&dev->np_lock);
+	ret = 0;
+
+	if (!dev->netdev_ops->ndo_netpolicy_init) {
+		ret = -ENOTSUPP;
+		goto unlock;
+	}
+
+	if (dev->netpolicy)
+		goto unlock;
+
+	dev->netpolicy = kzalloc(sizeof(*dev->netpolicy), GFP_ATOMIC);
+	if (!dev->netpolicy) {
+		ret = -ENOMEM;
+		goto unlock;
+	}
+
+	ret = dev->netdev_ops->ndo_netpolicy_init(dev, dev->netpolicy);
+	if (ret) {
+		kfree(dev->netpolicy);
+		dev->netpolicy = NULL;
+	}
+
+unlock:
+	spin_unlock(&dev->np_lock);
+	return ret;
+}
+
+void uninit_netpolicy(struct net_device *dev)
+{
+	spin_lock(&dev->np_lock);
+	if (dev->netpolicy) {
+		kfree(dev->netpolicy);
+		dev->netpolicy = NULL;
+	}
+	spin_unlock(&dev->np_lock);
+}
 
 static int __net_init netpolicy_net_init(struct net *net)
 {
 	struct net_device *dev, *aux;
 
+#ifdef CONFIG_PROC_FS
 	net->proc_netpolicy = proc_net_mkdir(net, "netpolicy",
 					     net->proc_net);
 	if (!net->proc_netpolicy)
 		return -ENOMEM;
+#endif /* CONFIG_PROC_FS */
 
 	for_each_netdev_safe(net, dev, aux) {
-		netpolicy_proc_dev_init(net, dev);
+		if (!init_netpolicy(dev)) {
+#ifdef CONFIG_PROC_FS
+			if (netpolicy_proc_dev_init(net, dev))
+				uninit_netpolicy(dev);
+			else
+#endif /* CONFIG_PROC_FS */
+			pr_info("NETPOLICY: Init net policy for %s\n", dev->name);
+		}
 	}
 
 	return 0;
 }
 
-#else /* CONFIG_PROC_FS */
-
-static int __net_init netpolicy_net_init(struct net *net)
-{
-	return 0;
-}
-#endif /* CONFIG_PROC_FS */
-
 static void __net_exit netpolicy_net_exit(struct net *net)
 {
+	struct net_device *dev, *aux;
+
+	for_each_netdev_safe(net, dev, aux)
+		uninit_netpolicy(dev);
 #ifdef CONFIG_PROC_FS
 	remove_proc_subtree("netpolicy", net->proc_net);
 #endif /* CONFIG_PROC_FS */
-- 
2.5.5

[toc] | [prev] | [next] | [standalone]

#1445561 — [RFC PATCH 06/30] net/netpolicy: get CPU information

From: Kan Liang <kan.liang@intel.com>

Net policy also needs to know CPU information. Currently, online
CPU number is enough.

Signed-off-by: Kan Liang <kan.liang@intel.com>
---
 net/core/netpolicy.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/net/core/netpolicy.c b/net/core/netpolicy.c
index 7c34c8a..075aaca 100644
--- a/net/core/netpolicy.c
+++ b/net/core/netpolicy.c
@@ -49,6 +49,11 @@ static void netpolicy_free_dev_info(struct netpolicy_dev_info *d_info)
 	kfree(d_info->tx_irq);
 }
 
+static u32 netpolicy_get_cpu_information(void)
+{
+	return num_online_cpus();
+}
+
 const char *policy_name[NET_POLICY_MAX] = {
 	"NONE"
 };
-- 
2.5.5

[toc] | [prev] | [next] | [standalone]

#1445562 — [RFC PATCH 07/30] net/netpolicy: create CPU and queue mapping

From: Kan Liang <kan.liang@intel.com>

Current implementation forces CPU and queue 1:1 mapping. This patch
introduces the function netpolicy_update_sys_map to create this mapping.
The result is stored in netpolicy_sys_info.
If the CPU count and queue count are different, the remaining
CPUs/queues are not used.
CPU hotplug, device hotplug or ethtool may change the CPU count or
queue count. For these cases, this function can also be called to
reconstruct the mapping. These cases will be handled later in this
series.

Signed-off-by: Kan Liang <kan.liang@intel.com>
---
 include/linux/netpolicy.h | 18 ++++++++++++
 net/core/netpolicy.c      | 74 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 92 insertions(+)

diff --git a/include/linux/netpolicy.h b/include/linux/netpolicy.h
index fc87d9b..a946b75c 100644
--- a/include/linux/netpolicy.h
+++ b/include/linux/netpolicy.h
@@ -30,9 +30,27 @@ struct netpolicy_dev_info {
 	u32	*tx_irq;
 };
 
+struct netpolicy_sys_map {
+	u32	cpu;
+	u32	queue;
+	u32	irq;
+};
+
+struct netpolicy_sys_info {
+	/*
+	 * Record the cpu and queue 1:1 mapping
+	 */
+	u32				avail_rx_num;
+	struct netpolicy_sys_map	*rx;
+	u32				avail_tx_num;
+	struct netpolicy_sys_map	*tx;
+};
+
 struct netpolicy_info {
 	enum netpolicy_name	cur_policy;
 	unsigned long avail_policy[BITS_TO_LONGS(NET_POLICY_MAX)];
+	/* cpu and queue mapping information */
+	struct netpolicy_sys_info	sys_info;
 };
 
 #endif /*__LINUX_NETPOLICY_H*/
diff --git a/net/core/netpolicy.c b/net/core/netpolicy.c
index 075aaca..ff7fc04 100644
--- a/net/core/netpolicy.c
+++ b/net/core/netpolicy.c
@@ -54,6 +54,80 @@ static u32 netpolicy_get_cpu_information(void)
 	return num_online_cpus();
 }
 
+static void netpolicy_free_sys_map(struct net_device *dev)
+{
+	struct netpolicy_sys_info *s_info = &dev->netpolicy->sys_info;
+
+	kfree(s_info->rx);
+	s_info->rx = NULL;
+	s_info->avail_rx_num = 0;
+	kfree(s_info->tx);
+	s_info->tx = NULL;
+	s_info->avail_tx_num = 0;
+}
+
+static int netpolicy_update_sys_map(struct net_device *dev,
+				    struct netpolicy_dev_info *d_info,
+				    u32 cpu)
+{
+	struct netpolicy_sys_info *s_info = &dev->netpolicy->sys_info;
+	u32 num, i, online_cpu;
+	cpumask_var_t cpumask;
+
+	if (!alloc_cpumask_var(&cpumask, GFP_ATOMIC))
+		return -ENOMEM;
+
+	/* update rx cpu map */
+	if (cpu > d_info->rx_num)
+		num = d_info->rx_num;
+	else
+		num = cpu;
+
+	s_info->avail_rx_num = num;
+	s_info->rx = kcalloc(num, sizeof(*s_info->rx), GFP_ATOMIC);
+	if (!s_info->rx)
+		goto err;
+	cpumask_copy(cpumask, cpu_online_mask);
+
+	i = 0;
+	for_each_cpu(online_cpu, cpumask) {
+		if (i == num)
+			break;
+		s_info->rx[i].cpu = online_cpu;
+		s_info->rx[i].queue = i;
+		s_info->rx[i].irq = d_info->rx_irq[i];
+		i++;
+	}
+
+	/* update tx cpu map */
+	if (cpu >= d_info->tx_num)
+		num = d_info->tx_num;
+	else
+		num = cpu;
+
+	s_info->avail_tx_num = num;
+	s_info->tx = kcalloc(num, sizeof(*s_info->tx), GFP_ATOMIC);
+	if (!s_info->tx)
+		goto err;
+
+	i = 0;
+	for_each_cpu(online_cpu, cpumask) {
+		if (i == num)
+			break;
+		s_info->tx[i].cpu = online_cpu;
+		s_info->tx[i].queue = i;
+		s_info->tx[i].irq = d_info->tx_irq[i];
+		i++;
+	}
+
+	free_cpumask_var(cpumask);
+	return 0;
+err:
+	netpolicy_free_sys_map(dev);
+	free_cpumask_var(cpumask);
+	return -ENOMEM;
+}
+
 const char *policy_name[NET_POLICY_MAX] = {
 	"NONE"
 };
-- 
2.5.5

[toc] | [prev] | [next] | [standalone]

#1445585

kan.liang@intel.com <kan.liang@intel.com> wrote:
> From: Kan Liang <kan.liang@intel.com>
> 
> It is a big challenge to get good network performance. First, the network
> performance is not good with default system settings. Second, it is too
> difficult to do automatic tuning for all possible workloads, since workloads
> have different requirements. Some workloads may want high throughput.

Seems you did lots of tests to find optimal settings for a given base
policy.

What is missing in the kernel UAPI so userspace could do these settings
on its own, without adding this policy stuff to the kernel?

It seems strange to me to add such policies to the kernel.
Addmittingly, documentation of some settings is non-existent and one needs
various different tools to set this (sysctl, procfs, sysfs, ethtool, etc).

But all of these details could be hidden from user.
Have you looked at tuna for instance?

[toc] | [prev] | [next] | [standalone]

#1445599

> It seems strange to me to add such policies to the kernel.
> Addmittingly, documentation of some settings is non-existent and one needs
> various different tools to set this (sysctl, procfs, sysfs, ethtool, etc).

The problem is that different applications need different policies.

The only entity which can efficiently negotiate between different
applications' conflicting requests is the kernel. And that is pretty 
much the basic job description of a kernel: multiplex hardware
efficiently between different users.

So yes the user space tuning approach works for simple cases
("only run workloads that require the same tuning"), but is ultimately not
very interesting nor scalable.

-Andi

[toc] | [prev] | [next] | [standalone]

#1445677

On Mon, Jul 18, 2016 at 8:45 AM, Andi Kleen <andi@firstfloor.org> wrote:
>> It seems strange to me to add such policies to the kernel.
>> Addmittingly, documentation of some settings is non-existent and one needs
>> various different tools to set this (sysctl, procfs, sysfs, ethtool, etc).
>
> The problem is that different applications need different policies.
>
> The only entity which can efficiently negotiate between different
> applications' conflicting requests is the kernel. And that is pretty
> much the basic job description of a kernel: multiplex hardware
> efficiently between different users.
>
> So yes the user space tuning approach works for simple cases
> ("only run workloads that require the same tuning"), but is ultimately not
> very interesting nor scalable.

I don't read the code yet, just the cover letter.

We have global tunings, per-network-namespace tunings, per-socket
tunings. It is still unclear why you can't just put different applications
into different namespaces/containers to get different policies.

[toc] | [prev] | [next] | [standalone]

#1445812

On Mon, Jul 18, 2016 at 1:14 PM, Liang, Kan <kan.liang@intel.com> wrote:
>
>
>>
>> On Mon, Jul 18, 2016 at 8:45 AM, Andi Kleen <andi@firstfloor.org> wrote:
>> >> It seems strange to me to add such policies to the kernel.
>> >> Addmittingly, documentation of some settings is non-existent and one
>> >> needs various different tools to set this (sysctl, procfs, sysfs, ethtool, etc).
>> >
>> > The problem is that different applications need different policies.
>> >
>> > The only entity which can efficiently negotiate between different
>> > applications' conflicting requests is the kernel. And that is pretty
>> > much the basic job description of a kernel: multiplex hardware
>> > efficiently between different users.
>> >
>> > So yes the user space tuning approach works for simple cases ("only
>> > run workloads that require the same tuning"), but is ultimately not
>> > very interesting nor scalable.
>>
>> I don't read the code yet, just the cover letter.
>>
>> We have global tunings, per-network-namespace tunings, per-socket tunings.
>> It is still unclear why you can't just put different applications into different
>> namespaces/containers to get different policies.
>
> In NET policy, we do per queue tunings.

Is it possible to isolate NIC queues for containers?

[toc] | [prev] | [next] | [standalone]

#1445823

> >>
> >> On Mon, Jul 18, 2016 at 8:45 AM, Andi Kleen <andi@firstfloor.org> wrote:
> >> >> It seems strange to me to add such policies to the kernel.
> >> >> Addmittingly, documentation of some settings is non-existent and
> >> >> one needs various different tools to set this (sysctl, procfs, sysfs,
> ethtool, etc).
> >> >
> >> > The problem is that different applications need different policies.
> >> >
> >> > The only entity which can efficiently negotiate between different
> >> > applications' conflicting requests is the kernel. And that is
> >> > pretty much the basic job description of a kernel: multiplex
> >> > hardware efficiently between different users.
> >> >
> >> > So yes the user space tuning approach works for simple cases ("only
> >> > run workloads that require the same tuning"), but is ultimately not
> >> > very interesting nor scalable.
> >>
> >> I don't read the code yet, just the cover letter.
> >>
> >> We have global tunings, per-network-namespace tunings, per-socket
> tunings.
> >> It is still unclear why you can't just put different applications
> >> into different namespaces/containers to get different policies.
> >
> > In NET policy, we do per queue tunings.
> 
> Is it possible to isolate NIC queues for containers?

Yes, but we don't  have containers support yet. 

[toc] | [prev] | [next] | [standalone]

#1445818

> 
> On Mon, Jul 18, 2016 at 8:45 AM, Andi Kleen <andi@firstfloor.org> wrote:
> >> It seems strange to me to add such policies to the kernel.
> >> Addmittingly, documentation of some settings is non-existent and one
> >> needs various different tools to set this (sysctl, procfs, sysfs, ethtool, etc).
> >
> > The problem is that different applications need different policies.
> >
> > The only entity which can efficiently negotiate between different
> > applications' conflicting requests is the kernel. And that is pretty
> > much the basic job description of a kernel: multiplex hardware
> > efficiently between different users.
> >
> > So yes the user space tuning approach works for simple cases ("only
> > run workloads that require the same tuning"), but is ultimately not
> > very interesting nor scalable.
> 
> I don't read the code yet, just the cover letter.
> 
> We have global tunings, per-network-namespace tunings, per-socket tunings.
> It is still unclear why you can't just put different applications into different
> namespaces/containers to get different policies.

In NET policy, we do per queue tunings.


Thanks,
Kan

[toc] | [prev] | [next] | [standalone]

#1445739

On 18.07.2016 17:45, Andi Kleen wrote:
>> It seems strange to me to add such policies to the kernel.
>> Addmittingly, documentation of some settings is non-existent and one needs
>> various different tools to set this (sysctl, procfs, sysfs, ethtool, etc).
> 
> The problem is that different applications need different policies.

I fear that if those policies get changed in future, people will rely on
some of their side-effects, causing us to add more and more policies
which basically just differ in those side-effects.

If you compare your policies to madvise or fadvise options, they seem a
have a much more strict and narrower effects, which can be reasoned much
more easily about.

> The only entity which can efficiently negotiate between different
> applications' conflicting requests is the kernel. And that is pretty 
> much the basic job description of a kernel: multiplex hardware
> efficiently between different users.

The multiplexing part seems to be not really relevant for the per-device
settings, thus being controllable from current user space just fine.
Per-task setting could be conflicting with per-socket settings which
could lead to non-deterministic behavior. Probably semantically it
should be made clear what overrides what here (here == cover letter).
Things like indeterminate allocation of sockets in a threaded
environment come to my mind. Also allocation strategy could very much
depend on the installed rss key.

> So yes the user space tuning approach works for simple cases
> ("only run workloads that require the same tuning"), but is ultimately not
> very interesting nor scalable.

I wonder if this can be attacked from a different angle. What would be
missing to add support for this in user space? The first possibility
that came to my mind is to just multiplex those hints in the kernel.
Implement a generic way to add metadata to sockets and allow tuning
daemons to retrieve them via sockdiag? I could imagine that if the
SO_INCOMING_CPU information would be visible in sockdiag, one could
already do more automatic tuning and basically allow to implement your
policy in user space.

Bye,
Hannes

[toc] | [prev] | [next] | [standalone]

#1445775

> I wonder if this can be attacked from a different angle. What would be
> missing to add support for this in user space? The first possibility
> that came to my mind is to just multiplex those hints in the kernel.

"just" is the handwaving part here -- you're proposing a micro kernel
approach where part of the multiplexing job that the kernel is doing
is farmed out to a message passing user space component.

I suspect this would be far more complicated to get right and
perform well than a straight forward monolithic kernel subsystem --
which is traditionally how Linux has approached things.

The daemon would always need to work with out of date state
compared to the latest, because it cannot do any locking with the
kernel state.  So you end up with a complex distributed system with multiple
agents "fighting" with each other, and the tuning agent
never being able to keep up with the actual work.

Also of course it would be fundamentally less efficient than
kernel code doing that, just because of the additional context
switches needed.

-Andi

[toc] | [prev] | [next] | [standalone]

#1445871

Hello,

On Mon, Jul 18, 2016, at 21:43, Andi Kleen wrote:
> > I wonder if this can be attacked from a different angle. What would be
> > missing to add support for this in user space? The first possibility
> > that came to my mind is to just multiplex those hints in the kernel.
> 
> "just" is the handwaving part here -- you're proposing a micro kernel
> approach where part of the multiplexing job that the kernel is doing
> is farmed out to a message passing user space component.
> 
> I suspect this would be far more complicated to get right and
> perform well than a straight forward monolithic kernel subsystem --
> which is traditionally how Linux has approached things.

At the same time having any kind of policy in the kernel was also always
avoided.

> The daemon would always need to work with out of date state
> compared to the latest, because it cannot do any locking with the
> kernel state.  So you end up with a complex distributed system with
> multiple
> agents "fighting" with each other, and the tuning agent
> never being able to keep up with the actual work.

But you don't want to have the tuning agents in the fast path? If you
really try to synchronously update all queue mappings/irqs during socket
creation or connect time this would add rtnl lock to basically socket
creation, as drivers require that. This would slow down basic socket
operations a lot and synchronize them with the management interface.

Even dst_entries are not synchronously updated anymore nowadays as that
would require too much locking overhead in the kernel.

> Also of course it would be fundamentally less efficient than
> kernel code doing that, just because of the additional context
> switches needed.

Synchronizing or configuring any kind of queues already requires
rtnl_mutex. I didn't test it but acquiring rtnl mutex in inet_recvmsg is
unlikely to fly performance wise and might even be very dangerous under
DoS attacks (like I see in 24/30).

Bye,
Hannes

[toc] | [prev] | [next] | [standalone]

#1445984

> 
> > Also of course it would be fundamentally less efficient than kernel
> > code doing that, just because of the additional context switches
> > needed.
> 
> Synchronizing or configuring any kind of queues already requires rtnl_mutex.
> I didn't test it but acquiring rtnl mutex in inet_recvmsg is unlikely to fly
> performance wise and

Yes, rtnl will bring some overheads. But the configuration is one time thing for
application or socket. It only happens on receiving first packet.
Unless the application/socket only transmit few packets, the overhead
could be ignored. If they only transmit few packets, why they care about
performance?

> might even be very dangerous under DoS attacks (like
> I see in 24/30).
> 
Patch 29/30 tries to prevent such case.

Thanks,
Kan

[toc] | [prev] | [next] | [standalone]

#1446024

From: "Liang, Kan" <kan.liang@intel.com>
Date: Tue, 19 Jul 2016 01:49:41 +0000

> Yes, rtnl will bring some overheads. But the configuration is one
> time thing for application or socket. It only happens on receiving
> first packet.

Thanks for destroying our connection rates.

This kind of overhead is simply unacceptable.

[toc] | [prev] | [next] | [standalone]

#1446491

> > Yes, rtnl will bring some overheads. But the configuration is one time
> > thing for application or socket. It only happens on receiving first
> > packet.
> 
> Thanks for destroying our connection rates.
> 
> This kind of overhead is simply unacceptable.

If so, I think I can make the configuration asynchronized for next
version. The connection rate should not be destroyed.

Thanks,
Kan

[toc] | [prev] | [next] | [standalone]

#1445603

> >
> > It is a big challenge to get good network performance. First, the
> > network performance is not good with default system settings. Second,
> > it is too difficult to do automatic tuning for all possible workloads,
> > since workloads have different requirements. Some workloads may want
> high throughput.
> 
> Seems you did lots of tests to find optimal settings for a given base policy.
> 
Yes. Current test only base on Intel i40e driver. The optimal settings should
vary for other devices. But adding settings for new device is not hard.

> What is missing in the kernel UAPI so userspace could do these settings on its
> own, without adding this policy stuff to the kernel?

The main purpose of the proposal is to simplify the configuration. Too many
options will let them confuse. 
For normal users, they just need to tell the kernel that they want high throughput
for the application. The kernel will take care of the rest.
So, I don't think we need an interface for user to set their own policy settings.

> 
> It seems strange to me to add such policies to the kernel.

But kernel is the only place which can merge all user's requests.

> Addmittingly, documentation of some settings is non-existent and one needs
> various different tools to set this (sysctl, procfs, sysfs, ethtool, etc).
> 
> But all of these details could be hidden from user.
> Have you looked at tuna for instance?

Not yet. Is there similar settings for network?

Thanks,
Kan

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | linux.kernel

csiph-web