Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1702217 > unrolled thread

[PATCH v10 0/9] Hyper-V: paravirtualized remote TLB flushing and hypercall improvements

Back to article view | Back to linux.kernel

Contents

  [PATCH v10 0/9] Hyper-V: paravirtualized remote TLB flushing and hypercall improvements Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-08-02 18:10 +0200
    [PATCH v10 1/9] x86/hyper-v: include hyperv/ only when CONFIG_HYPERV is set Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-08-02 18:10 +0200
      [tip:x86/platform] x86/hyper-v: Include hyperv/ only when  CONFIG_HYPERV is set tip-bot for Vitaly Kuznetsov <tipbot@zytor.com> - 2017-08-10 18:50 +0200
    [PATCH v10 3/9] x86/hyper-v: fast hypercall implementation Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-08-02 18:10 +0200
      [tip:x86/platform] x86/hyper-v: Introduce fast hypercall  implementation tip-bot for Vitaly Kuznetsov <tipbot@zytor.com> - 2017-08-10 18:50 +0200
    [PATCH v10 7/9] x86/hyper-v: use hypercall for remote TLB flush Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-08-02 18:20 +0200
      [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush tip-bot for Vitaly Kuznetsov <tipbot@zytor.com> - 2017-08-10 18:50 +0200
      [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush tip-bot for Vitaly Kuznetsov <tipbot@zytor.com> - 2017-08-10 20:30 +0200
        Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB  flush Peter Zijlstra <peterz@infradead.org> - 2017-08-10 21:00 +0200
          Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB  flush Peter Zijlstra <peterz@infradead.org> - 2017-08-10 21:30 +0200
            Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB  flush Peter Zijlstra <peterz@infradead.org> - 2017-08-11 11:10 +0200
              Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB  flush "Kirill A. Shutemov" <kirill@shutemov.name> - 2017-08-11 13:30 +0200
              Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-11 18:20 +0200
                Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB  flush Peter Zijlstra <peterz@infradead.org> - 2017-08-11 18:30 +0200
                  Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-08-14 15:30 +0200
                    Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-08-16 18:50 +0200
                      Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB  flush Boris Ostrovsky <boris.ostrovsky@oracle.com> - 2017-08-16 23:50 +0200
                        Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-08-17 10:00 +0200
            Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-08-11 11:30 +0200
              Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB  flush Peter Zijlstra <peterz@infradead.org> - 2017-08-11 13:00 +0200
                Re: [Xen-devel] [tip:x86/platform] x86/hyper-v: Use hypercall for  remote TLB flush Andrew Cooper <andrew.cooper3@citrix.com> - 2017-08-11 13:10 +0200
                  Re: [Xen-devel] [tip:x86/platform] x86/hyper-v: Use hypercall for  remote TLB flush Peter Zijlstra <peterz@infradead.org> - 2017-08-11 14:10 +0200
                    Re: [Xen-devel] [tip:x86/platform] x86/hyper-v: Use hypercall for  remote TLB flush Steven Rostedt <rostedt@goodmis.org> - 2017-08-16 02:10 +0200
                Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB  flush Juergen Gross <jgross@suse.com> - 2017-08-11 14:30 +0200
                  Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB  flush Peter Zijlstra <peterz@infradead.org> - 2017-08-11 14:40 +0200
                    Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB  flush Juergen Gross <jgross@suse.com> - 2017-08-11 14:50 +0200
                      Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB  flush Peter Zijlstra <peterz@infradead.org> - 2017-08-11 15:00 +0200
                        Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB  flush Juergen Gross <jgross@suse.com> - 2017-08-11 15:10 +0200
                          Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB  flush Peter Zijlstra <peterz@infradead.org> - 2017-08-11 15:50 +0200
    [PATCH v10 4/9] hyper-v: use fast hypercall for HVCALL_SIGNAL_EVENT Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-08-02 18:20 +0200
      [tip:x86/platform] hyper-v: Use fast hypercall for  HVCALL_SIGNAL_EVENT tip-bot for Vitaly Kuznetsov <tipbot@zytor.com> - 2017-08-10 18:50 +0200
    [PATCH v10 5/9] x86/hyper-v: implement rep hypercalls Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-08-02 18:20 +0200
      [tip:x86/platform] x86/hyper-v: Implement rep hypercalls tip-bot for Vitaly Kuznetsov <tipbot@zytor.com> - 2017-08-10 18:50 +0200
    [PATCH v10 8/9] x86/hyper-v: support extended CPU ranges for TLB flush hypercalls Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-08-02 18:20 +0200
    [PATCH v10 9/9] tracing/hyper-v: trace hyperv_mmu_flush_tlb_others() Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-08-02 18:20 +0200
    [PATCH v10 6/9] hyper-v: globalize vp_index Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-08-02 18:20 +0200
      [tip:x86/platform] hyper-v: Globalize vp_index tip-bot for Vitaly Kuznetsov <tipbot@zytor.com> - 2017-08-10 18:50 +0200
    Re: [PATCH v10 0/9] Hyper-V: paravirtualized remote TLB flushing and hypercall improvements Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-08-10 14:00 +0200
      Re: [PATCH v10 0/9] Hyper-V: paravirtualized remote TLB flushing and  hypercall improvements Ingo Molnar <mingo@kernel.org> - 2017-08-10 17:20 +0200
        Re: [PATCH v10 0/9] Hyper-V: paravirtualized remote TLB flushing and hypercall improvements Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-08-10 17:20 +0200
    Re: [PATCH v10 0/9] Hyper-V: paravirtualized remote TLB flushing and hypercall improvements Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-08-10 19:10 +0200

Page 1 of 3  [1] 2 3  Next page →

#1702217 — [PATCH v10 0/9] Hyper-V: paravirtualized remote TLB flushing and hypercall improvements

Changes since v9:
- Rebase to 4.13-rc3.
- Drop PATCH1 as it was already taken by Greg to char-misc tree. There're no
  functional dependencies on this patch so the series can go through a different tree
  (and it actually belongs to x86 if I got Ingo's comment right).
- Add in missing void return type in PATCH1 [Colin King, Ingo Molnar, Greg KH]
- A few minor fixes in what is now PATCH7: add pr_fmt, tiny style fix in
  hyperv_flush_tlb_others() [Andy Shevchenko]
- Fix "error: implicit declaration of function 'virt_to_phys'" in PATCH2
  reported by kbuild test robot (#include <asm/io.h>)
- Add Steven's 'Reviewed-by:' to PATCH9.

Original description:

Hyper-V supports hypercalls for doing local and remote TLB flushing and
gives its guests hints when using hypercall is preferred. While doing
hypercalls for local TLB flushes is probably not practical (and is not
being suggested by modern Hyper-V versions) remote TLB flush with a
hypercall brings significant improvement.

To test the series I wrote a special 'TLB trasher': on a 16 vCPU guest I
was creating 32 threads which were doing 100000 mmap/munmaps each on some
big file. Here are the results:

Before:
# time ./pthread_mmap ./randfile 
real	3m33.118s
user	0m3.698s
sys	3m16.624s

After:
# time ./pthread_mmap ./randfile 
real	2m19.920s
user	0m2.662s
sys	2m9.948s

This series brings a number of small improvements along the way: fast
hypercall implementation and using it for event signaling, rep hypercalls
implementation, hyperv tracing subsystem (which only traces the newly added
remote TLB flush for now).

Vitaly Kuznetsov (9):
  x86/hyper-v: include hyperv/ only when CONFIG_HYPERV is set
  x86/hyper-v: make hv_do_hypercall() inline
  x86/hyper-v: fast hypercall implementation
  hyper-v: use fast hypercall for HVCALL_SIGNAL_EVENT
  x86/hyper-v: implement rep hypercalls
  hyper-v: globalize vp_index
  x86/hyper-v: use hypercall for remote TLB flush
  x86/hyper-v: support extended CPU ranges for TLB flush hypercalls
  tracing/hyper-v: trace hyperv_mmu_flush_tlb_others()

 MAINTAINERS                         |   1 +
 arch/x86/Kbuild                     |   2 +-
 arch/x86/hyperv/Makefile            |   2 +-
 arch/x86/hyperv/hv_init.c           |  90 ++++++------
 arch/x86/hyperv/mmu.c               | 272 ++++++++++++++++++++++++++++++++++++
 arch/x86/include/asm/mshyperv.h     | 147 ++++++++++++++++++-
 arch/x86/include/asm/trace/hyperv.h |  40 ++++++
 arch/x86/include/uapi/asm/hyperv.h  |  17 +++
 arch/x86/kernel/cpu/mshyperv.c      |   1 +
 drivers/hv/channel_mgmt.c           |  20 +--
 drivers/hv/connection.c             |   7 +-
 drivers/hv/hv.c                     |   9 --
 drivers/hv/hyperv_vmbus.h           |  11 --
 drivers/hv/vmbus_drv.c              |  17 ---
 drivers/pci/host/pci-hyperv.c       |  54 +------
 include/linux/hyperv.h              |  17 +--
 16 files changed, 533 insertions(+), 174 deletions(-)
 create mode 100644 arch/x86/hyperv/mmu.c
 create mode 100644 arch/x86/include/asm/trace/hyperv.h

-- 
2.13.3

[toc] | [next] | [standalone]

#1702222 — [PATCH v10 1/9] x86/hyper-v: include hyperv/ only when CONFIG_HYPERV is set

Code is arch/x86/hyperv/ is only needed when CONFIG_HYPERV is set, the
'basic' support and detection lives in arch/x86/kernel/cpu/mshyperv.c
which is included when CONFIG_HYPERVISOR_GUEST is set.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Reviewed-by: Andy Shevchenko <andy.shevchenko@gmail.com>
Reviewed-by: Stephen Hemminger <sthemmin@microsoft.com>
---
 arch/x86/Kbuild                 | 2 +-
 arch/x86/include/asm/mshyperv.h | 7 ++++++-
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/arch/x86/Kbuild b/arch/x86/Kbuild
index 586b786b3edf..3e6f64073005 100644
--- a/arch/x86/Kbuild
+++ b/arch/x86/Kbuild
@@ -8,7 +8,7 @@ obj-$(CONFIG_KVM) += kvm/
 obj-$(CONFIG_XEN) += xen/
 
 # Hyper-V paravirtualization support
-obj-$(CONFIG_HYPERVISOR_GUEST) += hyperv/
+obj-$(subst m,y,$(CONFIG_HYPERV)) += hyperv/
 
 # lguest paravirtualization support
 obj-$(CONFIG_LGUEST_GUEST) += lguest/
diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index 2b58c8c1eeaa..baea2679a0aa 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -173,7 +173,12 @@ void hyperv_init(void);
 void hyperv_report_panic(struct pt_regs *regs);
 bool hv_is_hypercall_page_setup(void);
 void hyperv_cleanup(void);
-#endif
+#else /* CONFIG_HYPERV */
+static inline void hyperv_init(void) {}
+static inline bool hv_is_hypercall_page_setup(void) { return false; }
+static inline void hyperv_cleanup(void) {}
+#endif /* CONFIG_HYPERV */
+
 #ifdef CONFIG_HYPERV_TSCPAGE
 struct ms_hyperv_tsc_page *hv_get_tsc_page(void);
 static inline u64 hv_read_tsc_page(const struct ms_hyperv_tsc_page *tsc_pg)
-- 
2.13.3

[toc] | [prev] | [next] | [standalone]

#1708865 — [tip:x86/platform] x86/hyper-v: Include hyperv/ only when CONFIG_HYPERV is set

Commit-ID:  79cadff2d92bb8b1448f6dba6861d15adc3dc4cb
Gitweb:     http://git.kernel.org/tip/79cadff2d92bb8b1448f6dba6861d15adc3dc4cb
Author:     Vitaly Kuznetsov <vkuznets@redhat.com>
AuthorDate: Wed, 2 Aug 2017 18:09:13 +0200
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Thu, 10 Aug 2017 16:50:22 +0200

x86/hyper-v: Include hyperv/ only when CONFIG_HYPERV is set

Code is arch/x86/hyperv/ is only needed when CONFIG_HYPERV is set, the
'basic' support and detection lives in arch/x86/kernel/cpu/mshyperv.c
which is included when CONFIG_HYPERVISOR_GUEST is set.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Reviewed-by: Andy Shevchenko <andy.shevchenko@gmail.com>
Reviewed-by: Stephen Hemminger <sthemmin@microsoft.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Haiyang Zhang <haiyangz@microsoft.com>
Cc: Jork Loeser <Jork.Loeser@microsoft.com>
Cc: K. Y. Srinivasan <kys@microsoft.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Simon Xiao <sixiao@microsoft.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: devel@linuxdriverproject.org
Link: http://lkml.kernel.org/r/20170802160921.21791-2-vkuznets@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 arch/x86/Kbuild                 | 2 +-
 arch/x86/include/asm/mshyperv.h | 7 ++++++-
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/arch/x86/Kbuild b/arch/x86/Kbuild
index 586b786..3e6f640 100644
--- a/arch/x86/Kbuild
+++ b/arch/x86/Kbuild
@@ -8,7 +8,7 @@ obj-$(CONFIG_KVM) += kvm/
 obj-$(CONFIG_XEN) += xen/
 
 # Hyper-V paravirtualization support
-obj-$(CONFIG_HYPERVISOR_GUEST) += hyperv/
+obj-$(subst m,y,$(CONFIG_HYPERV)) += hyperv/
 
 # lguest paravirtualization support
 obj-$(CONFIG_LGUEST_GUEST) += lguest/
diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index 2b58c8c..baea267 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -173,7 +173,12 @@ void hyperv_init(void);
 void hyperv_report_panic(struct pt_regs *regs);
 bool hv_is_hypercall_page_setup(void);
 void hyperv_cleanup(void);
-#endif
+#else /* CONFIG_HYPERV */
+static inline void hyperv_init(void) {}
+static inline bool hv_is_hypercall_page_setup(void) { return false; }
+static inline void hyperv_cleanup(void) {}
+#endif /* CONFIG_HYPERV */
+
 #ifdef CONFIG_HYPERV_TSCPAGE
 struct ms_hyperv_tsc_page *hv_get_tsc_page(void);
 static inline u64 hv_read_tsc_page(const struct ms_hyperv_tsc_page *tsc_pg)

[toc] | [prev] | [next] | [standalone]

#1702224 — [PATCH v10 3/9] x86/hyper-v: fast hypercall implementation

Hyper-V supports 'fast' hypercalls when all parameters are passed through
registers. Implement an inline version of a simpliest of these calls:
hypercall with one 8-byte input and no output.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Reviewed-by: Andy Shevchenko <andy.shevchenko@gmail.com>
Reviewed-by: Stephen Hemminger <sthemmin@microsoft.com>
---
 arch/x86/include/asm/mshyperv.h | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index 6fa5e342cc86..e484255bd9de 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -209,6 +209,40 @@ static inline u64 hv_do_hypercall(u64 control, void *input, void *output)
 	return hv_status;
 }
 
+#define HV_HYPERCALL_FAST_BIT		BIT(16)
+
+/* Fast hypercall with 8 bytes of input and no output */
+static inline u64 hv_do_fast_hypercall8(u16 code, u64 input1)
+{
+	u64 hv_status, control = (u64)code | HV_HYPERCALL_FAST_BIT;
+	register void *__sp asm(_ASM_SP);
+
+#ifdef CONFIG_X86_64
+	{
+		__asm__ __volatile__("call *%4"
+				     : "=a" (hv_status), "+r" (__sp),
+				       "+c" (control), "+d" (input1)
+				     : "m" (hv_hypercall_pg)
+				     : "cc", "r8", "r9", "r10", "r11");
+	}
+#else
+	{
+		u32 input1_hi = upper_32_bits(input1);
+		u32 input1_lo = lower_32_bits(input1);
+
+		__asm__ __volatile__ ("call *%5"
+				      : "=A"(hv_status),
+					"+c"(input1_lo),
+					"+r"(__sp)
+				      :	"A" (control),
+					"b" (input1_hi),
+					"m" (hv_hypercall_pg)
+				      : "cc", "edi", "esi");
+	}
+#endif
+		return hv_status;
+}
+
 void hyperv_init(void);
 void hyperv_report_panic(struct pt_regs *regs);
 bool hv_is_hypercall_page_setup(void);
-- 
2.13.3

[toc] | [prev] | [next] | [standalone]

#1708880 — [tip:x86/platform] x86/hyper-v: Introduce fast hypercall implementation

Commit-ID:  6a8edbd0c54ae266b12f4f63e406313481c9d4bc
Gitweb:     http://git.kernel.org/tip/6a8edbd0c54ae266b12f4f63e406313481c9d4bc
Author:     Vitaly Kuznetsov <vkuznets@redhat.com>
AuthorDate: Wed, 2 Aug 2017 18:09:15 +0200
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Thu, 10 Aug 2017 16:50:22 +0200

x86/hyper-v: Introduce fast hypercall implementation

Hyper-V supports 'fast' hypercalls when all parameters are passed through
registers. Implement an inline version of a simpliest of these calls:
hypercall with one 8-byte input and no output.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Reviewed-by: Andy Shevchenko <andy.shevchenko@gmail.com>
Reviewed-by: Stephen Hemminger <sthemmin@microsoft.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Haiyang Zhang <haiyangz@microsoft.com>
Cc: Jork Loeser <Jork.Loeser@microsoft.com>
Cc: K. Y. Srinivasan <kys@microsoft.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Simon Xiao <sixiao@microsoft.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: devel@linuxdriverproject.org
Link: http://lkml.kernel.org/r/20170802160921.21791-4-vkuznets@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 arch/x86/include/asm/mshyperv.h | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index 6fa5e34..e484255 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -209,6 +209,40 @@ static inline u64 hv_do_hypercall(u64 control, void *input, void *output)
 	return hv_status;
 }
 
+#define HV_HYPERCALL_FAST_BIT		BIT(16)
+
+/* Fast hypercall with 8 bytes of input and no output */
+static inline u64 hv_do_fast_hypercall8(u16 code, u64 input1)
+{
+	u64 hv_status, control = (u64)code | HV_HYPERCALL_FAST_BIT;
+	register void *__sp asm(_ASM_SP);
+
+#ifdef CONFIG_X86_64
+	{
+		__asm__ __volatile__("call *%4"
+				     : "=a" (hv_status), "+r" (__sp),
+				       "+c" (control), "+d" (input1)
+				     : "m" (hv_hypercall_pg)
+				     : "cc", "r8", "r9", "r10", "r11");
+	}
+#else
+	{
+		u32 input1_hi = upper_32_bits(input1);
+		u32 input1_lo = lower_32_bits(input1);
+
+		__asm__ __volatile__ ("call *%5"
+				      : "=A"(hv_status),
+					"+c"(input1_lo),
+					"+r"(__sp)
+				      :	"A" (control),
+					"b" (input1_hi),
+					"m" (hv_hypercall_pg)
+				      : "cc", "edi", "esi");
+	}
+#endif
+		return hv_status;
+}
+
 void hyperv_init(void);
 void hyperv_report_panic(struct pt_regs *regs);
 bool hv_is_hypercall_page_setup(void);

[toc] | [prev] | [next] | [standalone]

#1702240 — [PATCH v10 7/9] x86/hyper-v: use hypercall for remote TLB flush

Hyper-V host can suggest us to use hypercall for doing remote TLB flush,
this is supposed to work faster than IPIs.

Implementation details: to do HvFlushVirtualAddress{Space,List} hypercalls
we need to put the input somewhere in memory and we don't really want to
have memory allocation on each call so we pre-allocate per cpu memory areas
on boot.

pv_ops patching is happening very early so we need to separate
hyperv_setup_mmu_ops() and hyper_alloc_mmu().

It is possible and easy to implement local TLB flushing too and there is
even a hint for that. However, I don't see a room for optimization on the
host side as both hypercall and native tlb flush will result in vmexit. The
hint is also not set on modern Hyper-V versions.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Reviewed-by: Andy Shevchenko <andy.shevchenko@gmail.com>
Reviewed-by: Stephen Hemminger <sthemmin@microsoft.com>
---
 arch/x86/hyperv/Makefile           |   2 +-
 arch/x86/hyperv/hv_init.c          |   2 +
 arch/x86/hyperv/mmu.c              | 138 +++++++++++++++++++++++++++++++++++++
 arch/x86/include/asm/mshyperv.h    |   3 +
 arch/x86/include/uapi/asm/hyperv.h |   7 ++
 arch/x86/kernel/cpu/mshyperv.c     |   1 +
 6 files changed, 152 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/hyperv/mmu.c

diff --git a/arch/x86/hyperv/Makefile b/arch/x86/hyperv/Makefile
index 171ae09864d7..367a8203cfcf 100644
--- a/arch/x86/hyperv/Makefile
+++ b/arch/x86/hyperv/Makefile
@@ -1 +1 @@
-obj-y		:= hv_init.o
+obj-y		:= hv_init.o mmu.o
diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
index e93b9a0b1b10..1a8eb550c40f 100644
--- a/arch/x86/hyperv/hv_init.c
+++ b/arch/x86/hyperv/hv_init.c
@@ -140,6 +140,8 @@ void hyperv_init(void)
 	hypercall_msr.guest_physical_address = vmalloc_to_pfn(hv_hypercall_pg);
 	wrmsrl(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
 
+	hyper_alloc_mmu();
+
 	/*
 	 * Register Hyper-V specific clocksource.
 	 */
diff --git a/arch/x86/hyperv/mmu.c b/arch/x86/hyperv/mmu.c
new file mode 100644
index 000000000000..9419a20b1d75
--- /dev/null
+++ b/arch/x86/hyperv/mmu.c
@@ -0,0 +1,138 @@
+#define pr_fmt(fmt)  "Hyper-V: " fmt
+
+#include <linux/hyperv.h>
+#include <linux/log2.h>
+#include <linux/slab.h>
+#include <linux/types.h>
+
+#include <asm/fpu/api.h>
+#include <asm/mshyperv.h>
+#include <asm/msr.h>
+#include <asm/tlbflush.h>
+
+/* HvFlushVirtualAddressSpace, HvFlushVirtualAddressList hypercalls */
+struct hv_flush_pcpu {
+	u64 address_space;
+	u64 flags;
+	u64 processor_mask;
+	u64 gva_list[];
+};
+
+/* Each gva in gva_list encodes up to 4096 pages to flush */
+#define HV_TLB_FLUSH_UNIT (4096 * PAGE_SIZE)
+
+static struct hv_flush_pcpu __percpu *pcpu_flush;
+
+/*
+ * Fills in gva_list starting from offset. Returns the number of items added.
+ */
+static inline int fill_gva_list(u64 gva_list[], int offset,
+				unsigned long start, unsigned long end)
+{
+	int gva_n = offset;
+	unsigned long cur = start, diff;
+
+	do {
+		diff = end > cur ? end - cur : 0;
+
+		gva_list[gva_n] = cur & PAGE_MASK;
+		/*
+		 * Lower 12 bits encode the number of additional
+		 * pages to flush (in addition to the 'cur' page).
+		 */
+		if (diff >= HV_TLB_FLUSH_UNIT)
+			gva_list[gva_n] |= ~PAGE_MASK;
+		else if (diff)
+			gva_list[gva_n] |= (diff - 1) >> PAGE_SHIFT;
+
+		cur += HV_TLB_FLUSH_UNIT;
+		gva_n++;
+
+	} while (cur < end);
+
+	return gva_n - offset;
+}
+
+static void hyperv_flush_tlb_others(const struct cpumask *cpus,
+				    const struct flush_tlb_info *info)
+{
+	int cpu, vcpu, gva_n, max_gvas;
+	struct hv_flush_pcpu *flush;
+	u64 status = U64_MAX;
+	unsigned long flags;
+
+	if (!pcpu_flush || !hv_hypercall_pg)
+		goto do_native;
+
+	if (cpumask_empty(cpus))
+		return;
+
+	local_irq_save(flags);
+
+	flush = this_cpu_ptr(pcpu_flush);
+
+	if (info->mm) {
+		flush->address_space = virt_to_phys(info->mm->pgd);
+		flush->flags = 0;
+	} else {
+		flush->address_space = 0;
+		flush->flags = HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES;
+	}
+
+	flush->processor_mask = 0;
+	if (cpumask_equal(cpus, cpu_present_mask)) {
+		flush->flags |= HV_FLUSH_ALL_PROCESSORS;
+	} else {
+		for_each_cpu(cpu, cpus) {
+			vcpu = hv_cpu_number_to_vp_number(cpu);
+			if (vcpu >= 64)
+				goto do_native;
+
+			__set_bit(vcpu, (unsigned long *)
+				  &flush->processor_mask);
+		}
+	}
+
+	/*
+	 * We can flush not more than max_gvas with one hypercall. Flush the
+	 * whole address space if we were asked to do more.
+	 */
+	max_gvas = (PAGE_SIZE - sizeof(*flush)) / sizeof(flush->gva_list[0]);
+
+	if (info->end == TLB_FLUSH_ALL) {
+		flush->flags |= HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY;
+		status = hv_do_hypercall(HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE,
+					 flush, NULL);
+	} else if (info->end &&
+		   ((info->end - info->start)/HV_TLB_FLUSH_UNIT) > max_gvas) {
+		status = hv_do_hypercall(HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE,
+					 flush, NULL);
+	} else {
+		gva_n = fill_gva_list(flush->gva_list, 0,
+				      info->start, info->end);
+		status = hv_do_rep_hypercall(HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST,
+					     gva_n, 0, flush, NULL);
+	}
+
+	local_irq_restore(flags);
+
+	if (!(status & HV_HYPERCALL_RESULT_MASK))
+		return;
+do_native:
+	native_flush_tlb_others(cpus, info);
+}
+
+void hyperv_setup_mmu_ops(void)
+{
+	if (ms_hyperv.hints & HV_X64_REMOTE_TLB_FLUSH_RECOMMENDED) {
+		pr_info("Using hypercall for remote TLB flush\n");
+		pv_mmu_ops.flush_tlb_others = hyperv_flush_tlb_others;
+		setup_clear_cpu_cap(X86_FEATURE_PCID);
+	}
+}
+
+void hyper_alloc_mmu(void)
+{
+	if (ms_hyperv.hints & HV_X64_REMOTE_TLB_FLUSH_RECOMMENDED)
+		pcpu_flush = __alloc_percpu(PAGE_SIZE, PAGE_SIZE);
+}
diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index efd2f80d3353..0d4b01c5e438 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -307,6 +307,8 @@ static inline int hv_cpu_number_to_vp_number(int cpu_number)
 }
 
 void hyperv_init(void);
+void hyperv_setup_mmu_ops(void);
+void hyper_alloc_mmu(void);
 void hyperv_report_panic(struct pt_regs *regs);
 bool hv_is_hypercall_page_setup(void);
 void hyperv_cleanup(void);
@@ -314,6 +316,7 @@ void hyperv_cleanup(void);
 static inline void hyperv_init(void) {}
 static inline bool hv_is_hypercall_page_setup(void) { return false; }
 static inline void hyperv_cleanup(void) {}
+static inline void hyperv_setup_mmu_ops(void) {}
 #endif /* CONFIG_HYPERV */
 
 #ifdef CONFIG_HYPERV_TSCPAGE
diff --git a/arch/x86/include/uapi/asm/hyperv.h b/arch/x86/include/uapi/asm/hyperv.h
index 127ddadee1a5..a6fdd3b82b4a 100644
--- a/arch/x86/include/uapi/asm/hyperv.h
+++ b/arch/x86/include/uapi/asm/hyperv.h
@@ -242,6 +242,8 @@
 		(~((1ull << HV_X64_MSR_HYPERCALL_PAGE_ADDRESS_SHIFT) - 1))
 
 /* Declare the various hypercall operations. */
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE	0x0002
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST	0x0003
 #define HVCALL_NOTIFY_LONG_SPIN_WAIT		0x0008
 #define HVCALL_POST_MESSAGE			0x005c
 #define HVCALL_SIGNAL_EVENT			0x005d
@@ -259,6 +261,11 @@
 #define HV_PROCESSOR_POWER_STATE_C2		2
 #define HV_PROCESSOR_POWER_STATE_C3		3
 
+#define HV_FLUSH_ALL_PROCESSORS			BIT(0)
+#define HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES	BIT(1)
+#define HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY	BIT(2)
+#define HV_FLUSH_USE_EXTENDED_RANGE_FORMAT	BIT(3)
+
 /* hypercall status code */
 #define HV_STATUS_SUCCESS			0
 #define HV_STATUS_INVALID_HYPERCALL_CODE	2
diff --git a/arch/x86/kernel/cpu/mshyperv.c b/arch/x86/kernel/cpu/mshyperv.c
index 70e717fccdd6..daefd67a66c7 100644
--- a/arch/x86/kernel/cpu/mshyperv.c
+++ b/arch/x86/kernel/cpu/mshyperv.c
@@ -249,6 +249,7 @@ static void __init ms_hyperv_init_platform(void)
 	 * Setup the hook to get control post apic initialization.
 	 */
 	x86_platform.apic_post_init = hyperv_init;
+	hyperv_setup_mmu_ops();
 #endif
 }
 
-- 
2.13.3

[toc] | [prev] | [next] | [standalone]

#1708881 — [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush

Commit-ID:  88b46342eb037d35decda4d651cfee5216f4f822
Gitweb:     http://git.kernel.org/tip/88b46342eb037d35decda4d651cfee5216f4f822
Author:     Vitaly Kuznetsov <vkuznets@redhat.com>
AuthorDate: Wed, 2 Aug 2017 18:09:19 +0200
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Thu, 10 Aug 2017 16:50:23 +0200

x86/hyper-v: Use hypercall for remote TLB flush

Hyper-V host can suggest us to use hypercall for doing remote TLB flush,
this is supposed to work faster than IPIs.

Implementation details: to do HvFlushVirtualAddress{Space,List} hypercalls
we need to put the input somewhere in memory and we don't really want to
have memory allocation on each call so we pre-allocate per cpu memory areas
on boot.

pv_ops patching is happening very early so we need to separate
hyperv_setup_mmu_ops() and hyper_alloc_mmu().

It is possible and easy to implement local TLB flushing too and there is
even a hint for that. However, I don't see a room for optimization on the
host side as both hypercall and native tlb flush will result in vmexit. The
hint is also not set on modern Hyper-V versions.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Reviewed-by: Andy Shevchenko <andy.shevchenko@gmail.com>
Reviewed-by: Stephen Hemminger <sthemmin@microsoft.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Haiyang Zhang <haiyangz@microsoft.com>
Cc: Jork Loeser <Jork.Loeser@microsoft.com>
Cc: K. Y. Srinivasan <kys@microsoft.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Simon Xiao <sixiao@microsoft.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: devel@linuxdriverproject.org
Link: http://lkml.kernel.org/r/20170802160921.21791-8-vkuznets@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 arch/x86/hyperv/Makefile           |   2 +-
 arch/x86/hyperv/hv_init.c          |   2 +
 arch/x86/hyperv/mmu.c              | 138 +++++++++++++++++++++++++++++++++++++
 arch/x86/include/asm/mshyperv.h    |   3 +
 arch/x86/include/uapi/asm/hyperv.h |   7 ++
 arch/x86/kernel/cpu/mshyperv.c     |   1 +
 6 files changed, 152 insertions(+), 1 deletion(-)

diff --git a/arch/x86/hyperv/Makefile b/arch/x86/hyperv/Makefile
index 171ae09..367a820 100644
--- a/arch/x86/hyperv/Makefile
+++ b/arch/x86/hyperv/Makefile
@@ -1 +1 @@
-obj-y		:= hv_init.o
+obj-y		:= hv_init.o mmu.o
diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
index e93b9a0..1a8eb55 100644
--- a/arch/x86/hyperv/hv_init.c
+++ b/arch/x86/hyperv/hv_init.c
@@ -140,6 +140,8 @@ void hyperv_init(void)
 	hypercall_msr.guest_physical_address = vmalloc_to_pfn(hv_hypercall_pg);
 	wrmsrl(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
 
+	hyper_alloc_mmu();
+
 	/*
 	 * Register Hyper-V specific clocksource.
 	 */
diff --git a/arch/x86/hyperv/mmu.c b/arch/x86/hyperv/mmu.c
new file mode 100644
index 0000000..9419a20
--- /dev/null
+++ b/arch/x86/hyperv/mmu.c
@@ -0,0 +1,138 @@
+#define pr_fmt(fmt)  "Hyper-V: " fmt
+
+#include <linux/hyperv.h>
+#include <linux/log2.h>
+#include <linux/slab.h>
+#include <linux/types.h>
+
+#include <asm/fpu/api.h>
+#include <asm/mshyperv.h>
+#include <asm/msr.h>
+#include <asm/tlbflush.h>
+
+/* HvFlushVirtualAddressSpace, HvFlushVirtualAddressList hypercalls */
+struct hv_flush_pcpu {
+	u64 address_space;
+	u64 flags;
+	u64 processor_mask;
+	u64 gva_list[];
+};
+
+/* Each gva in gva_list encodes up to 4096 pages to flush */
+#define HV_TLB_FLUSH_UNIT (4096 * PAGE_SIZE)
+
+static struct hv_flush_pcpu __percpu *pcpu_flush;
+
+/*
+ * Fills in gva_list starting from offset. Returns the number of items added.
+ */
+static inline int fill_gva_list(u64 gva_list[], int offset,
+				unsigned long start, unsigned long end)
+{
+	int gva_n = offset;
+	unsigned long cur = start, diff;
+
+	do {
+		diff = end > cur ? end - cur : 0;
+
+		gva_list[gva_n] = cur & PAGE_MASK;
+		/*
+		 * Lower 12 bits encode the number of additional
+		 * pages to flush (in addition to the 'cur' page).
+		 */
+		if (diff >= HV_TLB_FLUSH_UNIT)
+			gva_list[gva_n] |= ~PAGE_MASK;
+		else if (diff)
+			gva_list[gva_n] |= (diff - 1) >> PAGE_SHIFT;
+
+		cur += HV_TLB_FLUSH_UNIT;
+		gva_n++;
+
+	} while (cur < end);
+
+	return gva_n - offset;
+}
+
+static void hyperv_flush_tlb_others(const struct cpumask *cpus,
+				    const struct flush_tlb_info *info)
+{
+	int cpu, vcpu, gva_n, max_gvas;
+	struct hv_flush_pcpu *flush;
+	u64 status = U64_MAX;
+	unsigned long flags;
+
+	if (!pcpu_flush || !hv_hypercall_pg)
+		goto do_native;
+
+	if (cpumask_empty(cpus))
+		return;
+
+	local_irq_save(flags);
+
+	flush = this_cpu_ptr(pcpu_flush);
+
+	if (info->mm) {
+		flush->address_space = virt_to_phys(info->mm->pgd);
+		flush->flags = 0;
+	} else {
+		flush->address_space = 0;
+		flush->flags = HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES;
+	}
+
+	flush->processor_mask = 0;
+	if (cpumask_equal(cpus, cpu_present_mask)) {
+		flush->flags |= HV_FLUSH_ALL_PROCESSORS;
+	} else {
+		for_each_cpu(cpu, cpus) {
+			vcpu = hv_cpu_number_to_vp_number(cpu);
+			if (vcpu >= 64)
+				goto do_native;
+
+			__set_bit(vcpu, (unsigned long *)
+				  &flush->processor_mask);
+		}
+	}
+
+	/*
+	 * We can flush not more than max_gvas with one hypercall. Flush the
+	 * whole address space if we were asked to do more.
+	 */
+	max_gvas = (PAGE_SIZE - sizeof(*flush)) / sizeof(flush->gva_list[0]);
+
+	if (info->end == TLB_FLUSH_ALL) {
+		flush->flags |= HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY;
+		status = hv_do_hypercall(HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE,
+					 flush, NULL);
+	} else if (info->end &&
+		   ((info->end - info->start)/HV_TLB_FLUSH_UNIT) > max_gvas) {
+		status = hv_do_hypercall(HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE,
+					 flush, NULL);
+	} else {
+		gva_n = fill_gva_list(flush->gva_list, 0,
+				      info->start, info->end);
+		status = hv_do_rep_hypercall(HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST,
+					     gva_n, 0, flush, NULL);
+	}
+
+	local_irq_restore(flags);
+
+	if (!(status & HV_HYPERCALL_RESULT_MASK))
+		return;
+do_native:
+	native_flush_tlb_others(cpus, info);
+}
+
+void hyperv_setup_mmu_ops(void)
+{
+	if (ms_hyperv.hints & HV_X64_REMOTE_TLB_FLUSH_RECOMMENDED) {
+		pr_info("Using hypercall for remote TLB flush\n");
+		pv_mmu_ops.flush_tlb_others = hyperv_flush_tlb_others;
+		setup_clear_cpu_cap(X86_FEATURE_PCID);
+	}
+}
+
+void hyper_alloc_mmu(void)
+{
+	if (ms_hyperv.hints & HV_X64_REMOTE_TLB_FLUSH_RECOMMENDED)
+		pcpu_flush = __alloc_percpu(PAGE_SIZE, PAGE_SIZE);
+}
diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index efd2f80..0d4b01c 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -307,6 +307,8 @@ static inline int hv_cpu_number_to_vp_number(int cpu_number)
 }
 
 void hyperv_init(void);
+void hyperv_setup_mmu_ops(void);
+void hyper_alloc_mmu(void);
 void hyperv_report_panic(struct pt_regs *regs);
 bool hv_is_hypercall_page_setup(void);
 void hyperv_cleanup(void);
@@ -314,6 +316,7 @@ void hyperv_cleanup(void);
 static inline void hyperv_init(void) {}
 static inline bool hv_is_hypercall_page_setup(void) { return false; }
 static inline void hyperv_cleanup(void) {}
+static inline void hyperv_setup_mmu_ops(void) {}
 #endif /* CONFIG_HYPERV */
 
 #ifdef CONFIG_HYPERV_TSCPAGE
diff --git a/arch/x86/include/uapi/asm/hyperv.h b/arch/x86/include/uapi/asm/hyperv.h
index 127ddad..a6fdd3b 100644
--- a/arch/x86/include/uapi/asm/hyperv.h
+++ b/arch/x86/include/uapi/asm/hyperv.h
@@ -242,6 +242,8 @@
 		(~((1ull << HV_X64_MSR_HYPERCALL_PAGE_ADDRESS_SHIFT) - 1))
 
 /* Declare the various hypercall operations. */
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE	0x0002
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST	0x0003
 #define HVCALL_NOTIFY_LONG_SPIN_WAIT		0x0008
 #define HVCALL_POST_MESSAGE			0x005c
 #define HVCALL_SIGNAL_EVENT			0x005d
@@ -259,6 +261,11 @@
 #define HV_PROCESSOR_POWER_STATE_C2		2
 #define HV_PROCESSOR_POWER_STATE_C3		3
 
+#define HV_FLUSH_ALL_PROCESSORS			BIT(0)
+#define HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES	BIT(1)
+#define HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY	BIT(2)
+#define HV_FLUSH_USE_EXTENDED_RANGE_FORMAT	BIT(3)
+
 /* hypercall status code */
 #define HV_STATUS_SUCCESS			0
 #define HV_STATUS_INVALID_HYPERCALL_CODE	2
diff --git a/arch/x86/kernel/cpu/mshyperv.c b/arch/x86/kernel/cpu/mshyperv.c
index 70e717f..daefd67 100644
--- a/arch/x86/kernel/cpu/mshyperv.c
+++ b/arch/x86/kernel/cpu/mshyperv.c
@@ -249,6 +249,7 @@ static void __init ms_hyperv_init_platform(void)
 	 * Setup the hook to get control post apic initialization.
 	 */
 	x86_platform.apic_post_init = hyperv_init;
+	hyperv_setup_mmu_ops();
 #endif
 }
 

[toc] | [prev] | [next] | [standalone]

#1708953 — [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush

Commit-ID:  2ffd9e33ce4af4e8cfa3e17bf493defe8474e2eb
Gitweb:     http://git.kernel.org/tip/2ffd9e33ce4af4e8cfa3e17bf493defe8474e2eb
Author:     Vitaly Kuznetsov <vkuznets@redhat.com>
AuthorDate: Wed, 2 Aug 2017 18:09:19 +0200
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Thu, 10 Aug 2017 20:16:44 +0200

x86/hyper-v: Use hypercall for remote TLB flush

Hyper-V host can suggest us to use hypercall for doing remote TLB flush,
this is supposed to work faster than IPIs.

Implementation details: to do HvFlushVirtualAddress{Space,List} hypercalls
we need to put the input somewhere in memory and we don't really want to
have memory allocation on each call so we pre-allocate per cpu memory areas
on boot.

pv_ops patching is happening very early so we need to separate
hyperv_setup_mmu_ops() and hyper_alloc_mmu().

It is possible and easy to implement local TLB flushing too and there is
even a hint for that. However, I don't see a room for optimization on the
host side as both hypercall and native tlb flush will result in vmexit. The
hint is also not set on modern Hyper-V versions.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Reviewed-by: Andy Shevchenko <andy.shevchenko@gmail.com>
Reviewed-by: Stephen Hemminger <sthemmin@microsoft.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Haiyang Zhang <haiyangz@microsoft.com>
Cc: Jork Loeser <Jork.Loeser@microsoft.com>
Cc: K. Y. Srinivasan <kys@microsoft.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Simon Xiao <sixiao@microsoft.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: devel@linuxdriverproject.org
Link: http://lkml.kernel.org/r/20170802160921.21791-8-vkuznets@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 arch/x86/hyperv/Makefile           |   2 +-
 arch/x86/hyperv/hv_init.c          |   2 +
 arch/x86/hyperv/mmu.c              | 138 +++++++++++++++++++++++++++++++++++++
 arch/x86/include/asm/mshyperv.h    |   3 +
 arch/x86/include/uapi/asm/hyperv.h |   7 ++
 arch/x86/kernel/cpu/mshyperv.c     |   1 +
 drivers/hv/Kconfig                 |   1 +
 7 files changed, 153 insertions(+), 1 deletion(-)

diff --git a/arch/x86/hyperv/Makefile b/arch/x86/hyperv/Makefile
index 171ae09..367a820 100644
--- a/arch/x86/hyperv/Makefile
+++ b/arch/x86/hyperv/Makefile
@@ -1 +1 @@
-obj-y		:= hv_init.o
+obj-y		:= hv_init.o mmu.o
diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
index e93b9a0..1a8eb55 100644
--- a/arch/x86/hyperv/hv_init.c
+++ b/arch/x86/hyperv/hv_init.c
@@ -140,6 +140,8 @@ void hyperv_init(void)
 	hypercall_msr.guest_physical_address = vmalloc_to_pfn(hv_hypercall_pg);
 	wrmsrl(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
 
+	hyper_alloc_mmu();
+
 	/*
 	 * Register Hyper-V specific clocksource.
 	 */
diff --git a/arch/x86/hyperv/mmu.c b/arch/x86/hyperv/mmu.c
new file mode 100644
index 0000000..9419a20
--- /dev/null
+++ b/arch/x86/hyperv/mmu.c
@@ -0,0 +1,138 @@
+#define pr_fmt(fmt)  "Hyper-V: " fmt
+
+#include <linux/hyperv.h>
+#include <linux/log2.h>
+#include <linux/slab.h>
+#include <linux/types.h>
+
+#include <asm/fpu/api.h>
+#include <asm/mshyperv.h>
+#include <asm/msr.h>
+#include <asm/tlbflush.h>
+
+/* HvFlushVirtualAddressSpace, HvFlushVirtualAddressList hypercalls */
+struct hv_flush_pcpu {
+	u64 address_space;
+	u64 flags;
+	u64 processor_mask;
+	u64 gva_list[];
+};
+
+/* Each gva in gva_list encodes up to 4096 pages to flush */
+#define HV_TLB_FLUSH_UNIT (4096 * PAGE_SIZE)
+
+static struct hv_flush_pcpu __percpu *pcpu_flush;
+
+/*
+ * Fills in gva_list starting from offset. Returns the number of items added.
+ */
+static inline int fill_gva_list(u64 gva_list[], int offset,
+				unsigned long start, unsigned long end)
+{
+	int gva_n = offset;
+	unsigned long cur = start, diff;
+
+	do {
+		diff = end > cur ? end - cur : 0;
+
+		gva_list[gva_n] = cur & PAGE_MASK;
+		/*
+		 * Lower 12 bits encode the number of additional
+		 * pages to flush (in addition to the 'cur' page).
+		 */
+		if (diff >= HV_TLB_FLUSH_UNIT)
+			gva_list[gva_n] |= ~PAGE_MASK;
+		else if (diff)
+			gva_list[gva_n] |= (diff - 1) >> PAGE_SHIFT;
+
+		cur += HV_TLB_FLUSH_UNIT;
+		gva_n++;
+
+	} while (cur < end);
+
+	return gva_n - offset;
+}
+
+static void hyperv_flush_tlb_others(const struct cpumask *cpus,
+				    const struct flush_tlb_info *info)
+{
+	int cpu, vcpu, gva_n, max_gvas;
+	struct hv_flush_pcpu *flush;
+	u64 status = U64_MAX;
+	unsigned long flags;
+
+	if (!pcpu_flush || !hv_hypercall_pg)
+		goto do_native;
+
+	if (cpumask_empty(cpus))
+		return;
+
+	local_irq_save(flags);
+
+	flush = this_cpu_ptr(pcpu_flush);
+
+	if (info->mm) {
+		flush->address_space = virt_to_phys(info->mm->pgd);
+		flush->flags = 0;
+	} else {
+		flush->address_space = 0;
+		flush->flags = HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES;
+	}
+
+	flush->processor_mask = 0;
+	if (cpumask_equal(cpus, cpu_present_mask)) {
+		flush->flags |= HV_FLUSH_ALL_PROCESSORS;
+	} else {
+		for_each_cpu(cpu, cpus) {
+			vcpu = hv_cpu_number_to_vp_number(cpu);
+			if (vcpu >= 64)
+				goto do_native;
+
+			__set_bit(vcpu, (unsigned long *)
+				  &flush->processor_mask);
+		}
+	}
+
+	/*
+	 * We can flush not more than max_gvas with one hypercall. Flush the
+	 * whole address space if we were asked to do more.
+	 */
+	max_gvas = (PAGE_SIZE - sizeof(*flush)) / sizeof(flush->gva_list[0]);
+
+	if (info->end == TLB_FLUSH_ALL) {
+		flush->flags |= HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY;
+		status = hv_do_hypercall(HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE,
+					 flush, NULL);
+	} else if (info->end &&
+		   ((info->end - info->start)/HV_TLB_FLUSH_UNIT) > max_gvas) {
+		status = hv_do_hypercall(HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE,
+					 flush, NULL);
+	} else {
+		gva_n = fill_gva_list(flush->gva_list, 0,
+				      info->start, info->end);
+		status = hv_do_rep_hypercall(HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST,
+					     gva_n, 0, flush, NULL);
+	}
+
+	local_irq_restore(flags);
+
+	if (!(status & HV_HYPERCALL_RESULT_MASK))
+		return;
+do_native:
+	native_flush_tlb_others(cpus, info);
+}
+
+void hyperv_setup_mmu_ops(void)
+{
+	if (ms_hyperv.hints & HV_X64_REMOTE_TLB_FLUSH_RECOMMENDED) {
+		pr_info("Using hypercall for remote TLB flush\n");
+		pv_mmu_ops.flush_tlb_others = hyperv_flush_tlb_others;
+		setup_clear_cpu_cap(X86_FEATURE_PCID);
+	}
+}
+
+void hyper_alloc_mmu(void)
+{
+	if (ms_hyperv.hints & HV_X64_REMOTE_TLB_FLUSH_RECOMMENDED)
+		pcpu_flush = __alloc_percpu(PAGE_SIZE, PAGE_SIZE);
+}
diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index efd2f80..0d4b01c 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -307,6 +307,8 @@ static inline int hv_cpu_number_to_vp_number(int cpu_number)
 }
 
 void hyperv_init(void);
+void hyperv_setup_mmu_ops(void);
+void hyper_alloc_mmu(void);
 void hyperv_report_panic(struct pt_regs *regs);
 bool hv_is_hypercall_page_setup(void);
 void hyperv_cleanup(void);
@@ -314,6 +316,7 @@ void hyperv_cleanup(void);
 static inline void hyperv_init(void) {}
 static inline bool hv_is_hypercall_page_setup(void) { return false; }
 static inline void hyperv_cleanup(void) {}
+static inline void hyperv_setup_mmu_ops(void) {}
 #endif /* CONFIG_HYPERV */
 
 #ifdef CONFIG_HYPERV_TSCPAGE
diff --git a/arch/x86/include/uapi/asm/hyperv.h b/arch/x86/include/uapi/asm/hyperv.h
index 127ddad..a6fdd3b 100644
--- a/arch/x86/include/uapi/asm/hyperv.h
+++ b/arch/x86/include/uapi/asm/hyperv.h
@@ -242,6 +242,8 @@
 		(~((1ull << HV_X64_MSR_HYPERCALL_PAGE_ADDRESS_SHIFT) - 1))
 
 /* Declare the various hypercall operations. */
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE	0x0002
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST	0x0003
 #define HVCALL_NOTIFY_LONG_SPIN_WAIT		0x0008
 #define HVCALL_POST_MESSAGE			0x005c
 #define HVCALL_SIGNAL_EVENT			0x005d
@@ -259,6 +261,11 @@
 #define HV_PROCESSOR_POWER_STATE_C2		2
 #define HV_PROCESSOR_POWER_STATE_C3		3
 
+#define HV_FLUSH_ALL_PROCESSORS			BIT(0)
+#define HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES	BIT(1)
+#define HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY	BIT(2)
+#define HV_FLUSH_USE_EXTENDED_RANGE_FORMAT	BIT(3)
+
 /* hypercall status code */
 #define HV_STATUS_SUCCESS			0
 #define HV_STATUS_INVALID_HYPERCALL_CODE	2
diff --git a/arch/x86/kernel/cpu/mshyperv.c b/arch/x86/kernel/cpu/mshyperv.c
index 70e717f..daefd67 100644
--- a/arch/x86/kernel/cpu/mshyperv.c
+++ b/arch/x86/kernel/cpu/mshyperv.c
@@ -249,6 +249,7 @@ static void __init ms_hyperv_init_platform(void)
 	 * Setup the hook to get control post apic initialization.
 	 */
 	x86_platform.apic_post_init = hyperv_init;
+	hyperv_setup_mmu_ops();
 #endif
 }
 
diff --git a/drivers/hv/Kconfig b/drivers/hv/Kconfig
index c29cd53..50b89ea 100644
--- a/drivers/hv/Kconfig
+++ b/drivers/hv/Kconfig
@@ -3,6 +3,7 @@ menu "Microsoft Hyper-V guest support"
 config HYPERV
 	tristate "Microsoft Hyper-V client drivers"
 	depends on X86 && ACPI && PCI && X86_LOCAL_APIC && HYPERVISOR_GUEST
+	select PARAVIRT
 	help
 	  Select this option to run Linux as a Hyper-V client operating
 	  system.

[toc] | [prev] | [next] | [standalone]

#1708967 — Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush

On Thu, Aug 10, 2017 at 11:21:49AM -0700, tip-bot for Vitaly Kuznetsov wrote:
> Commit-ID:  2ffd9e33ce4af4e8cfa3e17bf493defe8474e2eb
> Gitweb:     http://git.kernel.org/tip/2ffd9e33ce4af4e8cfa3e17bf493defe8474e2eb
> Author:     Vitaly Kuznetsov <vkuznets@redhat.com>
> AuthorDate: Wed, 2 Aug 2017 18:09:19 +0200
> Committer:  Ingo Molnar <mingo@kernel.org>
> CommitDate: Thu, 10 Aug 2017 20:16:44 +0200
> 
> x86/hyper-v: Use hypercall for remote TLB flush
> 
> Hyper-V host can suggest us to use hypercall for doing remote TLB flush,
> this is supposed to work faster than IPIs.
> 
> Implementation details: to do HvFlushVirtualAddress{Space,List} hypercalls
> we need to put the input somewhere in memory and we don't really want to
> have memory allocation on each call so we pre-allocate per cpu memory areas
> on boot.
> 
> pv_ops patching is happening very early so we need to separate
> hyperv_setup_mmu_ops() and hyper_alloc_mmu().
> 
> It is possible and easy to implement local TLB flushing too and there is
> even a hint for that. However, I don't see a room for optimization on the
> host side as both hypercall and native tlb flush will result in vmexit. The
> hint is also not set on modern Hyper-V versions.

Hold on.. if we don't IPI for TLB invalidation. What serializes our
software page table walkers like fast_gup() ?

[toc] | [prev] | [next] | [standalone]

#1708982 — Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush

On Thu, Aug 10, 2017 at 07:08:22PM +0000, Jork Loeser wrote:

> > > Subject: Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush
> 
> > > Hold on.. if we don't IPI for TLB invalidation. What serializes our
> > > software page table walkers like fast_gup() ?
> > 
> > Hypervisor may implement this functionality via an IPI.
> > 
> > K. Y
> 
> HvFlushVirtualAddressList() states:
> This call guarantees that by the time control returns back to the
> caller, the observable effects of all flushes on the specified virtual
> processors have occurred.
> 
> HvFlushVirtualAddressListEx() refers to HvFlushVirtualAddressList() as adding sparse target VP lists.
> 
> Is this enough of a guarantee, or do you see other races?

That's nowhere near enough. We need the remote CPU to have completed any
guest IF section that was in progress at the time of the call.

So if a host IPI can interrupt a guest while the guest has IF cleared,
and we then process the host IPI -- clear the TLBs -- before resuming the
guest, which still has IF cleared, we've got a problem.

Because at that point, our software page-table walker, that relies on IF
being clear to guarantee the page-tables exist, because it holds off the
TLB invalidate and thereby the freeing of the pages, gets its pages
ripped out from under it.

[toc] | [prev] | [next] | [standalone]

#1709389 — Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush

On Fri, Aug 11, 2017 at 01:15:18AM +0000, Jork Loeser wrote:

> > > HvFlushVirtualAddressList() states:
> > > This call guarantees that by the time control returns back to the
> > > caller, the observable effects of all flushes on the specified virtual
> > > processors have occurred.
> > >
> > > HvFlushVirtualAddressListEx() refers to HvFlushVirtualAddressList() as adding
> > > sparse target VP lists.
> > >
> > > Is this enough of a guarantee, or do you see other races?
> > 
> > That's nowhere near enough. We need the remote CPU to have completed any
> > guest IF section that was in progress at the time of the call.
> > 
> > So if a host IPI can interrupt a guest while the guest has IF cleared, and we then
> > process the host IPI -- clear the TLBs -- before resuming the guest, which still has
> > IF cleared, we've got a problem.
> > 
> > Because at that point, our software page-table walker, that relies on IF being
> > clear to guarantee the page-tables exist, because it holds off the TLB invalidate
> > and thereby the freeing of the pages, gets its pages ripped out from under it.
> 
> I see, IF is used as a locking mechanism for the pages. Would
> CONFIG_HAVE_RCU_TABLE_FREE be an option for x86? There are caveats
> (statically enabled, RCU for page-free), yet if the resulting perf is
> still a gain it would be worthwhile for Hyper-V targeted kernels.

I'm sure we talked about using HAVE_RCU_TABLE_FREE for x86 (and yes that
would make it work again), but this was some years ago and I cannot
readily find those emails.

Kirill would you have any opinions?

[toc] | [prev] | [next] | [standalone]

#1709474 — Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush

On Fri, Aug 11, 2017 at 11:03:36AM +0200, Peter Zijlstra wrote:
> On Fri, Aug 11, 2017 at 01:15:18AM +0000, Jork Loeser wrote:
> 
> > > > HvFlushVirtualAddressList() states:
> > > > This call guarantees that by the time control returns back to the
> > > > caller, the observable effects of all flushes on the specified virtual
> > > > processors have occurred.
> > > >
> > > > HvFlushVirtualAddressListEx() refers to HvFlushVirtualAddressList() as adding
> > > > sparse target VP lists.
> > > >
> > > > Is this enough of a guarantee, or do you see other races?
> > > 
> > > That's nowhere near enough. We need the remote CPU to have completed any
> > > guest IF section that was in progress at the time of the call.
> > > 
> > > So if a host IPI can interrupt a guest while the guest has IF cleared, and we then
> > > process the host IPI -- clear the TLBs -- before resuming the guest, which still has
> > > IF cleared, we've got a problem.
> > > 
> > > Because at that point, our software page-table walker, that relies on IF being
> > > clear to guarantee the page-tables exist, because it holds off the TLB invalidate
> > > and thereby the freeing of the pages, gets its pages ripped out from under it.
> > 
> > I see, IF is used as a locking mechanism for the pages. Would
> > CONFIG_HAVE_RCU_TABLE_FREE be an option for x86? There are caveats
> > (statically enabled, RCU for page-free), yet if the resulting perf is
> > still a gain it would be worthwhile for Hyper-V targeted kernels.
> 
> I'm sure we talked about using HAVE_RCU_TABLE_FREE for x86 (and yes that
> would make it work again), but this was some years ago and I cannot
> readily find those emails.
> 
> Kirill would you have any opinions?

I guess we can try this. The main question is what would be performance
implications of such move.

-- 
 Kirill A. Shutemov

[toc] | [prev] | [next] | [standalone]

#1709811 — Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush

On Fri, Aug 11, 2017 at 2:03 AM, Peter Zijlstra <peterz@infradead.org> wrote:
>
> I'm sure we talked about using HAVE_RCU_TABLE_FREE for x86 (and yes that
> would make it work again), but this was some years ago and I cannot
> readily find those emails.

I think the only time we really talked about HAVE_RCU_TABLE_FREE for
x86 (at least that I was cc'd on) was not because of RCU freeing, but
because we just wanted to use the generic page table lookup code on
x86 *despite* not using RCU freeing.

And we just ended up renaming HAVE_GENERIC_RCU_GUP as HAVE_GENERIC_GUP.

There was only passing mention of maybe making x86 use RCU, but the
discussion was really about why the IF flag meant that x86 didn't need
to, iirc.

I don't recall us ever discussing *really* making x86 use RCU.

                    Linus

[toc] | [prev] | [next] | [standalone]

#1709823 — Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush

On Fri, Aug 11, 2017 at 09:16:29AM -0700, Linus Torvalds wrote:
> On Fri, Aug 11, 2017 at 2:03 AM, Peter Zijlstra <peterz@infradead.org> wrote:
> >
> > I'm sure we talked about using HAVE_RCU_TABLE_FREE for x86 (and yes that
> > would make it work again), but this was some years ago and I cannot
> > readily find those emails.
> 
> I think the only time we really talked about HAVE_RCU_TABLE_FREE for
> x86 (at least that I was cc'd on) was not because of RCU freeing, but
> because we just wanted to use the generic page table lookup code on
> x86 *despite* not using RCU freeing.
> 
> And we just ended up renaming HAVE_GENERIC_RCU_GUP as HAVE_GENERIC_GUP.
> 
> There was only passing mention of maybe making x86 use RCU, but the
> discussion was really about why the IF flag meant that x86 didn't need
> to, iirc.
> 
> I don't recall us ever discussing *really* making x86 use RCU.

Google finds me this:

  https://lwn.net/Articles/500188/

Which includes:

  http://www.mail-archive.com/kvm@vger.kernel.org/msg72918.html

which does as was suggested here, selects HAVE_RCU_TABLE_FREE for
PARAVIRT_TLB_FLUSH.

But yes, this is very much virt specific nonsense, native would never
need this.

[toc] | [prev] | [next] | [standalone]

#1710921 — Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush

Peter Zijlstra <peterz@infradead.org> writes:

> On Fri, Aug 11, 2017 at 09:16:29AM -0700, Linus Torvalds wrote:
>> On Fri, Aug 11, 2017 at 2:03 AM, Peter Zijlstra <peterz@infradead.org> wrote:
>> >
>> > I'm sure we talked about using HAVE_RCU_TABLE_FREE for x86 (and yes that
>> > would make it work again), but this was some years ago and I cannot
>> > readily find those emails.
>> 
>> I think the only time we really talked about HAVE_RCU_TABLE_FREE for
>> x86 (at least that I was cc'd on) was not because of RCU freeing, but
>> because we just wanted to use the generic page table lookup code on
>> x86 *despite* not using RCU freeing.
>> 
>> And we just ended up renaming HAVE_GENERIC_RCU_GUP as HAVE_GENERIC_GUP.
>> 
>> There was only passing mention of maybe making x86 use RCU, but the
>> discussion was really about why the IF flag meant that x86 didn't need
>> to, iirc.
>> 
>> I don't recall us ever discussing *really* making x86 use RCU.
>
> Google finds me this:
>
>   https://lwn.net/Articles/500188/
>
> Which includes:
>
>   http://www.mail-archive.com/kvm@vger.kernel.org/msg72918.html
>
> which does as was suggested here, selects HAVE_RCU_TABLE_FREE for
> PARAVIRT_TLB_FLUSH.
>
> But yes, this is very much virt specific nonsense, native would never
> need this.

In case we decide to go HAVE_RCU_TABLE_FREE for all PARAVIRT-enabled
kernels (as it seems to be the easiest/fastest way to fix Xen PV) - what
do you think about the required testing? Any suggestion for a
specifically crafted micro benchmark in addition to standard
ebizzy/kernbench/...?

Additionally, I see another option for us: enable 'rcu table free' on
boot (e.g. by taking tlb_remove_table to pv_ops and doing boot-time
patching for it) so bare metal and other hypervisors are not affected
by the change.

-- 
  Vitaly

[toc] | [prev] | [next] | [standalone]

#1713121 — Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush

[Multipart message — attachments visible in raw view] — view raw

Vitaly Kuznetsov <vkuznets@redhat.com> writes:

> Peter Zijlstra <peterz@infradead.org> writes:
>
>> On Fri, Aug 11, 2017 at 09:16:29AM -0700, Linus Torvalds wrote:
>>> On Fri, Aug 11, 2017 at 2:03 AM, Peter Zijlstra <peterz@infradead.org> wrote:
>>> >
>>> > I'm sure we talked about using HAVE_RCU_TABLE_FREE for x86 (and yes that
>>> > would make it work again), but this was some years ago and I cannot
>>> > readily find those emails.
>>> 
>>> I think the only time we really talked about HAVE_RCU_TABLE_FREE for
>>> x86 (at least that I was cc'd on) was not because of RCU freeing, but
>>> because we just wanted to use the generic page table lookup code on
>>> x86 *despite* not using RCU freeing.
>>> 
>>> And we just ended up renaming HAVE_GENERIC_RCU_GUP as HAVE_GENERIC_GUP.
>>> 
>>> There was only passing mention of maybe making x86 use RCU, but the
>>> discussion was really about why the IF flag meant that x86 didn't need
>>> to, iirc.
>>> 
>>> I don't recall us ever discussing *really* making x86 use RCU.
>>
>> Google finds me this:
>>
>>   https://lwn.net/Articles/500188/
>>
>> Which includes:
>>
>>   http://www.mail-archive.com/kvm@vger.kernel.org/msg72918.html
>>
>> which does as was suggested here, selects HAVE_RCU_TABLE_FREE for
>> PARAVIRT_TLB_FLUSH.
>>
>> But yes, this is very much virt specific nonsense, native would never
>> need this.
>
> In case we decide to go HAVE_RCU_TABLE_FREE for all PARAVIRT-enabled
> kernels (as it seems to be the easiest/fastest way to fix Xen PV) - what
> do you think about the required testing? Any suggestion for a
> specifically crafted micro benchmark in addition to standard
> ebizzy/kernbench/...?

In the meantime I tested HAVE_RCU_TABLE_FREE with kernbench (enablement
patch I used is attached; I know that it breaks other architectures) on
bare metal with PARAVIRT enabled in config. The results are:

6-CPU host:

Average Half load -j 3 Run (std deviation):
CURRENT					HAVE_RCU_TABLE_FREE
=======					===================
Elapsed Time 400.498 (0.179679)		Elapsed Time 399.909 (0.162853)
User Time 1098.72 (0.278536)		User Time 1097.59 (0.283894)
System Time 100.301 (0.201629)		System Time 99.736 (0.196254)
Percent CPU 299 (0)			Percent CPU 299 (0)
Context Switches 5774.1 (69.2121)	Context Switches 5744.4 (79.4162)
Sleeps 87621.2 (78.1093)		Sleeps 87586.1 (99.7079)

Average Optimal load -j 24 Run (std deviation):
CURRENT					HAVE_RCU_TABLE_FREE
=======					===================
Elapsed Time 219.03 (0.652534)		Elapsed Time 218.959 (0.598674)
User Time 1119.51 (21.3284)		User Time 1118.81 (21.7793)
System Time 100.499 (0.389308)		System Time 99.8335 (0.251423)
Percent CPU 432.5 (136.974)		Percent CPU 432.45 (136.922)
Context Switches 81827.4 (78029.5)	Context Switches 81818.5 (78051)
Sleeps 97124.8 (9822.4)			Sleeps 97207.9 (9955.04)

16-CPU host:

Average Half load -j 8 Run (std deviation):
CURRENT					HAVE_RCU_TABLE_FREE
=======					===================
Elapsed Time 213.538 (3.7891)		Elapsed Time 212.5 (3.10939)
User Time 1306.4 (1.83399)		User Time 1307.65 (1.01364)
System Time 194.59 (0.864378)		System Time 195.478 (0.794588)
Percent CPU 702.6 (13.5388)		Percent CPU 707 (11.1131)
Context Switches 21189.2 (1199.4)	Context Switches 21288.2 (552.388)
Sleeps 89390.2 (482.325)		Sleeps 89677 (277.06)

Average Optimal load -j 64 Run (std deviation):
CURRENT					HAVE_RCU_TABLE_FREE
=======					===================
Elapsed Time 137.866 (0.787928)		Elapsed Time 138.438 (0.218792)
User Time 1488.92 (192.399)		User Time 1489.92 (192.135)
System Time 234.981 (42.5806)		System Time 236.09 (42.8138)
Percent CPU 1057.1 (373.826)		Percent CPU 1057.1 (369.114)
Context Switches 187514 (175324)	Context Switches 187358 (175060)
Sleeps 112633 (24535.5)			Sleeps 111743 (23297.6)

As you can see, there's no notable difference. I'll think of a
microbenchmark though.

>
> Additionally, I see another option for us: enable 'rcu table free' on
> boot (e.g. by taking tlb_remove_table to pv_ops and doing boot-time
> patching for it) so bare metal and other hypervisors are not affected
> by the change.

It seems there's no need for that and we can keep things simple...

-- 
  Vitaly

[toc] | [prev] | [next] | [standalone]

#1713350 — Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush

On 08/16/2017 12:42 PM, Vitaly Kuznetsov wrote:
> Vitaly Kuznetsov <vkuznets@redhat.com> writes:
>
>> Peter Zijlstra <peterz@infradead.org> writes:
>>
>>> On Fri, Aug 11, 2017 at 09:16:29AM -0700, Linus Torvalds wrote:
>>>> On Fri, Aug 11, 2017 at 2:03 AM, Peter Zijlstra <peterz@infradead.org> wrote:
>>>>> I'm sure we talked about using HAVE_RCU_TABLE_FREE for x86 (and yes that
>>>>> would make it work again), but this was some years ago and I cannot
>>>>> readily find those emails.
>>>> I think the only time we really talked about HAVE_RCU_TABLE_FREE for
>>>> x86 (at least that I was cc'd on) was not because of RCU freeing, but
>>>> because we just wanted to use the generic page table lookup code on
>>>> x86 *despite* not using RCU freeing.
>>>>
>>>> And we just ended up renaming HAVE_GENERIC_RCU_GUP as HAVE_GENERIC_GUP.
>>>>
>>>> There was only passing mention of maybe making x86 use RCU, but the
>>>> discussion was really about why the IF flag meant that x86 didn't need
>>>> to, iirc.
>>>>
>>>> I don't recall us ever discussing *really* making x86 use RCU.
>>> Google finds me this:
>>>
>>>   https://lwn.net/Articles/500188/
>>>
>>> Which includes:
>>>
>>>   http://www.mail-archive.com/kvm@vger.kernel.org/msg72918.html
>>>
>>> which does as was suggested here, selects HAVE_RCU_TABLE_FREE for
>>> PARAVIRT_TLB_FLUSH.
>>>
>>> But yes, this is very much virt specific nonsense, native would never
>>> need this.
>> In case we decide to go HAVE_RCU_TABLE_FREE for all PARAVIRT-enabled
>> kernels (as it seems to be the easiest/fastest way to fix Xen PV) - what
>> do you think about the required testing? Any suggestion for a
>> specifically crafted micro benchmark in addition to standard
>> ebizzy/kernbench/...?
> In the meantime I tested HAVE_RCU_TABLE_FREE with kernbench (enablement
> patch I used is attached; I know that it breaks other architectures) on
> bare metal with PARAVIRT enabled in config. The results are:
>
> 6-CPU host:
>
> Average Half load -j 3 Run (std deviation):
> CURRENT					HAVE_RCU_TABLE_FREE
> =======					===================
> Elapsed Time 400.498 (0.179679)		Elapsed Time 399.909 (0.162853)
> User Time 1098.72 (0.278536)		User Time 1097.59 (0.283894)
> System Time 100.301 (0.201629)		System Time 99.736 (0.196254)
> Percent CPU 299 (0)			Percent CPU 299 (0)
> Context Switches 5774.1 (69.2121)	Context Switches 5744.4 (79.4162)
> Sleeps 87621.2 (78.1093)		Sleeps 87586.1 (99.7079)
>
> Average Optimal load -j 24 Run (std deviation):
> CURRENT					HAVE_RCU_TABLE_FREE
> =======					===================
> Elapsed Time 219.03 (0.652534)		Elapsed Time 218.959 (0.598674)
> User Time 1119.51 (21.3284)		User Time 1118.81 (21.7793)
> System Time 100.499 (0.389308)		System Time 99.8335 (0.251423)
> Percent CPU 432.5 (136.974)		Percent CPU 432.45 (136.922)
> Context Switches 81827.4 (78029.5)	Context Switches 81818.5 (78051)
> Sleeps 97124.8 (9822.4)			Sleeps 97207.9 (9955.04)
>
> 16-CPU host:
>
> Average Half load -j 8 Run (std deviation):
> CURRENT					HAVE_RCU_TABLE_FREE
> =======					===================
> Elapsed Time 213.538 (3.7891)		Elapsed Time 212.5 (3.10939)
> User Time 1306.4 (1.83399)		User Time 1307.65 (1.01364)
> System Time 194.59 (0.864378)		System Time 195.478 (0.794588)
> Percent CPU 702.6 (13.5388)		Percent CPU 707 (11.1131)
> Context Switches 21189.2 (1199.4)	Context Switches 21288.2 (552.388)
> Sleeps 89390.2 (482.325)		Sleeps 89677 (277.06)
>
> Average Optimal load -j 64 Run (std deviation):
> CURRENT					HAVE_RCU_TABLE_FREE
> =======					===================
> Elapsed Time 137.866 (0.787928)		Elapsed Time 138.438 (0.218792)
> User Time 1488.92 (192.399)		User Time 1489.92 (192.135)
> System Time 234.981 (42.5806)		System Time 236.09 (42.8138)
> Percent CPU 1057.1 (373.826)		Percent CPU 1057.1 (369.114)
> Context Switches 187514 (175324)	Context Switches 187358 (175060)
> Sleeps 112633 (24535.5)			Sleeps 111743 (23297.6)
>
> As you can see, there's no notable difference. I'll think of a
> microbenchmark though.

FWIW, I was about to send a very similar patch (but with only Xen-PV
enabling RCU-based free by default) and saw similar results with
kernbench, both Xen PV and baremetal.

>> Additionally, I see another option for us: enable 'rcu table free' on
>> boot (e.g. by taking tlb_remove_table to pv_ops and doing boot-time
>> patching for it) so bare metal and other hypervisors are not affected
>> by the change.
> It seems there's no need for that and we can keep things simple...
>
> -- Vitaly
>
> 0001-x86-enable-RCU-based-table-free-when-PARAVIRT.patch
>
>
> >From daf5117706920aebe793d1239fccac2edd4d680c Mon Sep 17 00:00:00 2001
> From: Vitaly Kuznetsov <vkuznets@redhat.com>
> Date: Mon, 14 Aug 2017 16:05:05 +0200
> Subject: [PATCH] x86: enable RCU based table free when PARAVIRT
>
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  arch/x86/Kconfig      |  1 +
>  arch/x86/mm/pgtable.c | 16 ++++++++++++++++
>  mm/memory.c           |  5 +++++
>  3 files changed, 22 insertions(+)
>
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 781521b7cf9e..9c1666ea04c9 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -167,6 +167,7 @@ config X86
>  	select HAVE_PERF_REGS
>  	select HAVE_PERF_USER_STACK_DUMP
>  	select HAVE_REGS_AND_STACK_ACCESS_API
> +	select HAVE_RCU_TABLE_FREE              if SMP && PARAVIRT
>  	select HAVE_RELIABLE_STACKTRACE		if X86_64 && FRAME_POINTER && STACK_VALIDATION
>  	select HAVE_STACK_VALIDATION		if X86_64
>  	select HAVE_SYSCALL_TRACEPOINTS
> diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
> index 508a708eb9a6..b6aaab9fb3b8 100644
> --- a/arch/x86/mm/pgtable.c
> +++ b/arch/x86/mm/pgtable.c
> @@ -56,7 +56,11 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte)
>  {
>  	pgtable_page_dtor(pte);
>  	paravirt_release_pte(page_to_pfn(pte));
> +#ifdef CONFIG_HAVE_RCU_TABLE_FREE
> +	tlb_remove_table(tlb, pte);
> +#else
>  	tlb_remove_page(tlb, pte);
> +#endif
>  }
>  
>  #if CONFIG_PGTABLE_LEVELS > 2
> @@ -72,21 +76,33 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd)
>  	tlb->need_flush_all = 1;
>  #endif
>  	pgtable_pmd_page_dtor(page);
> +#ifdef CONFIG_HAVE_RCU_TABLE_FREE
> +	tlb_remove_table(tlb, page);
> +#else
>  	tlb_remove_page(tlb, page);
> +#endif
>  }
>  
>  #if CONFIG_PGTABLE_LEVELS > 3
>  void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud)
>  {
>  	paravirt_release_pud(__pa(pud) >> PAGE_SHIFT);
> +#ifdef CONFIG_HAVE_RCU_TABLE_FREE
> +	tlb_remove_table(tlb, virt_to_page(pud));
> +#else
>  	tlb_remove_page(tlb, virt_to_page(pud));
> +#endif
>  }
>  
>  #if CONFIG_PGTABLE_LEVELS > 4
>  void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d)
>  {
>  	paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT);
> +#ifdef CONFIG_HAVE_RCU_TABLE_FREE
> +	tlb_remove_table(tlb, virt_to_page(p4d));
> +#else
>  	tlb_remove_page(tlb, virt_to_page(p4d));
> +#endif

This can probably be factored out.

>  }
>  #endif	/* CONFIG_PGTABLE_LEVELS > 4 */
>  #endif	/* CONFIG_PGTABLE_LEVELS > 3 */
> diff --git a/mm/memory.c b/mm/memory.c
> index e158f7ac6730..18d6671b6ae2 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -329,6 +329,11 @@ bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_
>   * See the comment near struct mmu_table_batch.
>   */
>  
> +static void __tlb_remove_table(void *table)
> +{
> +	free_page_and_swap_cache(table);
> +}
> +

This needs to be a per-arch routine (e.g. see arch/arm64/include/asm/tlb.h).

-boris

[toc] | [prev] | [next] | [standalone]

#1713630 — Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush

Boris Ostrovsky <boris.ostrovsky@oracle.com> writes:

> On 08/16/2017 12:42 PM, Vitaly Kuznetsov wrote:
>> Vitaly Kuznetsov <vkuznets@redhat.com> writes:
>>
>>> In case we decide to go HAVE_RCU_TABLE_FREE for all PARAVIRT-enabled
>>> kernels (as it seems to be the easiest/fastest way to fix Xen PV) - what
>>> do you think about the required testing? Any suggestion for a
>>> specifically crafted micro benchmark in addition to standard
>>> ebizzy/kernbench/...?
>> In the meantime I tested HAVE_RCU_TABLE_FREE with kernbench (enablement
>> patch I used is attached; I know that it breaks other architectures) on
>> bare metal with PARAVIRT enabled in config. The results are:
>>
>>...
>>
>> As you can see, there's no notable difference. I'll think of a
>> microbenchmark though.
>
> FWIW, I was about to send a very similar patch (but with only Xen-PV
> enabling RCU-based free by default) and saw similar results with
> kernbench, both Xen PV and baremetal.
>

Thanks for the confirmation,

I'd go with enabling it for PARAVIRT as we will need it for Hyper-V too.

<snip>

>>  
>>  #if CONFIG_PGTABLE_LEVELS > 4
>>  void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d)
>>  {
>>  	paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT);
>> +#ifdef CONFIG_HAVE_RCU_TABLE_FREE
>> +	tlb_remove_table(tlb, virt_to_page(p4d));
>> +#else
>>  	tlb_remove_page(tlb, virt_to_page(p4d));
>> +#endif
>
> This can probably be factored out.
>
>>  }
>>  #endif	/* CONFIG_PGTABLE_LEVELS > 4 */
>>  #endif	/* CONFIG_PGTABLE_LEVELS > 3 */
>> diff --git a/mm/memory.c b/mm/memory.c
>> index e158f7ac6730..18d6671b6ae2 100644
>> --- a/mm/memory.c
>> +++ b/mm/memory.c
>> @@ -329,6 +329,11 @@ bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_
>>   * See the comment near struct mmu_table_batch.
>>   */
>>  
>> +static void __tlb_remove_table(void *table)
>> +{
>> +	free_page_and_swap_cache(table);
>> +}
>> +
>
> This needs to be a per-arch routine (e.g. see arch/arm64/include/asm/tlb.h).
>

Yea, this was a quick-and-dirty x86-only patch.

-- 
  Vitaly

[toc] | [prev] | [next] | [standalone]

#1709403 — Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush

Peter Zijlstra <peterz@infradead.org> writes:

> On Thu, Aug 10, 2017 at 07:08:22PM +0000, Jork Loeser wrote:
>
>> > > Subject: Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush
>> 
>> > > Hold on.. if we don't IPI for TLB invalidation. What serializes our
>> > > software page table walkers like fast_gup() ?
>> > 
>> > Hypervisor may implement this functionality via an IPI.
>> > 
>> > K. Y
>> 
>> HvFlushVirtualAddressList() states:
>> This call guarantees that by the time control returns back to the
>> caller, the observable effects of all flushes on the specified virtual
>> processors have occurred.
>> 
>> HvFlushVirtualAddressListEx() refers to HvFlushVirtualAddressList() as adding sparse target VP lists.
>> 
>> Is this enough of a guarantee, or do you see other races?
>
> That's nowhere near enough. We need the remote CPU to have completed any
> guest IF section that was in progress at the time of the call.
>
> So if a host IPI can interrupt a guest while the guest has IF cleared,
> and we then process the host IPI -- clear the TLBs -- before resuming the
> guest, which still has IF cleared, we've got a problem.
>
> Because at that point, our software page-table walker, that relies on IF
> being clear to guarantee the page-tables exist, because it holds off the
> TLB invalidate and thereby the freeing of the pages, gets its pages
> ripped out from under it.

Oh, I see your concern. Hyper-V, however, is not the first x86
hypervisor trying to avoid IPIs on remote TLB flush, Xen does this
too. Briefly looking at xen_flush_tlb_others() I don't see anything
special, do we know how serialization is achieved there?

-- 
  Vitaly

[toc] | [prev] | [next] | [standalone]

#1709457 — Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush

On Fri, Aug 11, 2017 at 11:23:10AM +0200, Vitaly Kuznetsov wrote:
> Peter Zijlstra <peterz@infradead.org> writes:
> 
> > On Thu, Aug 10, 2017 at 07:08:22PM +0000, Jork Loeser wrote:
> >
> >> > > Subject: Re: [tip:x86/platform] x86/hyper-v: Use hypercall for remote TLB flush
> >> 
> >> > > Hold on.. if we don't IPI for TLB invalidation. What serializes our
> >> > > software page table walkers like fast_gup() ?
> >> > 
> >> > Hypervisor may implement this functionality via an IPI.
> >> > 
> >> > K. Y
> >> 
> >> HvFlushVirtualAddressList() states:
> >> This call guarantees that by the time control returns back to the
> >> caller, the observable effects of all flushes on the specified virtual
> >> processors have occurred.
> >> 
> >> HvFlushVirtualAddressListEx() refers to HvFlushVirtualAddressList() as adding sparse target VP lists.
> >> 
> >> Is this enough of a guarantee, or do you see other races?
> >
> > That's nowhere near enough. We need the remote CPU to have completed any
> > guest IF section that was in progress at the time of the call.
> >
> > So if a host IPI can interrupt a guest while the guest has IF cleared,
> > and we then process the host IPI -- clear the TLBs -- before resuming the
> > guest, which still has IF cleared, we've got a problem.
> >
> > Because at that point, our software page-table walker, that relies on IF
> > being clear to guarantee the page-tables exist, because it holds off the
> > TLB invalidate and thereby the freeing of the pages, gets its pages
> > ripped out from under it.
> 
> Oh, I see your concern. Hyper-V, however, is not the first x86
> hypervisor trying to avoid IPIs on remote TLB flush, Xen does this
> too. Briefly looking at xen_flush_tlb_others() I don't see anything
> special, do we know how serialization is achieved there?

No idea on how Xen works, I always just hope it goes away :-) But lets
ask some Xen folks.

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | linux.kernel

csiph-web