Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1533075 > unrolled thread

[PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration

Back to article view | Back to linux.kernel

Contents

  [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Liang Li <liang.z.li@intel.com> - 2016-11-30 10:00 +0100
    [PATCH kernel v5 5/5] virtio-balloon: tell host vm's unused page info Liang Li <liang.z.li@intel.com> - 2016-11-30 10:00 +0100
      Re: [PATCH kernel v5 5/5] virtio-balloon: tell host vm's unused page  info Dave Hansen <dave.hansen@intel.com> - 2016-11-30 20:20 +0100
        RE: [PATCH kernel v5 5/5] virtio-balloon: tell host vm's unused  page info "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-04 14:20 +0100
          Re: [PATCH kernel v5 5/5] virtio-balloon: tell host vm's unused page  info Dave Hansen <dave.hansen@intel.com> - 2016-12-05 18:30 +0100
            RE: [PATCH kernel v5 5/5] virtio-balloon: tell host vm's unused  page info "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-06 05:50 +0100
    [PATCH kernel v5 2/5] virtio-balloon: define new feature bit and head struct Liang Li <liang.z.li@intel.com> - 2016-11-30 10:00 +0100
    [PATCH kernel v5 4/5] virtio-balloon: define flags and head for host request vq Liang Li <liang.z.li@intel.com> - 2016-11-30 10:00 +0100
    [PATCH kernel v5 1/5] virtio-balloon: rework deflate to add page to a list Liang Li <liang.z.li@intel.com> - 2016-11-30 10:00 +0100
    Re: [PATCH kernel v5 0/5] Extend virtio-balloon for fast  (de)inflating & fast live migration David Hildenbrand <david@redhat.com> - 2016-12-06 09:50 +0100
      RE: [PATCH kernel v5 0/5] Extend virtio-balloon for fast  (de)inflating & fast live migration "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-07 14:40 +0100
        Re: [PATCH kernel v5 0/5] Extend virtio-balloon for fast  (de)inflating & fast live migration Dave Hansen <dave.hansen@intel.com> - 2016-12-07 16:40 +0100
          RE: [PATCH kernel v5 0/5] Extend virtio-balloon for fast  (de)inflating & fast live migration "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-09 04:10 +0100
        Re: [PATCH kernel v5 0/5] Extend virtio-balloon for fast  (de)inflating & fast live migration Dave Hansen <dave.hansen@intel.com> - 2016-12-07 16:50 +0100
          Re: [PATCH kernel v5 0/5] Extend virtio-balloon for fast  (de)inflating & fast live migration David Hildenbrand <david@redhat.com> - 2016-12-07 17:30 +0100
            Re: [PATCH kernel v5 0/5] Extend virtio-balloon for fast  (de)inflating & fast live migration Dave Hansen <dave.hansen@intel.com> - 2016-12-07 18:00 +0100
              Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for  fast (de)inflating & fast live migration Andrea Arcangeli <aarcange@redhat.com> - 2016-12-07 19:40 +0100
                Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast  (de)inflating & fast live migration Dave Hansen <dave.hansen@intel.com> - 2016-12-07 20:00 +0100
                  Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for  fast (de)inflating & fast live migration Andrea Arcangeli <aarcange@redhat.com> - 2016-12-07 20:00 +0100
                Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast  (de)inflating & fast live migration Dave Hansen <dave.hansen@intel.com> - 2016-12-07 21:00 +0100
                  Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for  fast (de)inflating & fast live migration Andrea Arcangeli <aarcange@redhat.com> - 2016-12-07 21:30 +0100
                    RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for  fast (de)inflating & fast live migration "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-09 05:50 +0100
                      Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast  (de)inflating & fast live migration Dave Hansen <dave.hansen@intel.com> - 2016-12-09 06:00 +0100
                        RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for  fast (de)inflating & fast live migration "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-09 06:40 +0100
                          Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for  fast (de)inflating & fast live migration Andrea Arcangeli <aarcange@redhat.com> - 2016-12-09 17:50 +0100
        Re: [PATCH kernel v5 0/5] Extend virtio-balloon for fast  (de)inflating & fast live migration David Hildenbrand <david@redhat.com> - 2016-12-07 16:50 +0100

Page 2 of 2 — ← Prev page 1 [2]

#1538059 — Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration

On Wed, Dec 07, 2016 at 11:54:34AM -0800, Dave Hansen wrote:
> We're talking about a bunch of different stuff which is all being
> conflated.  There are 3 issues here that I can see.  I'll attempt to
> summarize what I think is going on:
> 
> 1. Current patches do a hypercall for each order in the allocator.
>    This is inefficient, but independent from the underlying data
>    structure in the ABI, unless bitmaps are in play, which they aren't.
> 2. Should we have bitmaps in the ABI, even if they are not in use by the
>    guest implementation today?  Andrea says they have zero benefits
>    over a pfn/len scheme.  Dave doesn't think they have zero benefits
>    but isn't that attached to them.  QEMU's handling gets more
>    complicated when using a bitmap.
> 3. Should the ABI contain records each with a pfn/len pair or a
>    pfn/order pair?
>    3a. 'len' is more flexible, but will always be a power-of-two anyway
> 	for high-order pages (the common case)

Len wouldn't be a power of two practically only if we detect adjacent
pages of smaller order that may merge into larger orders we already
allocated (or the other way around).

[addr=2M, len=2M] allocated at order 9 pass
[addr=4M, len=1M] allocated at order 8 pass -> merge as [addr=2M, len=3M]

Not sure if it would be worth it, but that unless we do this, page-order or
len won't make much difference.

>    3b. if we decide not to have a bitmap, then we basically have plenty
> 	of space for 'len' and should just do it
>    3c. It's easiest for the hypervisor to turn pfn/len into the
>        madvise() calls that it needs.
> 
> Did I miss anything?

I think you summarized fine all my arguments in your summary.

> FWIW, I don't feel that strongly about the bitmap.  Li had one
> originally, but I think the code thus far has demonstrated a huge
> benefit without even having a bitmap.
> 
> I've got no objections to ripping the bitmap out of the ABI.

I think we need to see a statistic showing the number of bits set in
each bitmap in average, after some uptime and lru churn, like running
stresstest app for a while with I/O and then inflate the balloon and
count:

1) how many bits were set vs total number of bits used in bitmaps

2) how many times bitmaps were used vs bitmap_len = 0 case of single
   page

My guess would be like very low percentage for both points.

> Surely we can think of a few ways...
> 
> A bitmap is 64x more dense if the lists are unordered.  It means being
> able to store ~32k*2M=64G worth of 2M pages in one data page vs. ~1G.
> That's 64x fewer cachelines to touch, 64x fewer pages to move to the
> hypervisor and lets us allocate 1/64th the memory.  Given a maximum
> allocation that we're allowed, it lets us do 64x more per-pass.
> 
> Now, are those benefits worth it?  Maybe not, but let's not pretend they
> don't exist. ;)

In the best case there are benefits obviously, the question is how
common the best case is.

The best case if I understand correctly is all high order not
available, but plenty of order 0 pages available at phys address X,
X+8k, X+16k, X+(8k*nr_bits_in_bitmap). How common is that 0 pages
exist but they're not at an address < X or > X+(8k*nr_bits_in_bitmap)?

> Yes, the current code sends one batch of pages up to the hypervisor per
> order.  But, this has nothing to do with the underlying data structure,
> or the choice to have an order vs. len in the ABI.
> 
> What you describe here is obviously more efficient.

And it isn't possible with the current ABI.

So there is a connection with the MAX_ORDER..0 allocation loop and the
ABI change, but I agree any of the ABI proposed would still allow for
it this logic to be used. Bitmap or not bitmap, the loop would still
work.

[toc] | [prev] | [next] | [standalone]

#1539068 — RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration

> > 1. Current patches do a hypercall for each order in the allocator.
> >    This is inefficient, but independent from the underlying data
> >    structure in the ABI, unless bitmaps are in play, which they aren't.
> > 2. Should we have bitmaps in the ABI, even if they are not in use by the
> >    guest implementation today?  Andrea says they have zero benefits
> >    over a pfn/len scheme.  Dave doesn't think they have zero benefits
> >    but isn't that attached to them.  QEMU's handling gets more
> >    complicated when using a bitmap.
> > 3. Should the ABI contain records each with a pfn/len pair or a
> >    pfn/order pair?
> >    3a. 'len' is more flexible, but will always be a power-of-two anyway
> > 	for high-order pages (the common case)
> 
> Len wouldn't be a power of two practically only if we detect adjacent pages
> of smaller order that may merge into larger orders we already allocated (or
> the other way around).
> 
> [addr=2M, len=2M] allocated at order 9 pass [addr=4M, len=1M] allocated at
> order 8 pass -> merge as [addr=2M, len=3M]
> 
> Not sure if it would be worth it, but that unless we do this, page-order or len
> won't make much difference.
> 
> >    3b. if we decide not to have a bitmap, then we basically have plenty
> > 	of space for 'len' and should just do it
> >    3c. It's easiest for the hypervisor to turn pfn/len into the
> >        madvise() calls that it needs.
> >
> > Did I miss anything?
> 
> I think you summarized fine all my arguments in your summary.
> 
> > FWIW, I don't feel that strongly about the bitmap.  Li had one
> > originally, but I think the code thus far has demonstrated a huge
> > benefit without even having a bitmap.
> >
> > I've got no objections to ripping the bitmap out of the ABI.
> 
> I think we need to see a statistic showing the number of bits set in each
> bitmap in average, after some uptime and lru churn, like running stresstest
> app for a while with I/O and then inflate the balloon and
> count:
> 
> 1) how many bits were set vs total number of bits used in bitmaps
> 
> 2) how many times bitmaps were used vs bitmap_len = 0 case of single
>    page
> 
> My guess would be like very low percentage for both points.
> 

> So there is a connection with the MAX_ORDER..0 allocation loop and the ABI
> change, but I agree any of the ABI proposed would still allow for it this logic to
> be used. Bitmap or not bitmap, the loop would still work.

Hi guys,

What's the conclusion of your discussion? 
It seems you want some statistic before deciding whether to  ripping the bitmap from the ABI, am I right?

Thanks!
Liang 

[toc] | [prev] | [next] | [standalone]

#1539070 — Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration

On 12/08/2016 08:45 PM, Li, Liang Z wrote:
> What's the conclusion of your discussion? It seems you want some
> statistic before deciding whether to  ripping the bitmap from the
> ABI, am I right?

I think Andrea and David feel pretty strongly that we should remove the
bitmap, unless we have some data to support keeping it.  I don't feel as
strongly about it, but I think their critique of it is pretty valid.  I
think the consensus is that the bitmap needs to go.

The only real question IMNHO is whether we should do a power-of-2 or a
length.  But, if we have 12 bits, then the argument for doing length is
pretty strong.  We don't need anywhere near 12 bits if doing power-of-2.

[toc] | [prev] | [next] | [standalone]

#1539104 — RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration

> On 12/08/2016 08:45 PM, Li, Liang Z wrote:
> > What's the conclusion of your discussion? It seems you want some
> > statistic before deciding whether to  ripping the bitmap from the ABI,
> > am I right?
> 
> I think Andrea and David feel pretty strongly that we should remove the
> bitmap, unless we have some data to support keeping it.  I don't feel as
> strongly about it, but I think their critique of it is pretty valid.  I think the
> consensus is that the bitmap needs to go.
> 

Thanks for you clarification.

> The only real question IMNHO is whether we should do a power-of-2 or a
> length.  But, if we have 12 bits, then the argument for doing length is pretty
> strong.  We don't need anywhere near 12 bits if doing power-of-2.
> 
So each item can max represent 16MB Bytes, seems not big enough,
but enough for most case.
Things became much more simple without the bitmap, and I like simple solution too. :)

I will prepare the v6 and remove all the bitmap related stuffs. Thank you all!

Liang

[toc] | [prev] | [next] | [standalone]

#1539552 — Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration

Hello,

On Fri, Dec 09, 2016 at 05:35:45AM +0000, Li, Liang Z wrote:
> > On 12/08/2016 08:45 PM, Li, Liang Z wrote:
> > > What's the conclusion of your discussion? It seems you want some
> > > statistic before deciding whether to  ripping the bitmap from the ABI,
> > > am I right?
> > 
> > I think Andrea and David feel pretty strongly that we should remove the
> > bitmap, unless we have some data to support keeping it.  I don't feel as
> > strongly about it, but I think their critique of it is pretty valid.  I think the
> > consensus is that the bitmap needs to go.
> > 
> 
> Thanks for you clarification.
> 
> > The only real question IMNHO is whether we should do a power-of-2 or a
> > length.  But, if we have 12 bits, then the argument for doing length is pretty
> > strong.  We don't need anywhere near 12 bits if doing power-of-2.
> > 
> So each item can max represent 16MB Bytes, seems not big enough,
> but enough for most case.
> Things became much more simple without the bitmap, and I like simple solution too. :)
> 
> I will prepare the v6 and remove all the bitmap related stuffs. Thank you all!

Sounds great!

I suggested to check the statistics, because collecting those stats
looked simpler and quicker than removing all bitmap related stuff from
the patchset. However if you prefer to prepare a v6 without the bitmap
another perhaps more interesting way to evaluate the usefulness of the
bitmap is to just run the same benchmark and verify that there is no
regression compared to the bitmap enabled code.

The other issue with the bitmap is, the best case for the bitmap is
ever less likely to materialize the more RAM is added to the guest. It
won't regress linearly because after all there can be some locality
bias in the buddy splits, but if sync compaction is used in the large
order allocations tried before reaching order 0, the bitmap payoff
will regress close to linearly with the increase of RAM.

So it'd be good to check the stats or the benchmark on large guests,
at least one hundred gigabytes or so.

Changing topic but still about the ABI features needed, so it may be
relevant for this discussion:

1) vNUMA locality: i.e. allowing host to specify which vNODEs to take
   memory from, using alloc_pages_node in guest. So you can ask to
   take X pages from vnode A, Y pages from vnode B, in one vmenter.

2) allowing qemu to tell the guest to stop inflating the balloon and
   report a fragmentation limit being hit, when sync compaction
   powered allocations fails at certain power-of-two order granularity
   passed by qemu to the guest. This order constraint will be passed
   by default for hugetlbfs guests with 2MB hpage size, while it can
   be used optionally on THP backed guests. This option with THP
   guests would allow a highlevel management software to provide a
   "don't reduce guest performance" while shrinking the memory size of
   the guest from the GUI. If you deselect the option, you can shrink
   down to the last freeable 4k guest page, but doing so may have to
   split THP in the host (you don't know for sure if they were really
   THP but they could have been), and it may regress
   performance. Inflating the balloon while passing a minimum
   granularity "order" of the pages being zapped, will guarantee
   inflating the balloon cannot decrease guest performance
   instead. Plus it's needed for hugetlbfs anyway as far as I can
   tell. hugetlbfs would not be host enforceable even if the idea is
   not to free memory but only reduce the available memory of the
   guest (not without major changes that maps a hugetlb page with 4k
   ptes at least). While for a more cooperative usage of hugetlbfs
   guests, it's simply not useful to inflate the balloon at anything
   less than the "HPAGE_SIZE" hugetlbfs granularity.

We also plan to use userfaultfd to make the balloon driver host
enforced (will work fine on hugetlbfs 2M and tmpfs too) but that's
going to be invisible to the ABI so it's not strictly relevant for
this discussion.

On a side note, registering userfaultfd on the ballooned range, will
keep khugepaged at bay so it won't risk to re-inflating the
MADV_DONTNEED zapped sub-THP fragments no matter the sysfs tunings.

Thanks!
Andrea

[toc] | [prev] | [next] | [standalone]

#1537860 — Re: [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration

Am 07.12.2016 um 14:35 schrieb Li, Liang Z:
>> Am 30.11.2016 um 09:43 schrieb Liang Li:
>>> This patch set contains two parts of changes to the virtio-balloon.
>>>
>>> One is the change for speeding up the inflating & deflating process,
>>> the main idea of this optimization is to use bitmap to send the page
>>> information to host instead of the PFNs, to reduce the overhead of
>>> virtio data transmission, address translation and madvise(). This can
>>> help to improve the performance by about 85%.
>>
>> Do you have some statistics/some rough feeling how many consecutive bits are
>> usually set in the bitmaps? Is it really just purely random or is there some
>> granularity that is usually consecutive?
>>
>
> I did something similar. Filled the balloon with 15GB for a 16GB idle guest, by
> using bitmap, the madvise count was reduced to 605. when using the PFNs, the madvise count
> was 3932160. It means there are quite a lot consecutive bits in the bitmap.
> I didn't test for a guest with heavy memory workload.

Would it then even make sense to go one step further and report {pfn, 
length} combinations?

So simply send over an array of {pfn, length}?

This idea came up when talking to Andrea Arcangeli (put him on cc).

And it makes sense if you think about:

a) hugetlb backing: The host may only be able to free huge pages (we 
might want to communicate that to the guest later, that's another 
story). Still we would have to send bitmaps full of 4k frames (512 bits 
for 2mb frames). Of course, we could add a way to communicate that we 
are using a different bitmap-granularity.

b) if we really inflate huge memory regions (and it sounds like that 
according to your measurements), we can minimize the communication to 
the hypervisor and therefore the madvice calls.

c) we don't want to optimize for inflating guests with almost full 
memory (and therefore little consecutive memory areas) - my opinion :)


Thanks for the explanation!

-- 

David

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web