Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1533075 > unrolled thread
| Started by | Liang Li <liang.z.li@intel.com> |
|---|---|
| First post | 2016-11-30 10:00 +0100 |
| Last post | 2016-12-07 16:50 +0100 |
| Articles | 20 on this page of 40 — 6 participants |
Back to article view | Back to linux.kernel
[PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Liang Li <liang.z.li@intel.com> - 2016-11-30 10:00 +0100
[PATCH kernel v5 5/5] virtio-balloon: tell host vm's unused page info Liang Li <liang.z.li@intel.com> - 2016-11-30 10:00 +0100
Re: [PATCH kernel v5 5/5] virtio-balloon: tell host vm's unused page info Dave Hansen <dave.hansen@intel.com> - 2016-11-30 20:20 +0100
RE: [PATCH kernel v5 5/5] virtio-balloon: tell host vm's unused page info "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-04 14:20 +0100
Re: [PATCH kernel v5 5/5] virtio-balloon: tell host vm's unused page info Dave Hansen <dave.hansen@intel.com> - 2016-12-05 18:30 +0100
RE: [PATCH kernel v5 5/5] virtio-balloon: tell host vm's unused page info "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-06 05:50 +0100
[PATCH kernel v5 2/5] virtio-balloon: define new feature bit and head struct Liang Li <liang.z.li@intel.com> - 2016-11-30 10:00 +0100
[PATCH kernel v5 4/5] virtio-balloon: define flags and head for host request vq Liang Li <liang.z.li@intel.com> - 2016-11-30 10:00 +0100
[PATCH kernel v5 1/5] virtio-balloon: rework deflate to add page to a list Liang Li <liang.z.li@intel.com> - 2016-11-30 10:00 +0100
Re: [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration David Hildenbrand <david@redhat.com> - 2016-12-06 09:50 +0100
RE: [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-07 14:40 +0100
Re: [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Dave Hansen <dave.hansen@intel.com> - 2016-12-07 16:40 +0100
RE: [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-09 04:10 +0100
Re: [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Dave Hansen <dave.hansen@intel.com> - 2016-12-07 16:50 +0100
Re: [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration David Hildenbrand <david@redhat.com> - 2016-12-07 17:30 +0100
Re: [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Dave Hansen <dave.hansen@intel.com> - 2016-12-07 18:00 +0100
Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Andrea Arcangeli <aarcange@redhat.com> - 2016-12-07 19:40 +0100
Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Dave Hansen <dave.hansen@intel.com> - 2016-12-07 20:00 +0100
Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Andrea Arcangeli <aarcange@redhat.com> - 2016-12-07 20:00 +0100
Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Dave Hansen <dave.hansen@intel.com> - 2016-12-07 21:00 +0100
Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Andrea Arcangeli <aarcange@redhat.com> - 2016-12-07 21:30 +0100
RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-09 05:50 +0100
Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Dave Hansen <dave.hansen@intel.com> - 2016-12-09 06:00 +0100
RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-09 06:40 +0100
Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Andrea Arcangeli <aarcange@redhat.com> - 2016-12-09 17:50 +0100
RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-14 09:30 +0100
RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-14 10:10 +0100
Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Dave Hansen <dave.hansen@intel.com> - 2016-12-15 16:40 +0100
Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration "Michael S. Tsirkin" <mst@redhat.com> - 2016-12-15 17:00 +0100
RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-16 02:20 +0100
Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Andrea Arcangeli <aarcange@redhat.com> - 2016-12-16 16:50 +0100
RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-17 13:00 +0100
RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-16 01:50 +0100
Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Dave Hansen <dave.hansen@intel.com> - 2016-12-16 02:20 +0100
RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-16 02:40 +0100
RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-16 02:50 +0100
Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Dave Hansen <dave.hansen@intel.com> - 2016-12-16 02:50 +0100
Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration Andrea Arcangeli <aarcange@redhat.com> - 2016-12-16 17:10 +0100
RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration "Li, Liang Z" <liang.z.li@intel.com> - 2016-12-17 13:40 +0100
Re: [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration David Hildenbrand <david@redhat.com> - 2016-12-07 16:50 +0100
Page 2 of 2 — ← Prev page 1 [2]
| From | Andrea Arcangeli <aarcange@redhat.com> |
|---|---|
| Date | 2016-12-07 21:30 +0100 |
| Subject | Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sLR0J-4P6-7@gated-at.bofh.it> |
| In reply to | #1538040 |
On Wed, Dec 07, 2016 at 11:54:34AM -0800, Dave Hansen wrote: > We're talking about a bunch of different stuff which is all being > conflated. There are 3 issues here that I can see. I'll attempt to > summarize what I think is going on: > > 1. Current patches do a hypercall for each order in the allocator. > This is inefficient, but independent from the underlying data > structure in the ABI, unless bitmaps are in play, which they aren't. > 2. Should we have bitmaps in the ABI, even if they are not in use by the > guest implementation today? Andrea says they have zero benefits > over a pfn/len scheme. Dave doesn't think they have zero benefits > but isn't that attached to them. QEMU's handling gets more > complicated when using a bitmap. > 3. Should the ABI contain records each with a pfn/len pair or a > pfn/order pair? > 3a. 'len' is more flexible, but will always be a power-of-two anyway > for high-order pages (the common case) Len wouldn't be a power of two practically only if we detect adjacent pages of smaller order that may merge into larger orders we already allocated (or the other way around). [addr=2M, len=2M] allocated at order 9 pass [addr=4M, len=1M] allocated at order 8 pass -> merge as [addr=2M, len=3M] Not sure if it would be worth it, but that unless we do this, page-order or len won't make much difference. > 3b. if we decide not to have a bitmap, then we basically have plenty > of space for 'len' and should just do it > 3c. It's easiest for the hypervisor to turn pfn/len into the > madvise() calls that it needs. > > Did I miss anything? I think you summarized fine all my arguments in your summary. > FWIW, I don't feel that strongly about the bitmap. Li had one > originally, but I think the code thus far has demonstrated a huge > benefit without even having a bitmap. > > I've got no objections to ripping the bitmap out of the ABI. I think we need to see a statistic showing the number of bits set in each bitmap in average, after some uptime and lru churn, like running stresstest app for a while with I/O and then inflate the balloon and count: 1) how many bits were set vs total number of bits used in bitmaps 2) how many times bitmaps were used vs bitmap_len = 0 case of single page My guess would be like very low percentage for both points. > Surely we can think of a few ways... > > A bitmap is 64x more dense if the lists are unordered. It means being > able to store ~32k*2M=64G worth of 2M pages in one data page vs. ~1G. > That's 64x fewer cachelines to touch, 64x fewer pages to move to the > hypervisor and lets us allocate 1/64th the memory. Given a maximum > allocation that we're allowed, it lets us do 64x more per-pass. > > Now, are those benefits worth it? Maybe not, but let's not pretend they > don't exist. ;) In the best case there are benefits obviously, the question is how common the best case is. The best case if I understand correctly is all high order not available, but plenty of order 0 pages available at phys address X, X+8k, X+16k, X+(8k*nr_bits_in_bitmap). How common is that 0 pages exist but they're not at an address < X or > X+(8k*nr_bits_in_bitmap)? > Yes, the current code sends one batch of pages up to the hypervisor per > order. But, this has nothing to do with the underlying data structure, > or the choice to have an order vs. len in the ABI. > > What you describe here is obviously more efficient. And it isn't possible with the current ABI. So there is a connection with the MAX_ORDER..0 allocation loop and the ABI change, but I agree any of the ABI proposed would still allow for it this logic to be used. Bitmap or not bitmap, the loop would still work.
[toc] | [prev] | [next] | [standalone]
| From | "Li, Liang Z" <liang.z.li@intel.com> |
|---|---|
| Date | 2016-12-09 05:50 +0100 |
| Subject | RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sMli9-7cd-11@gated-at.bofh.it> |
| In reply to | #1538059 |
> > 1. Current patches do a hypercall for each order in the allocator. > > This is inefficient, but independent from the underlying data > > structure in the ABI, unless bitmaps are in play, which they aren't. > > 2. Should we have bitmaps in the ABI, even if they are not in use by the > > guest implementation today? Andrea says they have zero benefits > > over a pfn/len scheme. Dave doesn't think they have zero benefits > > but isn't that attached to them. QEMU's handling gets more > > complicated when using a bitmap. > > 3. Should the ABI contain records each with a pfn/len pair or a > > pfn/order pair? > > 3a. 'len' is more flexible, but will always be a power-of-two anyway > > for high-order pages (the common case) > > Len wouldn't be a power of two practically only if we detect adjacent pages > of smaller order that may merge into larger orders we already allocated (or > the other way around). > > [addr=2M, len=2M] allocated at order 9 pass [addr=4M, len=1M] allocated at > order 8 pass -> merge as [addr=2M, len=3M] > > Not sure if it would be worth it, but that unless we do this, page-order or len > won't make much difference. > > > 3b. if we decide not to have a bitmap, then we basically have plenty > > of space for 'len' and should just do it > > 3c. It's easiest for the hypervisor to turn pfn/len into the > > madvise() calls that it needs. > > > > Did I miss anything? > > I think you summarized fine all my arguments in your summary. > > > FWIW, I don't feel that strongly about the bitmap. Li had one > > originally, but I think the code thus far has demonstrated a huge > > benefit without even having a bitmap. > > > > I've got no objections to ripping the bitmap out of the ABI. > > I think we need to see a statistic showing the number of bits set in each > bitmap in average, after some uptime and lru churn, like running stresstest > app for a while with I/O and then inflate the balloon and > count: > > 1) how many bits were set vs total number of bits used in bitmaps > > 2) how many times bitmaps were used vs bitmap_len = 0 case of single > page > > My guess would be like very low percentage for both points. > > So there is a connection with the MAX_ORDER..0 allocation loop and the ABI > change, but I agree any of the ABI proposed would still allow for it this logic to > be used. Bitmap or not bitmap, the loop would still work. Hi guys, What's the conclusion of your discussion? It seems you want some statistic before deciding whether to ripping the bitmap from the ABI, am I right? Thanks! Liang
[toc] | [prev] | [next] | [standalone]
| From | Dave Hansen <dave.hansen@intel.com> |
|---|---|
| Date | 2016-12-09 06:00 +0100 |
| Subject | Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sMlrP-7fm-3@gated-at.bofh.it> |
| In reply to | #1539068 |
On 12/08/2016 08:45 PM, Li, Liang Z wrote: > What's the conclusion of your discussion? It seems you want some > statistic before deciding whether to ripping the bitmap from the > ABI, am I right? I think Andrea and David feel pretty strongly that we should remove the bitmap, unless we have some data to support keeping it. I don't feel as strongly about it, but I think their critique of it is pretty valid. I think the consensus is that the bitmap needs to go. The only real question IMNHO is whether we should do a power-of-2 or a length. But, if we have 12 bits, then the argument for doing length is pretty strong. We don't need anywhere near 12 bits if doing power-of-2.
[toc] | [prev] | [next] | [standalone]
| From | "Li, Liang Z" <liang.z.li@intel.com> |
|---|---|
| Date | 2016-12-09 06:40 +0100 |
| Subject | RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sMm4x-7Lb-15@gated-at.bofh.it> |
| In reply to | #1539070 |
> On 12/08/2016 08:45 PM, Li, Liang Z wrote: > > What's the conclusion of your discussion? It seems you want some > > statistic before deciding whether to ripping the bitmap from the ABI, > > am I right? > > I think Andrea and David feel pretty strongly that we should remove the > bitmap, unless we have some data to support keeping it. I don't feel as > strongly about it, but I think their critique of it is pretty valid. I think the > consensus is that the bitmap needs to go. > Thanks for you clarification. > The only real question IMNHO is whether we should do a power-of-2 or a > length. But, if we have 12 bits, then the argument for doing length is pretty > strong. We don't need anywhere near 12 bits if doing power-of-2. > So each item can max represent 16MB Bytes, seems not big enough, but enough for most case. Things became much more simple without the bitmap, and I like simple solution too. :) I will prepare the v6 and remove all the bitmap related stuffs. Thank you all! Liang
[toc] | [prev] | [next] | [standalone]
| From | Andrea Arcangeli <aarcange@redhat.com> |
|---|---|
| Date | 2016-12-09 17:50 +0100 |
| Subject | Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sMwwW-5CD-13@gated-at.bofh.it> |
| In reply to | #1539104 |
Hello, On Fri, Dec 09, 2016 at 05:35:45AM +0000, Li, Liang Z wrote: > > On 12/08/2016 08:45 PM, Li, Liang Z wrote: > > > What's the conclusion of your discussion? It seems you want some > > > statistic before deciding whether to ripping the bitmap from the ABI, > > > am I right? > > > > I think Andrea and David feel pretty strongly that we should remove the > > bitmap, unless we have some data to support keeping it. I don't feel as > > strongly about it, but I think their critique of it is pretty valid. I think the > > consensus is that the bitmap needs to go. > > > > Thanks for you clarification. > > > The only real question IMNHO is whether we should do a power-of-2 or a > > length. But, if we have 12 bits, then the argument for doing length is pretty > > strong. We don't need anywhere near 12 bits if doing power-of-2. > > > So each item can max represent 16MB Bytes, seems not big enough, > but enough for most case. > Things became much more simple without the bitmap, and I like simple solution too. :) > > I will prepare the v6 and remove all the bitmap related stuffs. Thank you all! Sounds great! I suggested to check the statistics, because collecting those stats looked simpler and quicker than removing all bitmap related stuff from the patchset. However if you prefer to prepare a v6 without the bitmap another perhaps more interesting way to evaluate the usefulness of the bitmap is to just run the same benchmark and verify that there is no regression compared to the bitmap enabled code. The other issue with the bitmap is, the best case for the bitmap is ever less likely to materialize the more RAM is added to the guest. It won't regress linearly because after all there can be some locality bias in the buddy splits, but if sync compaction is used in the large order allocations tried before reaching order 0, the bitmap payoff will regress close to linearly with the increase of RAM. So it'd be good to check the stats or the benchmark on large guests, at least one hundred gigabytes or so. Changing topic but still about the ABI features needed, so it may be relevant for this discussion: 1) vNUMA locality: i.e. allowing host to specify which vNODEs to take memory from, using alloc_pages_node in guest. So you can ask to take X pages from vnode A, Y pages from vnode B, in one vmenter. 2) allowing qemu to tell the guest to stop inflating the balloon and report a fragmentation limit being hit, when sync compaction powered allocations fails at certain power-of-two order granularity passed by qemu to the guest. This order constraint will be passed by default for hugetlbfs guests with 2MB hpage size, while it can be used optionally on THP backed guests. This option with THP guests would allow a highlevel management software to provide a "don't reduce guest performance" while shrinking the memory size of the guest from the GUI. If you deselect the option, you can shrink down to the last freeable 4k guest page, but doing so may have to split THP in the host (you don't know for sure if they were really THP but they could have been), and it may regress performance. Inflating the balloon while passing a minimum granularity "order" of the pages being zapped, will guarantee inflating the balloon cannot decrease guest performance instead. Plus it's needed for hugetlbfs anyway as far as I can tell. hugetlbfs would not be host enforceable even if the idea is not to free memory but only reduce the available memory of the guest (not without major changes that maps a hugetlb page with 4k ptes at least). While for a more cooperative usage of hugetlbfs guests, it's simply not useful to inflate the balloon at anything less than the "HPAGE_SIZE" hugetlbfs granularity. We also plan to use userfaultfd to make the balloon driver host enforced (will work fine on hugetlbfs 2M and tmpfs too) but that's going to be invisible to the ABI so it's not strictly relevant for this discussion. On a side note, registering userfaultfd on the ballooned range, will keep khugepaged at bay so it won't risk to re-inflating the MADV_DONTNEED zapped sub-THP fragments no matter the sysfs tunings. Thanks! Andrea
[toc] | [prev] | [next] | [standalone]
| From | "Li, Liang Z" <liang.z.li@intel.com> |
|---|---|
| Date | 2016-12-14 09:30 +0100 |
| Subject | RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sOd6O-49f-7@gated-at.bofh.it> |
| In reply to | #1539552 |
> fast (de)inflating & fast live migration > > Hello, > > On Fri, Dec 09, 2016 at 05:35:45AM +0000, Li, Liang Z wrote: > > > On 12/08/2016 08:45 PM, Li, Liang Z wrote: > > > > What's the conclusion of your discussion? It seems you want some > > > > statistic before deciding whether to ripping the bitmap from the > > > > ABI, am I right? > > > > > > I think Andrea and David feel pretty strongly that we should remove > > > the bitmap, unless we have some data to support keeping it. I don't > > > feel as strongly about it, but I think their critique of it is > > > pretty valid. I think the consensus is that the bitmap needs to go. > > > > > > > Thanks for you clarification. > > > > > The only real question IMNHO is whether we should do a power-of-2 or > > > a length. But, if we have 12 bits, then the argument for doing > > > length is pretty strong. We don't need anywhere near 12 bits if doing > power-of-2. > > > > > So each item can max represent 16MB Bytes, seems not big enough, but > > enough for most case. > > Things became much more simple without the bitmap, and I like simple > > solution too. :) > > > > I will prepare the v6 and remove all the bitmap related stuffs. Thank you all! > > Sounds great! > > I suggested to check the statistics, because collecting those stats looked > simpler and quicker than removing all bitmap related stuff from the patchset. > However if you prefer to prepare a v6 without the bitmap another perhaps > more interesting way to evaluate the usefulness of the bitmap is to just run > the same benchmark and verify that there is no regression compared to the > bitmap enabled code. > > The other issue with the bitmap is, the best case for the bitmap is ever less > likely to materialize the more RAM is added to the guest. It won't regress > linearly because after all there can be some locality bias in the buddy splits, > but if sync compaction is used in the large order allocations tried before > reaching order 0, the bitmap payoff will regress close to linearly with the > increase of RAM. > > So it'd be good to check the stats or the benchmark on large guests, at least > one hundred gigabytes or so. > > Changing topic but still about the ABI features needed, so it may be relevant > for this discussion: > > 1) vNUMA locality: i.e. allowing host to specify which vNODEs to take > memory from, using alloc_pages_node in guest. So you can ask to > take X pages from vnode A, Y pages from vnode B, in one vmenter. > > 2) allowing qemu to tell the guest to stop inflating the balloon and > report a fragmentation limit being hit, when sync compaction > powered allocations fails at certain power-of-two order granularity > passed by qemu to the guest. This order constraint will be passed > by default for hugetlbfs guests with 2MB hpage size, while it can > be used optionally on THP backed guests. This option with THP > guests would allow a highlevel management software to provide a > "don't reduce guest performance" while shrinking the memory size of > the guest from the GUI. If you deselect the option, you can shrink > down to the last freeable 4k guest page, but doing so may have to > split THP in the host (you don't know for sure if they were really > THP but they could have been), and it may regress > performance. Inflating the balloon while passing a minimum > granularity "order" of the pages being zapped, will guarantee > inflating the balloon cannot decrease guest performance > instead. Plus it's needed for hugetlbfs anyway as far as I can > tell. hugetlbfs would not be host enforceable even if the idea is > not to free memory but only reduce the available memory of the > guest (not without major changes that maps a hugetlb page with 4k > ptes at least). While for a more cooperative usage of hugetlbfs > guests, it's simply not useful to inflate the balloon at anything > less than the "HPAGE_SIZE" hugetlbfs granularity. > > We also plan to use userfaultfd to make the balloon driver host enforced (will > work fine on hugetlbfs 2M and tmpfs too) but that's going to be invisible to > the ABI so it's not strictly relevant for this discussion. > > On a side note, registering userfaultfd on the ballooned range, will keep > khugepaged at bay so it won't risk to re-inflating the MADV_DONTNEED > zapped sub-THP fragments no matter the sysfs tunings. > Thanks for your elaboration! > Thanks! > Andrea
[toc] | [prev] | [next] | [standalone]
| From | "Li, Liang Z" <liang.z.li@intel.com> |
|---|---|
| Date | 2016-12-14 10:10 +0100 |
| Subject | RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sOdJw-4BA-11@gated-at.bofh.it> |
| In reply to | #1539070 |
> Subject: Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for > fast (de)inflating & fast live migration > > On 12/08/2016 08:45 PM, Li, Liang Z wrote: > > What's the conclusion of your discussion? It seems you want some > > statistic before deciding whether to ripping the bitmap from the ABI, > > am I right? > > I think Andrea and David feel pretty strongly that we should remove the > bitmap, unless we have some data to support keeping it. I don't feel as > strongly about it, but I think their critique of it is pretty valid. I think the > consensus is that the bitmap needs to go. > > The only real question IMNHO is whether we should do a power-of-2 or a > length. But, if we have 12 bits, then the argument for doing length is pretty > strong. We don't need anywhere near 12 bits if doing power-of-2. Just found the MAX_ORDER should be limited to 12 if use length instead of order, If the MAX_ORDER is configured to a value bigger than 12, it will make things more complex to handle this case. If use order, we need to break a large memory range whose length is not the power of 2 into several small ranges, it also make the code complex. It seems we leave too many bit for the pfn, and the bits leave for length is not enough, How about keep 45 bits for the pfn and 19 bits for length, 45 bits for pfn can cover 57 bits physical address, that should be enough in the near feature. What's your opinion? thanks! Liang
[toc] | [prev] | [next] | [standalone]
| From | Dave Hansen <dave.hansen@intel.com> |
|---|---|
| Date | 2016-12-15 16:40 +0100 |
| Subject | Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sOGiu-7uh-25@gated-at.bofh.it> |
| In reply to | #1541769 |
On 12/14/2016 12:59 AM, Li, Liang Z wrote: >> Subject: Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for >> fast (de)inflating & fast live migration >> >> On 12/08/2016 08:45 PM, Li, Liang Z wrote: >>> What's the conclusion of your discussion? It seems you want some >>> statistic before deciding whether to ripping the bitmap from the ABI, >>> am I right? >> >> I think Andrea and David feel pretty strongly that we should remove the >> bitmap, unless we have some data to support keeping it. I don't feel as >> strongly about it, but I think their critique of it is pretty valid. I think the >> consensus is that the bitmap needs to go. >> >> The only real question IMNHO is whether we should do a power-of-2 or a >> length. But, if we have 12 bits, then the argument for doing length is pretty >> strong. We don't need anywhere near 12 bits if doing power-of-2. > > Just found the MAX_ORDER should be limited to 12 if use length instead of order, > If the MAX_ORDER is configured to a value bigger than 12, it will make things more > complex to handle this case. > > If use order, we need to break a large memory range whose length is not the power of 2 into several > small ranges, it also make the code complex. I can't imagine it makes the code that much more complex. It adds a for loop. Right? > It seems we leave too many bit for the pfn, and the bits leave for length is not enough, > How about keep 45 bits for the pfn and 19 bits for length, 45 bits for pfn can cover 57 bits > physical address, that should be enough in the near feature. > > What's your opinion? I still think 'order' makes a lot of sense. But, as you say, 57 bits is enough for x86 for a while. Other architectures.... who knows?
[toc] | [prev] | [next] | [standalone]
| From | "Michael S. Tsirkin" <mst@redhat.com> |
|---|---|
| Date | 2016-12-15 17:00 +0100 |
| Subject | Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sOGBP-7B0-9@gated-at.bofh.it> |
| In reply to | #1542835 |
On Thu, Dec 15, 2016 at 07:34:33AM -0800, Dave Hansen wrote: > On 12/14/2016 12:59 AM, Li, Liang Z wrote: > >> Subject: Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for > >> fast (de)inflating & fast live migration > >> > >> On 12/08/2016 08:45 PM, Li, Liang Z wrote: > >>> What's the conclusion of your discussion? It seems you want some > >>> statistic before deciding whether to ripping the bitmap from the ABI, > >>> am I right? > >> > >> I think Andrea and David feel pretty strongly that we should remove the > >> bitmap, unless we have some data to support keeping it. I don't feel as > >> strongly about it, but I think their critique of it is pretty valid. I think the > >> consensus is that the bitmap needs to go. > >> > >> The only real question IMNHO is whether we should do a power-of-2 or a > >> length. But, if we have 12 bits, then the argument for doing length is pretty > >> strong. We don't need anywhere near 12 bits if doing power-of-2. > > > > Just found the MAX_ORDER should be limited to 12 if use length instead of order, > > If the MAX_ORDER is configured to a value bigger than 12, it will make things more > > complex to handle this case. > > > > If use order, we need to break a large memory range whose length is not the power of 2 into several > > small ranges, it also make the code complex. > > I can't imagine it makes the code that much more complex. It adds a for > loop. Right? > > > It seems we leave too many bit for the pfn, and the bits leave for length is not enough, > > How about keep 45 bits for the pfn and 19 bits for length, 45 bits for pfn can cover 57 bits > > physical address, that should be enough in the near feature. > > > > What's your opinion? > > I still think 'order' makes a lot of sense. But, as you say, 57 bits is > enough for x86 for a while. Other architectures.... who knows? I think you can probably assume page size >= 4K. But I would not want to make any other assumptions. E.g. there are systems that absolutely require you to set high bits for DMA. I think we really want both length and order. I understand how you are trying to pack them as tightly as possible. However, I thought of a trick, we don't need to encode all possible orders. For example, with 2 bits of order, we can make them mean: 00 - 4K pages 01 - 2M pages 02 - 1G pages guest can program the sizes for each order through config space. We will have 10 bits left for legth. It might make sense to also allow guest to program the number of bits used for order, this will make it easy to extend without host changes. -- MST
[toc] | [prev] | [next] | [standalone]
| From | "Li, Liang Z" <liang.z.li@intel.com> |
|---|---|
| Date | 2016-12-16 02:20 +0100 |
| Subject | RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sOPlM-4TI-11@gated-at.bofh.it> |
| In reply to | #1542848 |
> On Thu, Dec 15, 2016 at 07:34:33AM -0800, Dave Hansen wrote: > > On 12/14/2016 12:59 AM, Li, Liang Z wrote: > > >> Subject: Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend > > >> virtio-balloon for fast (de)inflating & fast live migration > > >> > > >> On 12/08/2016 08:45 PM, Li, Liang Z wrote: > > >>> What's the conclusion of your discussion? It seems you want some > > >>> statistic before deciding whether to ripping the bitmap from the > > >>> ABI, am I right? > > >> > > >> I think Andrea and David feel pretty strongly that we should remove > > >> the bitmap, unless we have some data to support keeping it. I > > >> don't feel as strongly about it, but I think their critique of it > > >> is pretty valid. I think the consensus is that the bitmap needs to go. > > >> > > >> The only real question IMNHO is whether we should do a power-of-2 > > >> or a length. But, if we have 12 bits, then the argument for doing > > >> length is pretty strong. We don't need anywhere near 12 bits if doing > power-of-2. > > > > > > Just found the MAX_ORDER should be limited to 12 if use length > > > instead of order, If the MAX_ORDER is configured to a value bigger > > > than 12, it will make things more complex to handle this case. > > > > > > If use order, we need to break a large memory range whose length is > > > not the power of 2 into several small ranges, it also make the code > complex. > > > > I can't imagine it makes the code that much more complex. It adds a > > for loop. Right? > > > > > It seems we leave too many bit for the pfn, and the bits leave for > > > length is not enough, How about keep 45 bits for the pfn and 19 bits > > > for length, 45 bits for pfn can cover 57 bits physical address, that should > be enough in the near feature. > > > > > > What's your opinion? > > > > I still think 'order' makes a lot of sense. But, as you say, 57 bits > > is enough for x86 for a while. Other architectures.... who knows? > > I think you can probably assume page size >= 4K. But I would not want to > make any other assumptions. E.g. there are systems that absolutely require > you to set high bits for DMA. > > I think we really want both length and order. > > I understand how you are trying to pack them as tightly as possible. > > However, I thought of a trick, we don't need to encode all possible orders. > For example, with 2 bits of order, we can make them mean: > 00 - 4K pages > 01 - 2M pages > 02 - 1G pages > > guest can program the sizes for each order through config space. > > We will have 10 bits left for legth. > Please don't, we just get rid of the bitmap for simplification. :) > It might make sense to also allow guest to program the number of bits used > for order, this will make it easy to extend without host changes. > There still exist the case if the MAX_ORDER is configured to a large value, e.g. 36 for a system with huge amount of memory, then there is only 28 bits left for the pfn, which is not enough. Should we limit the MAX_ORDER? I don't think so. It seems use order is better. Thanks! Liang > -- > MST
[toc] | [prev] | [next] | [standalone]
| From | Andrea Arcangeli <aarcange@redhat.com> |
|---|---|
| Date | 2016-12-16 16:50 +0100 |
| Subject | Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sP2VI-5xB-23@gated-at.bofh.it> |
| In reply to | #1543159 |
On Fri, Dec 16, 2016 at 01:12:21AM +0000, Li, Liang Z wrote: > There still exist the case if the MAX_ORDER is configured to a large value, e.g. 36 for a system > with huge amount of memory, then there is only 28 bits left for the pfn, which is not enough. Not related to the balloon but how would it help to set MAX_ORDER to 36? What the MAX_ORDER affects is that you won't be able to ask the kernel page allocator for contiguous memory bigger than 1<<(MAX_ORDER-1), but that's a driver issue not relevant to the amount of RAM. Drivers won't suddenly start to ask the kernel allocator to allocate compound pages at orders >= 11 just because more RAM was added. The higher the MAX_ORDER the slower the kernel runs simply so the smaller the MAX_ORDER the better. > Should we limit the MAX_ORDER? I don't think so. We shouldn't strictly depend on MAX_ORDER value but it's mostly limited already even if configurable at build time. We definitely need it to reach at least the hugepage size, then it's mostly driver issue, but drivers requiring large contiguous allocations should rely on CMA only or vmalloc if they only require it virtually contiguous, and not rely on larger MAX_ORDER that would slowdown all kernel allocations/freeing.
[toc] | [prev] | [next] | [standalone]
| From | "Li, Liang Z" <liang.z.li@intel.com> |
|---|---|
| Date | 2016-12-17 13:00 +0100 |
| Subject | RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sPlOF-Ha-15@gated-at.bofh.it> |
| In reply to | #1543539 |
> On Fri, Dec 16, 2016 at 01:12:21AM +0000, Li, Liang Z wrote: > > There still exist the case if the MAX_ORDER is configured to a large > > value, e.g. 36 for a system with huge amount of memory, then there is only > 28 bits left for the pfn, which is not enough. > > Not related to the balloon but how would it help to set MAX_ORDER to 36? > My point here is MAX_ORDER may be configured to a big value. > What the MAX_ORDER affects is that you won't be able to ask the kernel > page allocator for contiguous memory bigger than 1<<(MAX_ORDER-1), but > that's a driver issue not relevant to the amount of RAM. Drivers won't > suddenly start to ask the kernel allocator to allocate compound pages at > orders >= 11 just because more RAM was added. > > The higher the MAX_ORDER the slower the kernel runs simply so the smaller > the MAX_ORDER the better. > > > Should we limit the MAX_ORDER? I don't think so. > > We shouldn't strictly depend on MAX_ORDER value but it's mostly limited > already even if configurable at build time. > I didn't know that and will take a look, thanks for your information. Liang > We definitely need it to reach at least the hugepage size, then it's mostly > driver issue, but drivers requiring large contiguous allocations should rely on > CMA only or vmalloc if they only require it virtually contiguous, and not rely > on larger MAX_ORDER that would slowdown all kernel allocations/freeing.
[toc] | [prev] | [next] | [standalone]
| From | "Li, Liang Z" <liang.z.li@intel.com> |
|---|---|
| Date | 2016-12-16 01:50 +0100 |
| Subject | RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sOOSJ-4rO-1@gated-at.bofh.it> |
| In reply to | #1542835 |
> Subject: Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for > fast (de)inflating & fast live migration > > On 12/14/2016 12:59 AM, Li, Liang Z wrote: > >> Subject: Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon > >> for fast (de)inflating & fast live migration > >> > >> On 12/08/2016 08:45 PM, Li, Liang Z wrote: > >>> What's the conclusion of your discussion? It seems you want some > >>> statistic before deciding whether to ripping the bitmap from the > >>> ABI, am I right? > >> > >> I think Andrea and David feel pretty strongly that we should remove > >> the bitmap, unless we have some data to support keeping it. I don't > >> feel as strongly about it, but I think their critique of it is pretty > >> valid. I think the consensus is that the bitmap needs to go. > >> > >> The only real question IMNHO is whether we should do a power-of-2 or > >> a length. But, if we have 12 bits, then the argument for doing > >> length is pretty strong. We don't need anywhere near 12 bits if doing > power-of-2. > > > > Just found the MAX_ORDER should be limited to 12 if use length instead > > of order, If the MAX_ORDER is configured to a value bigger than 12, it > > will make things more complex to handle this case. > > > > If use order, we need to break a large memory range whose length is > > not the power of 2 into several small ranges, it also make the code complex. > > I can't imagine it makes the code that much more complex. It adds a for loop. > Right? > Yes, just a little. :) > > It seems we leave too many bit for the pfn, and the bits leave for > > length is not enough, How about keep 45 bits for the pfn and 19 bits > > for length, 45 bits for pfn can cover 57 bits physical address, that should be > enough in the near feature. > > > > What's your opinion? > > I still think 'order' makes a lot of sense. But, as you say, 57 bits is enough for > x86 for a while. Other architectures.... who knows? Yes.
[toc] | [prev] | [next] | [standalone]
| From | Dave Hansen <dave.hansen@intel.com> |
|---|---|
| Date | 2016-12-16 02:20 +0100 |
| Subject | Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sOPlM-4TI-9@gated-at.bofh.it> |
| In reply to | #1543149 |
On 12/15/2016 04:48 PM, Li, Liang Z wrote: >>> It seems we leave too many bit for the pfn, and the bits leave for >>> length is not enough, How about keep 45 bits for the pfn and 19 bits >>> for length, 45 bits for pfn can cover 57 bits physical address, that should be >> enough in the near feature. >>> What's your opinion? >> I still think 'order' makes a lot of sense. But, as you say, 57 bits is enough for >> x86 for a while. Other architectures.... who knows? Thinking about this some more... There are really only two cases that matter: 4k pages and "much bigger" ones. Squeezing each 4k page into 8 bytes of metadata helps guarantee that this scheme won't regress over the old scheme in any cases. For bigger ranges, 8 vs 16 bytes means *nothing*. And 16 bytes will be as good or better than the old scheme for everything which is >4k. How about this: * 52 bits of 'pfn', 5 bits of 'order', 7 bits of 'length' * One special 'length' value to mean "actual length in next 8 bytes" That should be pretty simple to produce and decode. We have two record sizes, but I think it is manageable.
[toc] | [prev] | [next] | [standalone]
| From | "Li, Liang Z" <liang.z.li@intel.com> |
|---|---|
| Date | 2016-12-16 02:40 +0100 |
| Subject | RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sOPF7-50D-3@gated-at.bofh.it> |
| In reply to | #1543160 |
> On 12/15/2016 04:48 PM, Li, Liang Z wrote: > >>> It seems we leave too many bit for the pfn, and the bits leave for > >>> length is not enough, How about keep 45 bits for the pfn and 19 bits > >>> for length, 45 bits for pfn can cover 57 bits physical address, that > >>> should be > >> enough in the near feature. > >>> What's your opinion? > >> I still think 'order' makes a lot of sense. But, as you say, 57 bits > >> is enough for > >> x86 for a while. Other architectures.... who knows? > > Thinking about this some more... There are really only two cases that > matter: 4k pages and "much bigger" ones. > > Squeezing each 4k page into 8 bytes of metadata helps guarantee that this > scheme won't regress over the old scheme in any cases. For bigger ranges, 8 > vs 16 bytes means *nothing*. And 16 bytes will be as good or better than > the old scheme for everything which is >4k. > > How about this: > * 52 bits of 'pfn', 5 bits of 'order', 7 bits of 'length' > * One special 'length' value to mean "actual length in next 8 bytes" > > That should be pretty simple to produce and decode. We have two record > sizes, but I think it is manageable. It works, Now that we intend to use another 8 bytes for length Why not: Use 52 bits for 'pfn', 12 bits for 'length', when the 12 bits is not long enough for the 'length' Set the 'length' to a special value to indicate the "actual length in next 8 bytes". That will be much more simple. Right? Liang
[toc] | [prev] | [next] | [standalone]
| From | "Li, Liang Z" <liang.z.li@intel.com> |
|---|---|
| Date | 2016-12-16 02:50 +0100 |
| Subject | RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sOPON-53T-1@gated-at.bofh.it> |
| In reply to | #1543163 |
> On 12/15/2016 05:38 PM, Li, Liang Z wrote: > > > > Use 52 bits for 'pfn', 12 bits for 'length', when the 12 bits is not long enough > for the 'length' > > Set the 'length' to a special value to indicate the "actual length in next 8 > bytes". > > > > That will be much more simple. Right? > > Sounds fine to me. Thanks for your inspiration! Liang
[toc] | [prev] | [next] | [standalone]
| From | Dave Hansen <dave.hansen@intel.com> |
|---|---|
| Date | 2016-12-16 02:50 +0100 |
| Subject | Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sOPON-53T-3@gated-at.bofh.it> |
| In reply to | #1543163 |
On 12/15/2016 05:38 PM, Li, Liang Z wrote: > > Use 52 bits for 'pfn', 12 bits for 'length', when the 12 bits is not long enough for the 'length' > Set the 'length' to a special value to indicate the "actual length in next 8 bytes". > > That will be much more simple. Right? Sounds fine to me.
[toc] | [prev] | [next] | [standalone]
| From | Andrea Arcangeli <aarcange@redhat.com> |
|---|---|
| Date | 2016-12-16 17:10 +0100 |
| Subject | Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sP3f3-5TP-9@gated-at.bofh.it> |
| In reply to | #1543168 |
On Thu, Dec 15, 2016 at 05:40:45PM -0800, Dave Hansen wrote: > On 12/15/2016 05:38 PM, Li, Liang Z wrote: > > > > Use 52 bits for 'pfn', 12 bits for 'length', when the 12 bits is not long enough for the 'length' > > Set the 'length' to a special value to indicate the "actual length in next 8 bytes". > > > > That will be much more simple. Right? > > Sounds fine to me. > Sounds fine to me too indeed. I'm only wondering what is the major point for compressing gpfn+len in 8 bytes in the common case, you already use sg_init_table to send down two pages, we could send three as well and avoid all math and bit shifts and ors, or not? I agree with the above because from a performance prospective I tend to think the above proposal will run at least theoretically faster because the other way is to waste double amount of CPU cache, and bit mangling in the encoding and the later decoding on qemu side should be faster than accessing an array of double size, but then I'm not sure if it's measurable optimization. So I'd be curious to know the exact motivation and if it is to reduce the CPU cache usage or if there's some other fundamental reason to compress it. The header already tells qemu how big is the array payload, couldn't we just add more pages if one isn't enough? Thanks, Andrea
[toc] | [prev] | [next] | [standalone]
| From | "Li, Liang Z" <liang.z.li@intel.com> |
|---|---|
| Date | 2016-12-17 13:40 +0100 |
| Subject | RE: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sPmrn-1aF-7@gated-at.bofh.it> |
| In reply to | #1543554 |
> Subject: Re: [Qemu-devel] [PATCH kernel v5 0/5] Extend virtio-balloon for > fast (de)inflating & fast live migration > > On Thu, Dec 15, 2016 at 05:40:45PM -0800, Dave Hansen wrote: > > On 12/15/2016 05:38 PM, Li, Liang Z wrote: > > > > > > Use 52 bits for 'pfn', 12 bits for 'length', when the 12 bits is not long > enough for the 'length' > > > Set the 'length' to a special value to indicate the "actual length in next 8 > bytes". > > > > > > That will be much more simple. Right? > > > > Sounds fine to me. > > > > Sounds fine to me too indeed. > > I'm only wondering what is the major point for compressing gpfn+len in > 8 bytes in the common case, you already use sg_init_table to send down two > pages, we could send three as well and avoid all math and bit shifts and ors, > or not? > Yes, we can use more pages for that. > I agree with the above because from a performance prospective I tend to > think the above proposal will run at least theoretically faster because the > other way is to waste double amount of CPU cache, and bit mangling in the > encoding and the later decoding on qemu side should be faster than > accessing an array of double size, but then I'm not sure if it's measurable > optimization. So I'd be curious to know the exact motivation and if it is to > reduce the CPU cache usage or if there's some other fundamental reason to > compress it. > The header already tells qemu how big is the array payload, couldn't we just > add more pages if one isn't enough? > The original intention to compress the PFN and length it's to reduce the memory required. Even the code was changed a lot from the previous versions, I think this is still true. Now we allocate a specified buffer size to save the 'PFN|length', when the buffer is not big enough to save all the page info for a specified order. A double size buffer will be allocated. This is what we want to avoid because the allocation may fail and allocation takes some time, for fast live migration, time is a critical factor we have to consider, more time takes means more unnecessary pages are sent, because live migration starts before the request for unused pages get response. Thanks Liang > Thanks, > Andrea
[toc] | [prev] | [next] | [standalone]
| From | David Hildenbrand <david@redhat.com> |
|---|---|
| Date | 2016-12-07 16:50 +0100 |
| Subject | Re: [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration |
| Message-ID | <sLMDM-1V4-37@gated-at.bofh.it> |
| In reply to | #1537755 |
Am 07.12.2016 um 14:35 schrieb Li, Liang Z:
>> Am 30.11.2016 um 09:43 schrieb Liang Li:
>>> This patch set contains two parts of changes to the virtio-balloon.
>>>
>>> One is the change for speeding up the inflating & deflating process,
>>> the main idea of this optimization is to use bitmap to send the page
>>> information to host instead of the PFNs, to reduce the overhead of
>>> virtio data transmission, address translation and madvise(). This can
>>> help to improve the performance by about 85%.
>>
>> Do you have some statistics/some rough feeling how many consecutive bits are
>> usually set in the bitmaps? Is it really just purely random or is there some
>> granularity that is usually consecutive?
>>
>
> I did something similar. Filled the balloon with 15GB for a 16GB idle guest, by
> using bitmap, the madvise count was reduced to 605. when using the PFNs, the madvise count
> was 3932160. It means there are quite a lot consecutive bits in the bitmap.
> I didn't test for a guest with heavy memory workload.
Would it then even make sense to go one step further and report {pfn,
length} combinations?
So simply send over an array of {pfn, length}?
This idea came up when talking to Andrea Arcangeli (put him on cc).
And it makes sense if you think about:
a) hugetlb backing: The host may only be able to free huge pages (we
might want to communicate that to the guest later, that's another
story). Still we would have to send bitmaps full of 4k frames (512 bits
for 2mb frames). Of course, we could add a way to communicate that we
are using a different bitmap-granularity.
b) if we really inflate huge memory regions (and it sounds like that
according to your measurements), we can minimize the communication to
the hypervisor and therefore the madvice calls.
c) we don't want to optimize for inflating guests with almost full
memory (and therefore little consecutive memory areas) - my opinion :)
Thanks for the explanation!
--
David
[toc] | [prev] | [standalone]
Page 2 of 2 — ← Prev page 1 [2]
Back to top | Article view | linux.kernel
csiph-web