Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1382587 > unrolled thread

[PATCH kernel 0/2] speed up live migration by skipping free pages

Back to article view | Back to linux.kernel

Contents

  [PATCH kernel 0/2] speed up live migration by skipping free pages Liang Li <liang.z.li@intel.com> - 2016-04-19 16:50 +0200
    [PATCH kernel 2/2] virtio-balloon: extend balloon driver to support the new feature Liang Li <liang.z.li@intel.com> - 2016-04-19 16:50 +0200
      Re: [PATCH kernel 2/2] virtio-balloon: extend balloon driver to  support the new feature "Michael S. Tsirkin" <mst@redhat.com> - 2016-04-19 18:20 +0200
    Re: [PATCH kernel 0/2] speed up live migration by skipping free pages Amit Shah <amit.shah@redhat.com> - 2016-04-25 08:10 +0200
      Re: [PATCH kernel 0/2] speed up live migration by skipping free pages "Michael S. Tsirkin" <mst@redhat.com> - 2016-04-25 13:10 +0200
        Re: [PATCH kernel 0/2] speed up live migration by skipping free pages Amit Shah <amit.shah@redhat.com> - 2016-04-25 14:10 +0200
          Re: [PATCH kernel 0/2] speed up live migration by skipping free pages "Michael S. Tsirkin" <mst@redhat.com> - 2016-04-25 15:00 +0200
            Re: [PATCH kernel 0/2] speed up live migration by skipping free pages "Dr. David Alan Gilbert" <dgilbert@redhat.com> - 2016-04-25 15:20 +0200

#1382587 — [PATCH kernel 0/2] speed up live migration by skipping free pages

Current QEMU live migration implementation mark all guest's RAM pages
as dirtied in the ram bulk stage, all these pages will be processed
and it consumes quite a lot of CPU cycles and network bandwidth.

From guest's point of view, it doesn't care about the content in free
page. We can make use of this fact and skip processing the free
pages, this can save a lot CPU cycles and reduce the network traffic
significantly while speed up the live migration process obviously.

This patch set is the kernel side implementation.

The virtio-balloon driver is extended to send the free page bitmap
from guest to QEMU.

After getting the free page bitmap, QEMU can use it to filter out
guest's free pages. This make the live migration process much more
efficient.

In order to skip more free pages, we add an interface to let the user
decide whether dropping the cache in guest during live migration.

Liang Li (2):
  mm: add the related functions to build the free page bitmap
  virtio-balloon: extend balloon driver to support the new feature

 drivers/virtio/virtio_balloon.c     | 100 ++++++++++++++++++++++++++++++++++--
 fs/drop_caches.c                    |  22 +++++---
 include/linux/fs.h                  |   1 +
 include/uapi/linux/virtio_balloon.h |   1 +
 mm/page_alloc.c                     |  46 +++++++++++++++++
 5 files changed, 157 insertions(+), 13 deletions(-)

-- 
1.8.3.1

[toc] | [next] | [standalone]

#1382588 — [PATCH kernel 2/2] virtio-balloon: extend balloon driver to support the new feature

Extend the virtio balloon to support the new feature
VIRTIO_BALLOON_F_GET_FREE_PAGES, so that we can use it to send
the free page bitmap from guest to QEMU, the free page bitmap will
be used for live migration optimization.

Signed-off-by: Liang Li <liang.z.li@intel.com>
---
 drivers/virtio/virtio_balloon.c     | 100 ++++++++++++++++++++++++++++++++++--
 include/uapi/linux/virtio_balloon.h |   1 +
 2 files changed, 96 insertions(+), 5 deletions(-)

diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index 7b6d74f..cf17694 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -45,9 +45,17 @@ static int oom_pages = OOM_VBALLOON_DEFAULT_PAGES;
 module_param(oom_pages, int, S_IRUSR | S_IWUSR);
 MODULE_PARM_DESC(oom_pages, "pages to free on OOM");
 
+extern void get_free_pages(unsigned long *free_page_bitmap,
+				unsigned long len, int drop);
+extern unsigned long get_max_pfn(void);
+
+struct cache_drop_ctrl {
+	u64 ctrl;
+};
+
 struct virtio_balloon {
 	struct virtio_device *vdev;
-	struct virtqueue *inflate_vq, *deflate_vq, *stats_vq;
+	struct virtqueue *inflate_vq, *deflate_vq, *stats_vq, *free_pages_vq;
 
 	/* The balloon servicing is delegated to a freezable workqueue. */
 	struct work_struct update_balloon_stats_work;
@@ -77,6 +85,10 @@ struct virtio_balloon {
 	unsigned int num_pfns;
 	u32 pfns[VIRTIO_BALLOON_ARRAY_PFNS_MAX];
 
+	unsigned long *free_pages;
+	unsigned long bmap_len;
+	struct cache_drop_ctrl cache_drop;
+
 	/* Memory statistics */
 	struct virtio_balloon_stat stats[VIRTIO_BALLOON_S_NR];
 
@@ -256,6 +268,64 @@ static void update_balloon_stats(struct virtio_balloon *vb)
 				pages_to_bytes(available));
 }
 
+static void update_free_pages_stats(struct virtio_balloon *vb)
+{
+	unsigned long bitmap_bytes, max_pfn;
+
+	max_pfn = get_max_pfn();
+	bitmap_bytes = ALIGN(max_pfn, BITS_PER_LONG) / 8;
+
+	if (!vb->free_pages)
+		vb->free_pages = kzalloc(bitmap_bytes, GFP_KERNEL);
+	else {
+		if (bitmap_bytes < vb->bmap_len)
+			memset(vb->free_pages, 0, bitmap_bytes);
+		else {
+			kfree(vb->free_pages);
+			vb->free_pages = kzalloc(bitmap_bytes, GFP_KERNEL);
+		}
+	}
+	if (!vb->free_pages) {
+		vb->bmap_len = 0;
+		return;
+	}
+
+	vb->bmap_len = bitmap_bytes;
+	get_free_pages(vb->free_pages, max_pfn, vb->cache_drop.ctrl);
+}
+
+static void free_pages_handle_rq(struct virtio_balloon *vb)
+{
+	struct virtqueue *vq;
+	struct scatterlist sg[2];
+	unsigned int len;
+	struct cache_drop_ctl *ptr_cache_drop;
+	struct scatterlist sg_in;
+
+	vq = vb->free_pages_vq;
+	ptr_cache_drop = virtqueue_get_buf(vq, &len);
+
+	if (!ptr_cache_drop || len != sizeof(vb->cache_drop))
+		return;
+	update_free_pages_stats(vb);
+	sg_init_table(sg, 2);
+	sg_set_buf(&sg[0], &(vb->bmap_len), sizeof(vb->bmap_len));
+	sg_set_buf(&sg[1], vb->free_pages, vb->bmap_len);
+
+	sg_init_one(&sg_in, &vb->cache_drop, sizeof(vb->cache_drop));
+
+	virtqueue_add_outbuf(vq, &sg[0], 2, vb, GFP_KERNEL);
+	virtqueue_add_inbuf(vq, &sg_in, 1, &vb->cache_drop, GFP_KERNEL);
+	virtqueue_kick(vq);
+}
+
+static void free_pages_rq(struct virtqueue *vq)
+{
+	struct virtio_balloon *vb = vq->vdev->priv;
+
+	free_pages_handle_rq(vb);
+}
+
 /*
  * While most virtqueues communicate guest-initiated requests to the hypervisor,
  * the stats queue operates in reverse.  The driver initializes the virtqueue
@@ -392,16 +462,22 @@ static void update_balloon_size_func(struct work_struct *work)
 
 static int init_vqs(struct virtio_balloon *vb)
 {
-	struct virtqueue *vqs[3];
-	vq_callback_t *callbacks[] = { balloon_ack, balloon_ack, stats_request };
-	static const char * const names[] = { "inflate", "deflate", "stats" };
+	struct virtqueue *vqs[4];
+	vq_callback_t *callbacks[] = { balloon_ack, balloon_ack,
+					 stats_request, free_pages_rq };
+	const char *names[] = { "inflate", "deflate", "stats", "free_pages" };
 	int err, nvqs;
 
 	/*
 	 * We expect two virtqueues: inflate and deflate, and
 	 * optionally stat.
 	 */
-	nvqs = virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_STATS_VQ) ? 3 : 2;
+	if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_GET_FREE_PAGES))
+		nvqs = 4;
+	else
+		nvqs = virtio_has_feature(vb->vdev,
+					  VIRTIO_BALLOON_F_STATS_VQ) ? 3 : 2;
+
 	err = vb->vdev->config->find_vqs(vb->vdev, nvqs, vqs, callbacks, names);
 	if (err)
 		return err;
@@ -422,6 +498,16 @@ static int init_vqs(struct virtio_balloon *vb)
 			BUG();
 		virtqueue_kick(vb->stats_vq);
 	}
+	if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_GET_FREE_PAGES)) {
+		struct scatterlist sg_in;
+
+		vb->free_pages_vq = vqs[3];
+		sg_init_one(&sg_in, &vb->cache_drop, sizeof(vb->cache_drop));
+		if (virtqueue_add_inbuf(vb->free_pages_vq, &sg_in, 1,
+		    &vb->cache_drop, GFP_KERNEL) < 0)
+			BUG();
+		virtqueue_kick(vb->free_pages_vq);
+	}
 	return 0;
 }
 
@@ -505,6 +591,8 @@ static int virtballoon_probe(struct virtio_device *vdev)
 		goto out;
 	}
 
+	vb->bmap_len = 0;
+	vb->free_pages = NULL;
 	INIT_WORK(&vb->update_balloon_stats_work, update_balloon_stats_func);
 	INIT_WORK(&vb->update_balloon_size_work, update_balloon_size_func);
 	spin_lock_init(&vb->stop_update_lock);
@@ -567,6 +655,7 @@ static void virtballoon_remove(struct virtio_device *vdev)
 	cancel_work_sync(&vb->update_balloon_stats_work);
 
 	remove_common(vb);
+	kfree(vb->free_pages);
 	kfree(vb);
 }
 
@@ -605,6 +694,7 @@ static unsigned int features[] = {
 	VIRTIO_BALLOON_F_MUST_TELL_HOST,
 	VIRTIO_BALLOON_F_STATS_VQ,
 	VIRTIO_BALLOON_F_DEFLATE_ON_OOM,
+	VIRTIO_BALLOON_F_GET_FREE_PAGES,
 };
 
 static struct virtio_driver virtio_balloon_driver = {
diff --git a/include/uapi/linux/virtio_balloon.h b/include/uapi/linux/virtio_balloon.h
index 343d7dd..2b41e4f 100644
--- a/include/uapi/linux/virtio_balloon.h
+++ b/include/uapi/linux/virtio_balloon.h
@@ -34,6 +34,7 @@
 #define VIRTIO_BALLOON_F_MUST_TELL_HOST	0 /* Tell before reclaiming pages */
 #define VIRTIO_BALLOON_F_STATS_VQ	1 /* Memory Stats virtqueue */
 #define VIRTIO_BALLOON_F_DEFLATE_ON_OOM	2 /* Deflate balloon on OOM */
+#define VIRTIO_BALLOON_F_GET_FREE_PAGES	3 /* Get free page bitmap */
 
 /* Size of a PFN in the balloon interface. */
 #define VIRTIO_BALLOON_PFN_SHIFT 12
-- 
1.8.3.1

[toc] | [prev] | [next] | [standalone]

#1382673 — Re: [PATCH kernel 2/2] virtio-balloon: extend balloon driver to support the new feature

On Tue, Apr 19, 2016 at 10:34:34PM +0800, Liang Li wrote:
> Extend the virtio balloon to support the new feature
> VIRTIO_BALLOON_F_GET_FREE_PAGES, so that we can use it to send
> the free page bitmap from guest to QEMU, the free page bitmap will
> be used for live migration optimization.
> 
> Signed-off-by: Liang Li <liang.z.li@intel.com>

Two points:
- please post description of your interface proposals
  to virtio tc comment list
- please split this up
	- feature to use a bitmap for inflate/deflate
	- a 3rd vq which does inflate/deflate in one go

  there seems no reason to use bitmap for free pages
  but not for inflate/deflate

> ---
>  drivers/virtio/virtio_balloon.c     | 100 ++++++++++++++++++++++++++++++++++--
>  include/uapi/linux/virtio_balloon.h |   1 +
>  2 files changed, 96 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
> index 7b6d74f..cf17694 100644
> --- a/drivers/virtio/virtio_balloon.c
> +++ b/drivers/virtio/virtio_balloon.c
> @@ -45,9 +45,17 @@ static int oom_pages = OOM_VBALLOON_DEFAULT_PAGES;
>  module_param(oom_pages, int, S_IRUSR | S_IWUSR);
>  MODULE_PARM_DESC(oom_pages, "pages to free on OOM");
>  
> +extern void get_free_pages(unsigned long *free_page_bitmap,
> +				unsigned long len, int drop);
> +extern unsigned long get_max_pfn(void);
> +
> +struct cache_drop_ctrl {
> +	u64 ctrl;
> +};
> +
>  struct virtio_balloon {
>  	struct virtio_device *vdev;
> -	struct virtqueue *inflate_vq, *deflate_vq, *stats_vq;
> +	struct virtqueue *inflate_vq, *deflate_vq, *stats_vq, *free_pages_vq;
>  
>  	/* The balloon servicing is delegated to a freezable workqueue. */
>  	struct work_struct update_balloon_stats_work;
> @@ -77,6 +85,10 @@ struct virtio_balloon {
>  	unsigned int num_pfns;
>  	u32 pfns[VIRTIO_BALLOON_ARRAY_PFNS_MAX];
>  
> +	unsigned long *free_pages;
> +	unsigned long bmap_len;
> +	struct cache_drop_ctrl cache_drop;
> +
>  	/* Memory statistics */
>  	struct virtio_balloon_stat stats[VIRTIO_BALLOON_S_NR];
>  
> @@ -256,6 +268,64 @@ static void update_balloon_stats(struct virtio_balloon *vb)
>  				pages_to_bytes(available));
>  }
>  
> +static void update_free_pages_stats(struct virtio_balloon *vb)
> +{
> +	unsigned long bitmap_bytes, max_pfn;
> +
> +	max_pfn = get_max_pfn();
> +	bitmap_bytes = ALIGN(max_pfn, BITS_PER_LONG) / 8;
> +
> +	if (!vb->free_pages)
> +		vb->free_pages = kzalloc(bitmap_bytes, GFP_KERNEL);
> +	else {
> +		if (bitmap_bytes < vb->bmap_len)
> +			memset(vb->free_pages, 0, bitmap_bytes);
> +		else {
> +			kfree(vb->free_pages);
> +			vb->free_pages = kzalloc(bitmap_bytes, GFP_KERNEL);
> +		}
> +	}
> +	if (!vb->free_pages) {
> +		vb->bmap_len = 0;
> +		return;
> +	}
> +
> +	vb->bmap_len = bitmap_bytes;
> +	get_free_pages(vb->free_pages, max_pfn, vb->cache_drop.ctrl);
> +}
> +
> +static void free_pages_handle_rq(struct virtio_balloon *vb)
> +{
> +	struct virtqueue *vq;
> +	struct scatterlist sg[2];
> +	unsigned int len;
> +	struct cache_drop_ctl *ptr_cache_drop;
> +	struct scatterlist sg_in;
> +
> +	vq = vb->free_pages_vq;
> +	ptr_cache_drop = virtqueue_get_buf(vq, &len);
> +
> +	if (!ptr_cache_drop || len != sizeof(vb->cache_drop))
> +		return;
> +	update_free_pages_stats(vb);
> +	sg_init_table(sg, 2);
> +	sg_set_buf(&sg[0], &(vb->bmap_len), sizeof(vb->bmap_len));
> +	sg_set_buf(&sg[1], vb->free_pages, vb->bmap_len);
> +
> +	sg_init_one(&sg_in, &vb->cache_drop, sizeof(vb->cache_drop));
> +
> +	virtqueue_add_outbuf(vq, &sg[0], 2, vb, GFP_KERNEL);
> +	virtqueue_add_inbuf(vq, &sg_in, 1, &vb->cache_drop, GFP_KERNEL);
> +	virtqueue_kick(vq);
> +}
> +
> +static void free_pages_rq(struct virtqueue *vq)
> +{
> +	struct virtio_balloon *vb = vq->vdev->priv;
> +
> +	free_pages_handle_rq(vb);
> +}
> +
>  /*
>   * While most virtqueues communicate guest-initiated requests to the hypervisor,
>   * the stats queue operates in reverse.  The driver initializes the virtqueue
> @@ -392,16 +462,22 @@ static void update_balloon_size_func(struct work_struct *work)
>  
>  static int init_vqs(struct virtio_balloon *vb)
>  {
> -	struct virtqueue *vqs[3];
> -	vq_callback_t *callbacks[] = { balloon_ack, balloon_ack, stats_request };
> -	static const char * const names[] = { "inflate", "deflate", "stats" };
> +	struct virtqueue *vqs[4];
> +	vq_callback_t *callbacks[] = { balloon_ack, balloon_ack,
> +					 stats_request, free_pages_rq };
> +	const char *names[] = { "inflate", "deflate", "stats", "free_pages" };
>  	int err, nvqs;
>  
>  	/*
>  	 * We expect two virtqueues: inflate and deflate, and
>  	 * optionally stat.
>  	 */
> -	nvqs = virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_STATS_VQ) ? 3 : 2;
> +	if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_GET_FREE_PAGES))
> +		nvqs = 4;
> +	else
> +		nvqs = virtio_has_feature(vb->vdev,
> +					  VIRTIO_BALLOON_F_STATS_VQ) ? 3 : 2;
> +
>  	err = vb->vdev->config->find_vqs(vb->vdev, nvqs, vqs, callbacks, names);
>  	if (err)
>  		return err;
> @@ -422,6 +498,16 @@ static int init_vqs(struct virtio_balloon *vb)
>  			BUG();
>  		virtqueue_kick(vb->stats_vq);
>  	}
> +	if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_GET_FREE_PAGES)) {
> +		struct scatterlist sg_in;
> +
> +		vb->free_pages_vq = vqs[3];
> +		sg_init_one(&sg_in, &vb->cache_drop, sizeof(vb->cache_drop));
> +		if (virtqueue_add_inbuf(vb->free_pages_vq, &sg_in, 1,
> +		    &vb->cache_drop, GFP_KERNEL) < 0)
> +			BUG();
> +		virtqueue_kick(vb->free_pages_vq);
> +	}
>  	return 0;
>  }
>  
> @@ -505,6 +591,8 @@ static int virtballoon_probe(struct virtio_device *vdev)
>  		goto out;
>  	}
>  
> +	vb->bmap_len = 0;
> +	vb->free_pages = NULL;
>  	INIT_WORK(&vb->update_balloon_stats_work, update_balloon_stats_func);
>  	INIT_WORK(&vb->update_balloon_size_work, update_balloon_size_func);
>  	spin_lock_init(&vb->stop_update_lock);
> @@ -567,6 +655,7 @@ static void virtballoon_remove(struct virtio_device *vdev)
>  	cancel_work_sync(&vb->update_balloon_stats_work);
>  
>  	remove_common(vb);
> +	kfree(vb->free_pages);
>  	kfree(vb);
>  }
>  
> @@ -605,6 +694,7 @@ static unsigned int features[] = {
>  	VIRTIO_BALLOON_F_MUST_TELL_HOST,
>  	VIRTIO_BALLOON_F_STATS_VQ,
>  	VIRTIO_BALLOON_F_DEFLATE_ON_OOM,
> +	VIRTIO_BALLOON_F_GET_FREE_PAGES,
>  };
>  
>  static struct virtio_driver virtio_balloon_driver = {
> diff --git a/include/uapi/linux/virtio_balloon.h b/include/uapi/linux/virtio_balloon.h
> index 343d7dd..2b41e4f 100644
> --- a/include/uapi/linux/virtio_balloon.h
> +++ b/include/uapi/linux/virtio_balloon.h
> @@ -34,6 +34,7 @@
>  #define VIRTIO_BALLOON_F_MUST_TELL_HOST	0 /* Tell before reclaiming pages */
>  #define VIRTIO_BALLOON_F_STATS_VQ	1 /* Memory Stats virtqueue */
>  #define VIRTIO_BALLOON_F_DEFLATE_ON_OOM	2 /* Deflate balloon on OOM */
> +#define VIRTIO_BALLOON_F_GET_FREE_PAGES	3 /* Get free page bitmap */
>  
>  /* Size of a PFN in the balloon interface. */
>  #define VIRTIO_BALLOON_PFN_SHIFT 12
> -- 
> 1.8.3.1

[toc] | [prev] | [next] | [standalone]

#1386026

On (Tue) 19 Apr 2016 [22:34:32], Liang Li wrote:
> Current QEMU live migration implementation mark all guest's RAM pages
> as dirtied in the ram bulk stage, all these pages will be processed
> and it consumes quite a lot of CPU cycles and network bandwidth.
> 
> From guest's point of view, it doesn't care about the content in free
> page. We can make use of this fact and skip processing the free
> pages, this can save a lot CPU cycles and reduce the network traffic
> significantly while speed up the live migration process obviously.
> 
> This patch set is the kernel side implementation.
> 
> The virtio-balloon driver is extended to send the free page bitmap
> from guest to QEMU.
> 
> After getting the free page bitmap, QEMU can use it to filter out
> guest's free pages. This make the live migration process much more
> efficient.
> 
> In order to skip more free pages, we add an interface to let the user
> decide whether dropping the cache in guest during live migration.

So if virtio-balloon is the way to go (i.e. speed is acceptable), I
just have one point then.  My main concern with using (or not using)
virtio-balloon was that a guest admin is going to disable the
virtio-balloon driver entirely because the admin won't want the guest
to give away pages to the host, esp. when the guest is to be a
high-performant one.

In this case, if a new command can be added to the balloon spec where
a guest driver indicates it's not going to participate in ballooning
activity (ie a guest will ignore any ballooning requests from the
host), but use the driver just for stats-sharing purposes, that can be
a workable solution here as well.  In that case, we can keep the
MM-related stuff inside the balloon driver, and also get the benefit
of the guest having control over how it uses its memory,
disincentivising guest admins from disabling the balloon entirely (it
will also benefit the guest to keep this driver loaded in such a
state, if migration is faster!).

		Amit

[toc] | [prev] | [next] | [standalone]

#1386256

On Mon, Apr 25, 2016 at 11:36:41AM +0530, Amit Shah wrote:
> On (Tue) 19 Apr 2016 [22:34:32], Liang Li wrote:
> > Current QEMU live migration implementation mark all guest's RAM pages
> > as dirtied in the ram bulk stage, all these pages will be processed
> > and it consumes quite a lot of CPU cycles and network bandwidth.
> > 
> > From guest's point of view, it doesn't care about the content in free
> > page. We can make use of this fact and skip processing the free
> > pages, this can save a lot CPU cycles and reduce the network traffic
> > significantly while speed up the live migration process obviously.
> > 
> > This patch set is the kernel side implementation.
> > 
> > The virtio-balloon driver is extended to send the free page bitmap
> > from guest to QEMU.
> > 
> > After getting the free page bitmap, QEMU can use it to filter out
> > guest's free pages. This make the live migration process much more
> > efficient.
> > 
> > In order to skip more free pages, we add an interface to let the user
> > decide whether dropping the cache in guest during live migration.
> 
> So if virtio-balloon is the way to go (i.e. speed is acceptable), I
> just have one point then.  My main concern with using (or not using)
> virtio-balloon was that a guest admin is going to disable the
> virtio-balloon driver entirely because the admin won't want the guest
> to give away pages to the host, esp. when the guest is to be a
> high-performant one.

The result will be the reverse of high-performance.

If you don't want to inflate a balloon, don't.

If you do but guest doesn't respond to inflate requests,
it's quite reasonable for host to kill it -
there is no way to distinguish between that and
guest being malicious.

I don't know of management tools doing that but
it's rather reasonable. What does happen is
some random guest memory is pushed it out to swap,
which is likely much worse than dropping unused memory
by moving it into the balloon.

> In this case, if a new command can be added to the balloon spec where
> a guest driver indicates it's not going to participate in ballooning
> activity (ie a guest will ignore any ballooning requests from the
> host), but use the driver just for stats-sharing purposes, that can be
> a workable solution here as well.  In that case, we can keep the
> MM-related stuff inside the balloon driver, and also get the benefit
> of the guest having control over how it uses its memory,
> disincentivising guest admins from disabling the balloon entirely (it
> will also benefit the guest to keep this driver loaded in such a
> state, if migration is faster!).
> 
> 		Amit

If there actually are people doing that, we should
figure out the reasons.

-- 
MST

[toc] | [prev] | [next] | [standalone]

#1386320

On (Mon) 25 Apr 2016 [14:04:06], Michael S. Tsirkin wrote:
> On Mon, Apr 25, 2016 at 11:36:41AM +0530, Amit Shah wrote:
> > On (Tue) 19 Apr 2016 [22:34:32], Liang Li wrote:
> > > Current QEMU live migration implementation mark all guest's RAM pages
> > > as dirtied in the ram bulk stage, all these pages will be processed
> > > and it consumes quite a lot of CPU cycles and network bandwidth.
> > > 
> > > From guest's point of view, it doesn't care about the content in free
> > > page. We can make use of this fact and skip processing the free
> > > pages, this can save a lot CPU cycles and reduce the network traffic
> > > significantly while speed up the live migration process obviously.
> > > 
> > > This patch set is the kernel side implementation.
> > > 
> > > The virtio-balloon driver is extended to send the free page bitmap
> > > from guest to QEMU.
> > > 
> > > After getting the free page bitmap, QEMU can use it to filter out
> > > guest's free pages. This make the live migration process much more
> > > efficient.
> > > 
> > > In order to skip more free pages, we add an interface to let the user
> > > decide whether dropping the cache in guest during live migration.
> > 
> > So if virtio-balloon is the way to go (i.e. speed is acceptable), I
> > just have one point then.  My main concern with using (or not using)
> > virtio-balloon was that a guest admin is going to disable the
> > virtio-balloon driver entirely because the admin won't want the guest
> > to give away pages to the host, esp. when the guest is to be a
> > high-performant one.
> 
> The result will be the reverse of high-performance.
> 
> If you don't want to inflate a balloon, don't.
> 
> If you do but guest doesn't respond to inflate requests,
> it's quite reasonable for host to kill it -
> there is no way to distinguish between that and
> guest being malicious.

With the new command I'm suggesting, the guest will let the host know
that it has enabled this option, and it won't free up any RAM for the
host.

Also, just because a guest doesn't release some memory (which the
guest owns anyway) doesn't make it malicious, and killing such guests
is never going to end well for that hosting provider.

> I don't know of management tools doing that but
> it's rather reasonable. What does happen is
> some random guest memory is pushed it out to swap,
> which is likely much worse than dropping unused memory
> by moving it into the balloon.

Even if the host (admin) gave a guarantee that there won't be any
ballooning activity involved that will slow down the guest, a guest
admin can be paranoid enough to disable ballooning.  If, however, this
is made known to the host, it's likely a win-win situation because the
host knows the guest needs its RAM, and the guest can still use the
driver to send stats which the host can use during migration for
speedups.


		Amit

[toc] | [prev] | [next] | [standalone]

#1386384

On Mon, Apr 25, 2016 at 05:38:30PM +0530, Amit Shah wrote:
> On (Mon) 25 Apr 2016 [14:04:06], Michael S. Tsirkin wrote:
> > On Mon, Apr 25, 2016 at 11:36:41AM +0530, Amit Shah wrote:
> > > On (Tue) 19 Apr 2016 [22:34:32], Liang Li wrote:
> > > > Current QEMU live migration implementation mark all guest's RAM pages
> > > > as dirtied in the ram bulk stage, all these pages will be processed
> > > > and it consumes quite a lot of CPU cycles and network bandwidth.
> > > > 
> > > > From guest's point of view, it doesn't care about the content in free
> > > > page. We can make use of this fact and skip processing the free
> > > > pages, this can save a lot CPU cycles and reduce the network traffic
> > > > significantly while speed up the live migration process obviously.
> > > > 
> > > > This patch set is the kernel side implementation.
> > > > 
> > > > The virtio-balloon driver is extended to send the free page bitmap
> > > > from guest to QEMU.
> > > > 
> > > > After getting the free page bitmap, QEMU can use it to filter out
> > > > guest's free pages. This make the live migration process much more
> > > > efficient.
> > > > 
> > > > In order to skip more free pages, we add an interface to let the user
> > > > decide whether dropping the cache in guest during live migration.
> > > 
> > > So if virtio-balloon is the way to go (i.e. speed is acceptable), I
> > > just have one point then.  My main concern with using (or not using)
> > > virtio-balloon was that a guest admin is going to disable the
> > > virtio-balloon driver entirely because the admin won't want the guest
> > > to give away pages to the host, esp. when the guest is to be a
> > > high-performant one.
> > 
> > The result will be the reverse of high-performance.
> > 
> > If you don't want to inflate a balloon, don't.
> > 
> > If you do but guest doesn't respond to inflate requests,
> > it's quite reasonable for host to kill it -
> > there is no way to distinguish between that and
> > guest being malicious.
> 
> With the new command I'm suggesting, the guest will let the host know
> that it has enabled this option, and it won't free up any RAM for the
> host.
> 
> Also, just because a guest doesn't release some memory (which the
> guest owns anyway) doesn't make it malicious, and killing such guests
> is never going to end well for that hosting provider.
> 
> > I don't know of management tools doing that but
> > it's rather reasonable. What does happen is
> > some random guest memory is pushed it out to swap,
> > which is likely much worse than dropping unused memory
> > by moving it into the balloon.
> 
> Even if the host (admin) gave a guarantee that there won't be any
> ballooning activity involved that will slow down the guest, a guest
> admin can be paranoid enough to disable ballooning.  If, however, this
> is made known to the host, it's likely a win-win situation because the
> host knows the guest needs its RAM, and the guest can still use the
> driver to send stats which the host can use during migration for
> speedups.
> 
> 
> 		Amit

We'd need to understand the usecase better to design a good interface
for this. AFAIK the normal usecase for ballooning is for
memory overcommit: asking guest to free up memory might work
better than swap which makes host initiate a bunch of IO.
How is not inflating in this case a good idea?
I'm afraid I don't understand why was inflating balloon
requested if we do not want the guest to inflate the balloon.  What does
"paranoid" mean in this context?  This seems to imply some kind of
security concern. Is guest likely to need all of its memory or a
specific portion of it? Is it likely to be a static configuration or a
dynamic one? If dynamic, does guest also want to avoid deflating the
balloon or only inflating it?


-- 
MST

[toc] | [prev] | [next] | [standalone]

#1386406

* Michael S. Tsirkin (mst@redhat.com) wrote:
> On Mon, Apr 25, 2016 at 05:38:30PM +0530, Amit Shah wrote:
> > On (Mon) 25 Apr 2016 [14:04:06], Michael S. Tsirkin wrote:
> > > On Mon, Apr 25, 2016 at 11:36:41AM +0530, Amit Shah wrote:
> > > > On (Tue) 19 Apr 2016 [22:34:32], Liang Li wrote:
> > > > > Current QEMU live migration implementation mark all guest's RAM pages
> > > > > as dirtied in the ram bulk stage, all these pages will be processed
> > > > > and it consumes quite a lot of CPU cycles and network bandwidth.
> > > > > 
> > > > > From guest's point of view, it doesn't care about the content in free
> > > > > page. We can make use of this fact and skip processing the free
> > > > > pages, this can save a lot CPU cycles and reduce the network traffic
> > > > > significantly while speed up the live migration process obviously.
> > > > > 
> > > > > This patch set is the kernel side implementation.
> > > > > 
> > > > > The virtio-balloon driver is extended to send the free page bitmap
> > > > > from guest to QEMU.
> > > > > 
> > > > > After getting the free page bitmap, QEMU can use it to filter out
> > > > > guest's free pages. This make the live migration process much more
> > > > > efficient.
> > > > > 
> > > > > In order to skip more free pages, we add an interface to let the user
> > > > > decide whether dropping the cache in guest during live migration.
> > > > 
> > > > So if virtio-balloon is the way to go (i.e. speed is acceptable), I
> > > > just have one point then.  My main concern with using (or not using)
> > > > virtio-balloon was that a guest admin is going to disable the
> > > > virtio-balloon driver entirely because the admin won't want the guest
> > > > to give away pages to the host, esp. when the guest is to be a
> > > > high-performant one.
> > > 
> > > The result will be the reverse of high-performance.
> > > 
> > > If you don't want to inflate a balloon, don't.
> > > 
> > > If you do but guest doesn't respond to inflate requests,
> > > it's quite reasonable for host to kill it -
> > > there is no way to distinguish between that and
> > > guest being malicious.
> > 
> > With the new command I'm suggesting, the guest will let the host know
> > that it has enabled this option, and it won't free up any RAM for the
> > host.
> > 
> > Also, just because a guest doesn't release some memory (which the
> > guest owns anyway) doesn't make it malicious, and killing such guests
> > is never going to end well for that hosting provider.
> > 
> > > I don't know of management tools doing that but
> > > it's rather reasonable. What does happen is
> > > some random guest memory is pushed it out to swap,
> > > which is likely much worse than dropping unused memory
> > > by moving it into the balloon.
> > 
> > Even if the host (admin) gave a guarantee that there won't be any
> > ballooning activity involved that will slow down the guest, a guest
> > admin can be paranoid enough to disable ballooning.  If, however, this
> > is made known to the host, it's likely a win-win situation because the
> > host knows the guest needs its RAM, and the guest can still use the
> > driver to send stats which the host can use during migration for
> > speedups.
> > 
> > 
> > 		Amit
> 
> We'd need to understand the usecase better to design a good interface
> for this. AFAIK the normal usecase for ballooning is for
> memory overcommit: asking guest to free up memory might work
> better than swap which makes host initiate a bunch of IO.
> How is not inflating in this case a good idea?
> I'm afraid I don't understand why was inflating balloon
> requested if we do not want the guest to inflate the balloon.  What does
> "paranoid" mean in this context?  This seems to imply some kind of
> security concern. Is guest likely to need all of its memory or a
> specific portion of it? Is it likely to be a static configuration or a
> dynamic one? If dynamic, does guest also want to avoid deflating the
> balloon or only inflating it?

The semantics we want here are subtly different from ballooning;

  a) In ballooning the host is asking for the guest to give up some ram;
     that's not quite the case here - although if the guest dropped some
     caches that *might* be helpful.
  b) In ballooning we're interested in just amounts of free memory; here
     we care about _which_ pages are free.
  c) We don't want to make it hard for the guest to start using some of the
     memory again; if it was hard then we suddenly get back into a balancing
     act of needing to quantatively talk about how much free RAM we have,
     and worry about whether we shouldn't tell the host about a small pot
     we keep back etc.

Dave

> -- 
> MST
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web