Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1301259 > unrolled thread
| Started by | Alexander Duyck <alexander.duyck@gmail.com> |
|---|---|
| First post | 2016-01-05 04:20 +0100 |
| Last post | 2016-01-05 17:20 +0100 |
| Articles | 13 — 3 participants |
Back to article view | Back to linux.kernel
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking Alexander Duyck <alexander.duyck@gmail.com> - 2016-01-05 04:20 +0100
Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking "Michael S. Tsirkin" <mst@redhat.com> - 2016-01-05 10:50 +0100
Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking "Dr. David Alan Gilbert" <dgilbert@redhat.com> - 2016-01-05 11:10 +0100
Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking "Michael S. Tsirkin" <mst@redhat.com> - 2016-01-05 11:40 +0100
Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking "Dr. David Alan Gilbert" <dgilbert@redhat.com> - 2016-01-05 11:50 +0100
Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking "Michael S. Tsirkin" <mst@redhat.com> - 2016-01-05 12:10 +0100
Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking "Dr. David Alan Gilbert" <dgilbert@redhat.com> - 2016-01-05 12:10 +0100
Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking "Michael S. Tsirkin" <mst@redhat.com> - 2016-01-05 12:20 +0100
Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking "Michael S. Tsirkin" <mst@redhat.com> - 2016-01-05 12:10 +0100
Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking "Michael S. Tsirkin" <mst@redhat.com> - 2016-01-05 12:10 +0100
Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking "Dr. David Alan Gilbert" <dgilbert@redhat.com> - 2016-01-05 13:50 +0100
Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking "Michael S. Tsirkin" <mst@redhat.com> - 2016-01-05 14:20 +0100
Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking Alexander Duyck <alexander.duyck@gmail.com> - 2016-01-05 17:20 +0100
| From | Alexander Duyck <alexander.duyck@gmail.com> |
|---|---|
| Date | 2016-01-05 04:20 +0100 |
| Subject | Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking |
| Message-ID | <qNqka-4nh-5@gated-at.bofh.it> |
On Mon, Jan 4, 2016 at 12:41 PM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote: > On Sun, Dec 13, 2015 at 01:28:09PM -0800, Alexander Duyck wrote: >> This patch set is meant to be the guest side code for a proof of concept >> involving leaving pass-through devices in the guest during the warm-up >> phase of guest live migration. In order to accomplish this I have added a > > What does that mean? 'warm-up-phase'? It is the first phase in a pre-copy migration. https://en.wikipedia.org/wiki/Live_migration Basically in this phase all the memory is marked as dirty and then copied. Any memory that changes gets marked as dirty as well. Currently DMA circumvents this as the user space dirty page tracking isn't able to track DMA. >> new function called dma_mark_dirty that will mark the pages associated with >> the DMA transaction as dirty in the case of either an unmap or a >> sync_.*_for_cpu where the DMA direction is either DMA_FROM_DEVICE or >> DMA_BIDIRECTIONAL. The pass-through device must still be removed before >> the stop-and-copy phase, however allowing the device to be present should >> significantly improve the performance of the guest during the warm-up >> period. > > .. if the warm-up phase is short I presume? If the warm-up phase takes > a long time (busy guest that is of 1TB size) it wouldn't help much as the > tracking of these DMA's may be quite long? > >> >> This current implementation is very preliminary and there are number of >> items still missing. Specifically in order to make this a more complete >> solution we need to support: >> 1. Notifying hypervisor that drivers are dirtying DMA pages received > > .. And somehow giving the hypervisor the GPFN so it can retain the PFN in > the VT-d as long as possible. Yes, what has happened is that the host went through and marked all memory as read-only. So trying to do any operation that requires write access triggers a page fault which is then used by the host to track pages that were dirtied. >> 2. Bypassing page dirtying when it is not needed. > > How would this work with with device doing DMA operations _after_ the migration? > That is the driver submits and DMA READ.. migrates away, device is unplugged, > VT-d context is torn down - device does the DMA READ gets an VT-d error... > > and what then? How should the device on the other host replay the DMA READ? The device has to quiesce before the migration can occur. We cannot have any DMA mappings still open when we reach the stop-and-copy phase of the migration. The solution I have proposed here works for streaming mappings but doesn't solve the case for things like dma_alloc_coherent where a bidirectional mapping is maintained between the CPU and the device. >> >> The two mechanisms referenced above would likely require coordination with >> QEMU and as such are open to discussion. I haven't attempted to address >> them as I am not sure there is a consensus as of yet. My personal >> preference would be to add a vendor-specific configuration block to the >> emulated pci-bridge interfaces created by QEMU that would allow us to >> essentially extend shpc to support guest live migration with pass-through >> devices. > > shpc? That is kind of what I was thinking. We basically need some mechanism to allow for the host to ask the device to quiesce. It has been proposed to possibly even look at something like an ACPI interface since I know ACPI is used by QEMU to manage hot-plug in the standard case. - Alex -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [next] | [standalone]
| From | "Michael S. Tsirkin" <mst@redhat.com> |
|---|---|
| Date | 2016-01-05 10:50 +0100 |
| Subject | Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking |
| Message-ID | <qNwpz-eM-7@gated-at.bofh.it> |
| In reply to | #1301259 |
On Mon, Jan 04, 2016 at 07:11:25PM -0800, Alexander Duyck wrote: > >> The two mechanisms referenced above would likely require coordination with > >> QEMU and as such are open to discussion. I haven't attempted to address > >> them as I am not sure there is a consensus as of yet. My personal > >> preference would be to add a vendor-specific configuration block to the > >> emulated pci-bridge interfaces created by QEMU that would allow us to > >> essentially extend shpc to support guest live migration with pass-through > >> devices. > > > > shpc? > > That is kind of what I was thinking. We basically need some mechanism > to allow for the host to ask the device to quiesce. It has been > proposed to possibly even look at something like an ACPI interface > since I know ACPI is used by QEMU to manage hot-plug in the standard > case. > > - Alex Start by using hot-unplug for this! Really use your patch guest side, and write host side to allow starting migration with the device, but defer completing it. So 1.- host tells guest to start tracking memory writes 2.- guest acks 3.- migration starts 4.- most memory is migrated 5.- host tells guest to eject device 6.- guest acks 7.- stop vm and migrate rest of state It will already be a win since hot unplug after migration starts and most memory has been migrated is better than hot unplug before migration starts. Then measure downtime and profile. Then we can look at ways to quiesce device faster which really means step 5 is replaced with "host tells guest to quiesce device and dirty (or just unmap!) all memory mapped for write by device". -- MST -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | "Dr. David Alan Gilbert" <dgilbert@redhat.com> |
|---|---|
| Date | 2016-01-05 11:10 +0100 |
| Subject | Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking |
| Message-ID | <qNwIX-Cq-21@gated-at.bofh.it> |
| In reply to | #1301377 |
* Michael S. Tsirkin (mst@redhat.com) wrote: > On Mon, Jan 04, 2016 at 07:11:25PM -0800, Alexander Duyck wrote: > > >> The two mechanisms referenced above would likely require coordination with > > >> QEMU and as such are open to discussion. I haven't attempted to address > > >> them as I am not sure there is a consensus as of yet. My personal > > >> preference would be to add a vendor-specific configuration block to the > > >> emulated pci-bridge interfaces created by QEMU that would allow us to > > >> essentially extend shpc to support guest live migration with pass-through > > >> devices. > > > > > > shpc? > > > > That is kind of what I was thinking. We basically need some mechanism > > to allow for the host to ask the device to quiesce. It has been > > proposed to possibly even look at something like an ACPI interface > > since I know ACPI is used by QEMU to manage hot-plug in the standard > > case. > > > > - Alex > > > Start by using hot-unplug for this! > > Really use your patch guest side, and write host side > to allow starting migration with the device, but > defer completing it. > > So > > 1.- host tells guest to start tracking memory writes > 2.- guest acks > 3.- migration starts > 4.- most memory is migrated > 5.- host tells guest to eject device > 6.- guest acks > 7.- stop vm and migrate rest of state > > > It will already be a win since hot unplug after migration starts and > most memory has been migrated is better than hot unplug before migration > starts. > > Then measure downtime and profile. Then we can look at ways > to quiesce device faster which really means step 5 is replaced > with "host tells guest to quiesce device and dirty (or just unmap!) > all memory mapped for write by device". Doing a hot-unplug is going to upset the guests network stacks view of the world; that's something we don't want to change. Dave > > -- > MST -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | "Michael S. Tsirkin" <mst@redhat.com> |
|---|---|
| Date | 2016-01-05 11:40 +0100 |
| Subject | Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking |
| Message-ID | <qNxbX-O8-17@gated-at.bofh.it> |
| In reply to | #1301395 |
On Tue, Jan 05, 2016 at 10:01:04AM +0000, Dr. David Alan Gilbert wrote: > * Michael S. Tsirkin (mst@redhat.com) wrote: > > On Mon, Jan 04, 2016 at 07:11:25PM -0800, Alexander Duyck wrote: > > > >> The two mechanisms referenced above would likely require coordination with > > > >> QEMU and as such are open to discussion. I haven't attempted to address > > > >> them as I am not sure there is a consensus as of yet. My personal > > > >> preference would be to add a vendor-specific configuration block to the > > > >> emulated pci-bridge interfaces created by QEMU that would allow us to > > > >> essentially extend shpc to support guest live migration with pass-through > > > >> devices. > > > > > > > > shpc? > > > > > > That is kind of what I was thinking. We basically need some mechanism > > > to allow for the host to ask the device to quiesce. It has been > > > proposed to possibly even look at something like an ACPI interface > > > since I know ACPI is used by QEMU to manage hot-plug in the standard > > > case. > > > > > > - Alex > > > > > > Start by using hot-unplug for this! > > > > Really use your patch guest side, and write host side > > to allow starting migration with the device, but > > defer completing it. > > > > So > > > > 1.- host tells guest to start tracking memory writes > > 2.- guest acks > > 3.- migration starts > > 4.- most memory is migrated > > 5.- host tells guest to eject device > > 6.- guest acks > > 7.- stop vm and migrate rest of state > > > > > > It will already be a win since hot unplug after migration starts and > > most memory has been migrated is better than hot unplug before migration > > starts. > > > > Then measure downtime and profile. Then we can look at ways > > to quiesce device faster which really means step 5 is replaced > > with "host tells guest to quiesce device and dirty (or just unmap!) > > all memory mapped for write by device". > > > Doing a hot-unplug is going to upset the guests network stacks view > of the world; that's something we don't want to change. > > Dave It might but if you store the IP and restore it quickly after migration e.g. using guest agent, as opposed to DHCP, then it won't. It allows calming the device down in a generic way, specific drivers can then implement the fast quiesce. > > > > -- > > MST > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | "Dr. David Alan Gilbert" <dgilbert@redhat.com> |
|---|---|
| Date | 2016-01-05 11:50 +0100 |
| Subject | Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking |
| Message-ID | <qNxlD-Sz-1@gated-at.bofh.it> |
| In reply to | #1301411 |
* Michael S. Tsirkin (mst@redhat.com) wrote: > On Tue, Jan 05, 2016 at 10:01:04AM +0000, Dr. David Alan Gilbert wrote: > > * Michael S. Tsirkin (mst@redhat.com) wrote: > > > On Mon, Jan 04, 2016 at 07:11:25PM -0800, Alexander Duyck wrote: > > > > >> The two mechanisms referenced above would likely require coordination with > > > > >> QEMU and as such are open to discussion. I haven't attempted to address > > > > >> them as I am not sure there is a consensus as of yet. My personal > > > > >> preference would be to add a vendor-specific configuration block to the > > > > >> emulated pci-bridge interfaces created by QEMU that would allow us to > > > > >> essentially extend shpc to support guest live migration with pass-through > > > > >> devices. > > > > > > > > > > shpc? > > > > > > > > That is kind of what I was thinking. We basically need some mechanism > > > > to allow for the host to ask the device to quiesce. It has been > > > > proposed to possibly even look at something like an ACPI interface > > > > since I know ACPI is used by QEMU to manage hot-plug in the standard > > > > case. > > > > > > > > - Alex > > > > > > > > > Start by using hot-unplug for this! > > > > > > Really use your patch guest side, and write host side > > > to allow starting migration with the device, but > > > defer completing it. > > > > > > So > > > > > > 1.- host tells guest to start tracking memory writes > > > 2.- guest acks > > > 3.- migration starts > > > 4.- most memory is migrated > > > 5.- host tells guest to eject device > > > 6.- guest acks > > > 7.- stop vm and migrate rest of state > > > > > > > > > It will already be a win since hot unplug after migration starts and > > > most memory has been migrated is better than hot unplug before migration > > > starts. > > > > > > Then measure downtime and profile. Then we can look at ways > > > to quiesce device faster which really means step 5 is replaced > > > with "host tells guest to quiesce device and dirty (or just unmap!) > > > all memory mapped for write by device". > > > > > > Doing a hot-unplug is going to upset the guests network stacks view > > of the world; that's something we don't want to change. > > > > Dave > > It might but if you store the IP and restore it quickly > after migration e.g. using guest agent, as opposed to DHCP, > then it won't. I thought if you hot-unplug then it will lose any outstanding connections on that device. > It allows calming the device down in a generic way, > specific drivers can then implement the fast quiesce. Except that if it breaks the guest networking it's useless. Dave > > > > > > > -- > > > MST > > -- > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | "Michael S. Tsirkin" <mst@redhat.com> |
|---|---|
| Date | 2016-01-05 12:10 +0100 |
| Subject | Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking |
| Message-ID | <qNxEZ-1fC-1@gated-at.bofh.it> |
| In reply to | #1301414 |
On Tue, Jan 05, 2016 at 10:45:25AM +0000, Dr. David Alan Gilbert wrote: > * Michael S. Tsirkin (mst@redhat.com) wrote: > > On Tue, Jan 05, 2016 at 10:01:04AM +0000, Dr. David Alan Gilbert wrote: > > > * Michael S. Tsirkin (mst@redhat.com) wrote: > > > > On Mon, Jan 04, 2016 at 07:11:25PM -0800, Alexander Duyck wrote: > > > > > >> The two mechanisms referenced above would likely require coordination with > > > > > >> QEMU and as such are open to discussion. I haven't attempted to address > > > > > >> them as I am not sure there is a consensus as of yet. My personal > > > > > >> preference would be to add a vendor-specific configuration block to the > > > > > >> emulated pci-bridge interfaces created by QEMU that would allow us to > > > > > >> essentially extend shpc to support guest live migration with pass-through > > > > > >> devices. > > > > > > > > > > > > shpc? > > > > > > > > > > That is kind of what I was thinking. We basically need some mechanism > > > > > to allow for the host to ask the device to quiesce. It has been > > > > > proposed to possibly even look at something like an ACPI interface > > > > > since I know ACPI is used by QEMU to manage hot-plug in the standard > > > > > case. > > > > > > > > > > - Alex > > > > > > > > > > > > Start by using hot-unplug for this! > > > > > > > > Really use your patch guest side, and write host side > > > > to allow starting migration with the device, but > > > > defer completing it. > > > > > > > > So > > > > > > > > 1.- host tells guest to start tracking memory writes > > > > 2.- guest acks > > > > 3.- migration starts > > > > 4.- most memory is migrated > > > > 5.- host tells guest to eject device > > > > 6.- guest acks > > > > 7.- stop vm and migrate rest of state > > > > > > > > > > > > It will already be a win since hot unplug after migration starts and > > > > most memory has been migrated is better than hot unplug before migration > > > > starts. > > > > > > > > Then measure downtime and profile. Then we can look at ways > > > > to quiesce device faster which really means step 5 is replaced > > > > with "host tells guest to quiesce device and dirty (or just unmap!) > > > > all memory mapped for write by device". > > > > > > > > > Doing a hot-unplug is going to upset the guests network stacks view > > > of the world; that's something we don't want to change. > > > > > > Dave > > > > It might but if you store the IP and restore it quickly > > after migration e.g. using guest agent, as opposed to DHCP, > > then it won't. > > I thought if you hot-unplug then it will lose any outstanding connections > on that device. > > > It allows calming the device down in a generic way, > > specific drivers can then implement the fast quiesce. > > Except that if it breaks the guest networking it's useless. > > Dave Is hot unplug useless then? > > > > > > > > > > -- > > > > MST > > > -- > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | "Dr. David Alan Gilbert" <dgilbert@redhat.com> |
|---|---|
| Date | 2016-01-05 12:10 +0100 |
| Subject | Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking |
| Message-ID | <qNxF0-1fC-7@gated-at.bofh.it> |
| In reply to | #1301440 |
* Michael S. Tsirkin (mst@redhat.com) wrote: > On Tue, Jan 05, 2016 at 10:45:25AM +0000, Dr. David Alan Gilbert wrote: > > * Michael S. Tsirkin (mst@redhat.com) wrote: > > > On Tue, Jan 05, 2016 at 10:01:04AM +0000, Dr. David Alan Gilbert wrote: > > > > * Michael S. Tsirkin (mst@redhat.com) wrote: > > > > > On Mon, Jan 04, 2016 at 07:11:25PM -0800, Alexander Duyck wrote: > > > > > > >> The two mechanisms referenced above would likely require coordination with > > > > > > >> QEMU and as such are open to discussion. I haven't attempted to address > > > > > > >> them as I am not sure there is a consensus as of yet. My personal > > > > > > >> preference would be to add a vendor-specific configuration block to the > > > > > > >> emulated pci-bridge interfaces created by QEMU that would allow us to > > > > > > >> essentially extend shpc to support guest live migration with pass-through > > > > > > >> devices. > > > > > > > > > > > > > > shpc? > > > > > > > > > > > > That is kind of what I was thinking. We basically need some mechanism > > > > > > to allow for the host to ask the device to quiesce. It has been > > > > > > proposed to possibly even look at something like an ACPI interface > > > > > > since I know ACPI is used by QEMU to manage hot-plug in the standard > > > > > > case. > > > > > > > > > > > > - Alex > > > > > > > > > > > > > > > Start by using hot-unplug for this! > > > > > > > > > > Really use your patch guest side, and write host side > > > > > to allow starting migration with the device, but > > > > > defer completing it. > > > > > > > > > > So > > > > > > > > > > 1.- host tells guest to start tracking memory writes > > > > > 2.- guest acks > > > > > 3.- migration starts > > > > > 4.- most memory is migrated > > > > > 5.- host tells guest to eject device > > > > > 6.- guest acks > > > > > 7.- stop vm and migrate rest of state > > > > > > > > > > > > > > > It will already be a win since hot unplug after migration starts and > > > > > most memory has been migrated is better than hot unplug before migration > > > > > starts. > > > > > > > > > > Then measure downtime and profile. Then we can look at ways > > > > > to quiesce device faster which really means step 5 is replaced > > > > > with "host tells guest to quiesce device and dirty (or just unmap!) > > > > > all memory mapped for write by device". > > > > > > > > > > > > Doing a hot-unplug is going to upset the guests network stacks view > > > > of the world; that's something we don't want to change. > > > > > > > > Dave > > > > > > It might but if you store the IP and restore it quickly > > > after migration e.g. using guest agent, as opposed to DHCP, > > > then it won't. > > > > I thought if you hot-unplug then it will lose any outstanding connections > > on that device. > > > > > It allows calming the device down in a generic way, > > > specific drivers can then implement the fast quiesce. > > > > Except that if it breaks the guest networking it's useless. > > > > Dave > > Is hot unplug useless then? As a migration hack, yes, unless it's paired with a second network device as a redundent route. To do what's being suggested here it's got to be done at the device level and not visible to the networking stack. Dave > > > > > > > > > > > > > > -- > > > > > MST > > > > -- > > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > -- > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | "Michael S. Tsirkin" <mst@redhat.com> |
|---|---|
| Date | 2016-01-05 12:20 +0100 |
| Subject | Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking |
| Message-ID | <qNxOG-1jL-19@gated-at.bofh.it> |
| In reply to | #1301441 |
On Tue, Jan 05, 2016 at 11:03:38AM +0000, Dr. David Alan Gilbert wrote: > * Michael S. Tsirkin (mst@redhat.com) wrote: > > On Tue, Jan 05, 2016 at 10:45:25AM +0000, Dr. David Alan Gilbert wrote: > > > * Michael S. Tsirkin (mst@redhat.com) wrote: > > > > On Tue, Jan 05, 2016 at 10:01:04AM +0000, Dr. David Alan Gilbert wrote: > > > > > * Michael S. Tsirkin (mst@redhat.com) wrote: > > > > > > On Mon, Jan 04, 2016 at 07:11:25PM -0800, Alexander Duyck wrote: > > > > > > > >> The two mechanisms referenced above would likely require coordination with > > > > > > > >> QEMU and as such are open to discussion. I haven't attempted to address > > > > > > > >> them as I am not sure there is a consensus as of yet. My personal > > > > > > > >> preference would be to add a vendor-specific configuration block to the > > > > > > > >> emulated pci-bridge interfaces created by QEMU that would allow us to > > > > > > > >> essentially extend shpc to support guest live migration with pass-through > > > > > > > >> devices. > > > > > > > > > > > > > > > > shpc? > > > > > > > > > > > > > > That is kind of what I was thinking. We basically need some mechanism > > > > > > > to allow for the host to ask the device to quiesce. It has been > > > > > > > proposed to possibly even look at something like an ACPI interface > > > > > > > since I know ACPI is used by QEMU to manage hot-plug in the standard > > > > > > > case. > > > > > > > > > > > > > > - Alex > > > > > > > > > > > > > > > > > > Start by using hot-unplug for this! > > > > > > > > > > > > Really use your patch guest side, and write host side > > > > > > to allow starting migration with the device, but > > > > > > defer completing it. > > > > > > > > > > > > So > > > > > > > > > > > > 1.- host tells guest to start tracking memory writes > > > > > > 2.- guest acks > > > > > > 3.- migration starts > > > > > > 4.- most memory is migrated > > > > > > 5.- host tells guest to eject device > > > > > > 6.- guest acks > > > > > > 7.- stop vm and migrate rest of state > > > > > > > > > > > > > > > > > > It will already be a win since hot unplug after migration starts and > > > > > > most memory has been migrated is better than hot unplug before migration > > > > > > starts. > > > > > > > > > > > > Then measure downtime and profile. Then we can look at ways > > > > > > to quiesce device faster which really means step 5 is replaced > > > > > > with "host tells guest to quiesce device and dirty (or just unmap!) > > > > > > all memory mapped for write by device". > > > > > > > > > > > > > > > Doing a hot-unplug is going to upset the guests network stacks view > > > > > of the world; that's something we don't want to change. > > > > > > > > > > Dave > > > > > > > > It might but if you store the IP and restore it quickly > > > > after migration e.g. using guest agent, as opposed to DHCP, > > > > then it won't. > > > > > > I thought if you hot-unplug then it will lose any outstanding connections > > > on that device. > > > > > > > It allows calming the device down in a generic way, > > > > specific drivers can then implement the fast quiesce. > > > > > > Except that if it breaks the guest networking it's useless. > > > > > > Dave > > > > Is hot unplug useless then? > > As a migration hack, yes, Based on a premise that it breaks connections but it does not have to. > unless it's paired with a second network device > as a redundent route. You can do this too. But this is not a must at all. > To do what's being suggested here it's got to be done at the device level > and not visible to the networking stack. > > Dave Need for this was never demonstrated. > > > > > > > > > > > > > > > > > > -- > > > > > > MST > > > > > -- > > > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > > -- > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | "Michael S. Tsirkin" <mst@redhat.com> |
|---|---|
| Date | 2016-01-05 12:10 +0100 |
| Subject | Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking |
| Message-ID | <qNxF0-1fC-25@gated-at.bofh.it> |
| In reply to | #1301440 |
On Tue, Jan 05, 2016 at 12:59:54PM +0200, Michael S. Tsirkin wrote: > On Tue, Jan 05, 2016 at 10:45:25AM +0000, Dr. David Alan Gilbert wrote: > > * Michael S. Tsirkin (mst@redhat.com) wrote: > > > On Tue, Jan 05, 2016 at 10:01:04AM +0000, Dr. David Alan Gilbert wrote: > > > > * Michael S. Tsirkin (mst@redhat.com) wrote: > > > > > On Mon, Jan 04, 2016 at 07:11:25PM -0800, Alexander Duyck wrote: > > > > > > >> The two mechanisms referenced above would likely require coordination with > > > > > > >> QEMU and as such are open to discussion. I haven't attempted to address > > > > > > >> them as I am not sure there is a consensus as of yet. My personal > > > > > > >> preference would be to add a vendor-specific configuration block to the > > > > > > >> emulated pci-bridge interfaces created by QEMU that would allow us to > > > > > > >> essentially extend shpc to support guest live migration with pass-through > > > > > > >> devices. > > > > > > > > > > > > > > shpc? > > > > > > > > > > > > That is kind of what I was thinking. We basically need some mechanism > > > > > > to allow for the host to ask the device to quiesce. It has been > > > > > > proposed to possibly even look at something like an ACPI interface > > > > > > since I know ACPI is used by QEMU to manage hot-plug in the standard > > > > > > case. > > > > > > > > > > > > - Alex > > > > > > > > > > > > > > > Start by using hot-unplug for this! > > > > > > > > > > Really use your patch guest side, and write host side > > > > > to allow starting migration with the device, but > > > > > defer completing it. > > > > > > > > > > So > > > > > > > > > > 1.- host tells guest to start tracking memory writes > > > > > 2.- guest acks > > > > > 3.- migration starts > > > > > 4.- most memory is migrated > > > > > 5.- host tells guest to eject device > > > > > 6.- guest acks > > > > > 7.- stop vm and migrate rest of state > > > > > > > > > > > > > > > It will already be a win since hot unplug after migration starts and > > > > > most memory has been migrated is better than hot unplug before migration > > > > > starts. > > > > > > > > > > Then measure downtime and profile. Then we can look at ways > > > > > to quiesce device faster which really means step 5 is replaced > > > > > with "host tells guest to quiesce device and dirty (or just unmap!) > > > > > all memory mapped for write by device". > > > > > > > > > > > > Doing a hot-unplug is going to upset the guests network stacks view > > > > of the world; that's something we don't want to change. > > > > > > > > Dave > > > > > > It might but if you store the IP and restore it quickly > > > after migration e.g. using guest agent, as opposed to DHCP, > > > then it won't. > > > > I thought if you hot-unplug then it will lose any outstanding connections > > on that device. > > > > > It allows calming the device down in a generic way, > > > specific drivers can then implement the fast quiesce. > > > > Except that if it breaks the guest networking it's useless. > > > > Dave > > Is hot unplug useless then? Actually I misunderstood the question, unplug does not have to break guest networking. > > > > > > > > > > > > > -- > > > > > MST > > > > -- > > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > -- > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | "Michael S. Tsirkin" <mst@redhat.com> |
|---|---|
| Date | 2016-01-05 12:10 +0100 |
| Subject | Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking |
| Message-ID | <qNxF1-1fC-33@gated-at.bofh.it> |
| In reply to | #1301414 |
On Tue, Jan 05, 2016 at 10:45:25AM +0000, Dr. David Alan Gilbert wrote: > * Michael S. Tsirkin (mst@redhat.com) wrote: > > On Tue, Jan 05, 2016 at 10:01:04AM +0000, Dr. David Alan Gilbert wrote: > > > * Michael S. Tsirkin (mst@redhat.com) wrote: > > > > On Mon, Jan 04, 2016 at 07:11:25PM -0800, Alexander Duyck wrote: > > > > > >> The two mechanisms referenced above would likely require coordination with > > > > > >> QEMU and as such are open to discussion. I haven't attempted to address > > > > > >> them as I am not sure there is a consensus as of yet. My personal > > > > > >> preference would be to add a vendor-specific configuration block to the > > > > > >> emulated pci-bridge interfaces created by QEMU that would allow us to > > > > > >> essentially extend shpc to support guest live migration with pass-through > > > > > >> devices. > > > > > > > > > > > > shpc? > > > > > > > > > > That is kind of what I was thinking. We basically need some mechanism > > > > > to allow for the host to ask the device to quiesce. It has been > > > > > proposed to possibly even look at something like an ACPI interface > > > > > since I know ACPI is used by QEMU to manage hot-plug in the standard > > > > > case. > > > > > > > > > > - Alex > > > > > > > > > > > > Start by using hot-unplug for this! > > > > > > > > Really use your patch guest side, and write host side > > > > to allow starting migration with the device, but > > > > defer completing it. > > > > > > > > So > > > > > > > > 1.- host tells guest to start tracking memory writes > > > > 2.- guest acks > > > > 3.- migration starts > > > > 4.- most memory is migrated > > > > 5.- host tells guest to eject device > > > > 6.- guest acks > > > > 7.- stop vm and migrate rest of state > > > > > > > > > > > > It will already be a win since hot unplug after migration starts and > > > > most memory has been migrated is better than hot unplug before migration > > > > starts. > > > > > > > > Then measure downtime and profile. Then we can look at ways > > > > to quiesce device faster which really means step 5 is replaced > > > > with "host tells guest to quiesce device and dirty (or just unmap!) > > > > all memory mapped for write by device". > > > > > > > > > Doing a hot-unplug is going to upset the guests network stacks view > > > of the world; that's something we don't want to change. > > > > > > Dave > > > > It might but if you store the IP and restore it quickly > > after migration e.g. using guest agent, as opposed to DHCP, > > then it won't. > > I thought if you hot-unplug then it will lose any outstanding connections > on that device. Which connections and which device? TCP connections and an ethernet device? These are on different layers so of course you don't lose them. Just do not change the IP address. Some guests send a signal to applications to close connections when all links go down. One can work around this in a variety of ways. > > It allows calming the device down in a generic way, > > specific drivers can then implement the fast quiesce. > > Except that if it breaks the guest networking it's useless. > > Dave > > > > > > > > > > > -- > > > > MST > > > -- > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | "Dr. David Alan Gilbert" <dgilbert@redhat.com> |
|---|---|
| Date | 2016-01-05 13:50 +0100 |
| Subject | Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking |
| Message-ID | <qNzdN-2Op-27@gated-at.bofh.it> |
| In reply to | #1301446 |
* Michael S. Tsirkin (mst@redhat.com) wrote: > On Tue, Jan 05, 2016 at 10:45:25AM +0000, Dr. David Alan Gilbert wrote: > > * Michael S. Tsirkin (mst@redhat.com) wrote: > > > On Tue, Jan 05, 2016 at 10:01:04AM +0000, Dr. David Alan Gilbert wrote: > > > > * Michael S. Tsirkin (mst@redhat.com) wrote: > > > > > On Mon, Jan 04, 2016 at 07:11:25PM -0800, Alexander Duyck wrote: > > > > > > >> The two mechanisms referenced above would likely require coordination with > > > > > > >> QEMU and as such are open to discussion. I haven't attempted to address > > > > > > >> them as I am not sure there is a consensus as of yet. My personal > > > > > > >> preference would be to add a vendor-specific configuration block to the > > > > > > >> emulated pci-bridge interfaces created by QEMU that would allow us to > > > > > > >> essentially extend shpc to support guest live migration with pass-through > > > > > > >> devices. > > > > > > > > > > > > > > shpc? > > > > > > > > > > > > That is kind of what I was thinking. We basically need some mechanism > > > > > > to allow for the host to ask the device to quiesce. It has been > > > > > > proposed to possibly even look at something like an ACPI interface > > > > > > since I know ACPI is used by QEMU to manage hot-plug in the standard > > > > > > case. > > > > > > > > > > > > - Alex > > > > > > > > > > > > > > > Start by using hot-unplug for this! > > > > > > > > > > Really use your patch guest side, and write host side > > > > > to allow starting migration with the device, but > > > > > defer completing it. > > > > > > > > > > So > > > > > > > > > > 1.- host tells guest to start tracking memory writes > > > > > 2.- guest acks > > > > > 3.- migration starts > > > > > 4.- most memory is migrated > > > > > 5.- host tells guest to eject device > > > > > 6.- guest acks > > > > > 7.- stop vm and migrate rest of state > > > > > > > > > > > > > > > It will already be a win since hot unplug after migration starts and > > > > > most memory has been migrated is better than hot unplug before migration > > > > > starts. > > > > > > > > > > Then measure downtime and profile. Then we can look at ways > > > > > to quiesce device faster which really means step 5 is replaced > > > > > with "host tells guest to quiesce device and dirty (or just unmap!) > > > > > all memory mapped for write by device". > > > > > > > > > > > > Doing a hot-unplug is going to upset the guests network stacks view > > > > of the world; that's something we don't want to change. > > > > > > > > Dave > > > > > > It might but if you store the IP and restore it quickly > > > after migration e.g. using guest agent, as opposed to DHCP, > > > then it won't. > > > > I thought if you hot-unplug then it will lose any outstanding connections > > on that device. > > Which connections and which device? TCP connections and an ethernet > device? These are on different layers so of course you don't lose them. > Just do not change the IP address. > > Some guests send a signal to applications to close connections > when all links go down. One can work around this > in a variety of ways. So, OK, I was surprised that a simple connection didn't go down when I tested and just removed the network card; I'd thought stuff was more aggressive when there was no route. But as you say, some stuff does close connections when the links go down/away so we do need to work around that; and any new outgoing connections get a 'no route to host'. So I'm still nervous what will break. Dave > > > > It allows calming the device down in a generic way, > > > specific drivers can then implement the fast quiesce. > > > > Except that if it breaks the guest networking it's useless. > > > > Dave > > > > > > > > > > > > > > > -- > > > > > MST > > > > -- > > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > -- > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | "Michael S. Tsirkin" <mst@redhat.com> |
|---|---|
| Date | 2016-01-05 14:20 +0100 |
| Subject | Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking |
| Message-ID | <qNzGO-3gi-7@gated-at.bofh.it> |
| In reply to | #1301504 |
On Tue, Jan 05, 2016 at 12:43:03PM +0000, Dr. David Alan Gilbert wrote: > * Michael S. Tsirkin (mst@redhat.com) wrote: > > On Tue, Jan 05, 2016 at 10:45:25AM +0000, Dr. David Alan Gilbert wrote: > > > * Michael S. Tsirkin (mst@redhat.com) wrote: > > > > On Tue, Jan 05, 2016 at 10:01:04AM +0000, Dr. David Alan Gilbert wrote: > > > > > * Michael S. Tsirkin (mst@redhat.com) wrote: > > > > > > On Mon, Jan 04, 2016 at 07:11:25PM -0800, Alexander Duyck wrote: > > > > > > > >> The two mechanisms referenced above would likely require coordination with > > > > > > > >> QEMU and as such are open to discussion. I haven't attempted to address > > > > > > > >> them as I am not sure there is a consensus as of yet. My personal > > > > > > > >> preference would be to add a vendor-specific configuration block to the > > > > > > > >> emulated pci-bridge interfaces created by QEMU that would allow us to > > > > > > > >> essentially extend shpc to support guest live migration with pass-through > > > > > > > >> devices. > > > > > > > > > > > > > > > > shpc? > > > > > > > > > > > > > > That is kind of what I was thinking. We basically need some mechanism > > > > > > > to allow for the host to ask the device to quiesce. It has been > > > > > > > proposed to possibly even look at something like an ACPI interface > > > > > > > since I know ACPI is used by QEMU to manage hot-plug in the standard > > > > > > > case. > > > > > > > > > > > > > > - Alex > > > > > > > > > > > > > > > > > > Start by using hot-unplug for this! > > > > > > > > > > > > Really use your patch guest side, and write host side > > > > > > to allow starting migration with the device, but > > > > > > defer completing it. > > > > > > > > > > > > So > > > > > > > > > > > > 1.- host tells guest to start tracking memory writes > > > > > > 2.- guest acks > > > > > > 3.- migration starts > > > > > > 4.- most memory is migrated > > > > > > 5.- host tells guest to eject device > > > > > > 6.- guest acks > > > > > > 7.- stop vm and migrate rest of state > > > > > > > > > > > > > > > > > > It will already be a win since hot unplug after migration starts and > > > > > > most memory has been migrated is better than hot unplug before migration > > > > > > starts. > > > > > > > > > > > > Then measure downtime and profile. Then we can look at ways > > > > > > to quiesce device faster which really means step 5 is replaced > > > > > > with "host tells guest to quiesce device and dirty (or just unmap!) > > > > > > all memory mapped for write by device". > > > > > > > > > > > > > > > Doing a hot-unplug is going to upset the guests network stacks view > > > > > of the world; that's something we don't want to change. > > > > > > > > > > Dave > > > > > > > > It might but if you store the IP and restore it quickly > > > > after migration e.g. using guest agent, as opposed to DHCP, > > > > then it won't. > > > > > > I thought if you hot-unplug then it will lose any outstanding connections > > > on that device. > > > > Which connections and which device? TCP connections and an ethernet > > device? These are on different layers so of course you don't lose them. > > Just do not change the IP address. > > > > Some guests send a signal to applications to close connections > > when all links go down. One can work around this > > in a variety of ways. > > So, OK, I was surprised that a simple connection didn't go down when > I tested and just removed the network card; I'd thought stuff was more > aggressive when there was no route. > But as you say, some stuff does close connections when the links go down/away > so we do need to work around that; and any new outgoing connections get > a 'no route to host'. You can create a dummy device in guest for the duration of migration. Use guest agent to move IP address there and that should be enough to trick most guests. > So I'm still nervous what will break. > > Dave I'm not saying nothing breaks. Far being from it. For example, some NAT or firewall implementations keep state per interface and these might lose state (if using NAT/stateful firewall within guest). So yes it *would* be useful to teach guests, for example, that a device is "not dead, just resting" and that another device will shortly come and take its place. But the simple setup is already useful and worth supporting, and merging things gradually will help this project finally get off the ground. > > > > > > It allows calming the device down in a generic way, > > > > specific drivers can then implement the fast quiesce. > > > > > > Except that if it breaks the guest networking it's useless. > > > > > > Dave > > > > > > > > > > > > > > > > > > > -- > > > > > > MST > > > > > -- > > > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > > -- > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Alexander Duyck <alexander.duyck@gmail.com> |
|---|---|
| Date | 2016-01-05 17:20 +0100 |
| Message-ID | <qNCv1-5eI-31@gated-at.bofh.it> |
| In reply to | #1301377 |
On Tue, Jan 5, 2016 at 1:40 AM, Michael S. Tsirkin <mst@redhat.com> wrote: > On Mon, Jan 04, 2016 at 07:11:25PM -0800, Alexander Duyck wrote: >> >> The two mechanisms referenced above would likely require coordination with >> >> QEMU and as such are open to discussion. I haven't attempted to address >> >> them as I am not sure there is a consensus as of yet. My personal >> >> preference would be to add a vendor-specific configuration block to the >> >> emulated pci-bridge interfaces created by QEMU that would allow us to >> >> essentially extend shpc to support guest live migration with pass-through >> >> devices. >> > >> > shpc? >> >> That is kind of what I was thinking. We basically need some mechanism >> to allow for the host to ask the device to quiesce. It has been >> proposed to possibly even look at something like an ACPI interface >> since I know ACPI is used by QEMU to manage hot-plug in the standard >> case. >> >> - Alex > > > Start by using hot-unplug for this! > > Really use your patch guest side, and write host side > to allow starting migration with the device, but > defer completing it. Yeah, I'm fully on board with this idea, though I'm not really working on this right now since last I knew the folks on this thread from Intel were working on it. My patches were mostly meant to be a nudge in this direction so that we could get away from the driver specific code. > So > > 1.- host tells guest to start tracking memory writes > 2.- guest acks > 3.- migration starts > 4.- most memory is migrated > 5.- host tells guest to eject device > 6.- guest acks > 7.- stop vm and migrate rest of state > Sounds about right. The only way this differs from what I see as the final solution for this is that instead of fully ejecting the device in step 5 the driver would instead pause the device and give the host something like 10 seconds to stop the VM and resume with the same device connected if it is available. We would probably also need to look at a solution that would force the device to be ejected or abort prior to starting the migration if it doesn't give us the ack in step 2. > It will already be a win since hot unplug after migration starts and > most memory has been migrated is better than hot unplug before migration > starts. Right. Generally the longer the VF can be maintained as a part of the guest the longer the network performance is improved versus using a purely virtual interface. > Then measure downtime and profile. Then we can look at ways > to quiesce device faster which really means step 5 is replaced > with "host tells guest to quiesce device and dirty (or just unmap!) > all memory mapped for write by device". Step 5 will be the spot where we really need to start modifying drivers. Specifically we probably need to go through and clean-up things so that we can reduce as many of the delays in the driver suspend/resume path as possible. I suspect there is quite a bit that can be done there that would probably also improve boot and shutdown times since those are also impacted by the devices. - Alex -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web