1*20e6b156SSteve SistareCheckPoint and Restart (CPR) 2*20e6b156SSteve Sistare============================ 3*20e6b156SSteve Sistare 4*20e6b156SSteve SistareCPR is the umbrella name for a set of migration modes in which the 5*20e6b156SSteve SistareVM is migrated to a new QEMU instance on the same host. It is 6*20e6b156SSteve Sistareintended for use when the goal is to update host software components 7*20e6b156SSteve Sistarethat run the VM, such as QEMU or even the host kernel. At this time, 8*20e6b156SSteve Sistarecpr-reboot is the only available mode. 9*20e6b156SSteve Sistare 10*20e6b156SSteve SistareBecause QEMU is restarted on the same host, with access to the same 11*20e6b156SSteve Sistarelocal devices, CPR is allowed in certain cases where normal migration 12*20e6b156SSteve Sistarewould be blocked. However, the user must not modify the contents of 13*20e6b156SSteve Sistareguest block devices between quitting old QEMU and starting new QEMU. 14*20e6b156SSteve Sistare 15*20e6b156SSteve SistareCPR unconditionally stops VM execution before memory is saved, and 16*20e6b156SSteve Sistarethus does not depend on any form of dirty page tracking. 17*20e6b156SSteve Sistare 18*20e6b156SSteve Sistarecpr-reboot mode 19*20e6b156SSteve Sistare--------------- 20*20e6b156SSteve Sistare 21*20e6b156SSteve SistareIn this mode, QEMU stops the VM, and writes VM state to the migration 22*20e6b156SSteve SistareURI, which will typically be a file. After quitting QEMU, the user 23*20e6b156SSteve Sistareresumes by running QEMU with the ``-incoming`` option. Because the 24*20e6b156SSteve Sistareold and new QEMU instances are not active concurrently, the URI cannot 25*20e6b156SSteve Sistarebe a type that streams data from one instance to the other. 26*20e6b156SSteve Sistare 27*20e6b156SSteve SistareGuest RAM can be saved in place if backed by shared memory, or can be 28*20e6b156SSteve Sistarecopied to a file. The former is more efficient and is therefore 29*20e6b156SSteve Sistarepreferred. 30*20e6b156SSteve Sistare 31*20e6b156SSteve SistareAfter state and memory are saved, the user may update userland host 32*20e6b156SSteve Sistaresoftware before restarting QEMU and resuming the VM. Further, if 33*20e6b156SSteve Sistarethe RAM is backed by persistent shared memory, such as a DAX device, 34*20e6b156SSteve Sistarethen the user may reboot to a new host kernel before restarting QEMU. 35*20e6b156SSteve Sistare 36*20e6b156SSteve SistareThis mode supports VFIO devices provided the user first puts the 37*20e6b156SSteve Sistareguest in the suspended runstate, such as by issuing the 38*20e6b156SSteve Sistare``guest-suspend-ram`` command to the QEMU guest agent. The agent 39*20e6b156SSteve Sistaremust be pre-installed in the guest, and the guest must support 40*20e6b156SSteve Sistaresuspend to RAM. Beware that suspension can take a few seconds, so 41*20e6b156SSteve Sistarethe user should poll to see the suspended state before proceeding 42*20e6b156SSteve Sistarewith the CPR operation. 43*20e6b156SSteve Sistare 44*20e6b156SSteve SistareUsage 45*20e6b156SSteve Sistare^^^^^ 46*20e6b156SSteve Sistare 47*20e6b156SSteve SistareIt is recommended that guest RAM be backed with some type of shared 48*20e6b156SSteve Sistarememory, such as ``memory-backend-file,share=on``, and that the 49*20e6b156SSteve Sistare``x-ignore-shared`` capability be set. This combination allows memory 50*20e6b156SSteve Sistareto be saved in place. Otherwise, after QEMU stops the VM, all guest 51*20e6b156SSteve SistareRAM is copied to the migration URI. 52*20e6b156SSteve Sistare 53*20e6b156SSteve SistareOutgoing: 54*20e6b156SSteve Sistare * Set the migration mode parameter to ``cpr-reboot``. 55*20e6b156SSteve Sistare * Set the ``x-ignore-shared`` capability if desired. 56*20e6b156SSteve Sistare * Issue the ``migrate`` command. It is recommended the the URI be a 57*20e6b156SSteve Sistare ``file`` type, but one can use other types such as ``exec``, 58*20e6b156SSteve Sistare provided the command captures all the data from the outgoing side, 59*20e6b156SSteve Sistare and provides all the data to the incoming side. 60*20e6b156SSteve Sistare * Quit when QEMU reaches the postmigrate state. 61*20e6b156SSteve Sistare 62*20e6b156SSteve SistareIncoming: 63*20e6b156SSteve Sistare * Start QEMU with the ``-incoming defer`` option. 64*20e6b156SSteve Sistare * Set the migration mode parameter to ``cpr-reboot``. 65*20e6b156SSteve Sistare * Set the ``x-ignore-shared`` capability if desired. 66*20e6b156SSteve Sistare * Issue the ``migrate-incoming`` command. 67*20e6b156SSteve Sistare * If the VM was running when the outgoing ``migrate`` command was 68*20e6b156SSteve Sistare issued, then QEMU automatically resumes VM execution. 69*20e6b156SSteve Sistare 70*20e6b156SSteve SistareExample 1 71*20e6b156SSteve Sistare^^^^^^^^^ 72*20e6b156SSteve Sistare:: 73*20e6b156SSteve Sistare 74*20e6b156SSteve Sistare # qemu-kvm -monitor stdio 75*20e6b156SSteve Sistare -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m 4G 76*20e6b156SSteve Sistare ... 77*20e6b156SSteve Sistare 78*20e6b156SSteve Sistare (qemu) info status 79*20e6b156SSteve Sistare VM status: running 80*20e6b156SSteve Sistare (qemu) migrate_set_parameter mode cpr-reboot 81*20e6b156SSteve Sistare (qemu) migrate_set_capability x-ignore-shared on 82*20e6b156SSteve Sistare (qemu) migrate -d file:vm.state 83*20e6b156SSteve Sistare (qemu) info status 84*20e6b156SSteve Sistare VM status: paused (postmigrate) 85*20e6b156SSteve Sistare (qemu) quit 86*20e6b156SSteve Sistare 87*20e6b156SSteve Sistare ### optionally update kernel and reboot 88*20e6b156SSteve Sistare # systemctl kexec 89*20e6b156SSteve Sistare kexec_core: Starting new kernel 90*20e6b156SSteve Sistare ... 91*20e6b156SSteve Sistare 92*20e6b156SSteve Sistare # qemu-kvm ... -incoming defer 93*20e6b156SSteve Sistare (qemu) info status 94*20e6b156SSteve Sistare VM status: paused (inmigrate) 95*20e6b156SSteve Sistare (qemu) migrate_set_parameter mode cpr-reboot 96*20e6b156SSteve Sistare (qemu) migrate_set_capability x-ignore-shared on 97*20e6b156SSteve Sistare (qemu) migrate_incoming file:vm.state 98*20e6b156SSteve Sistare (qemu) info status 99*20e6b156SSteve Sistare VM status: running 100*20e6b156SSteve Sistare 101*20e6b156SSteve SistareExample 2: VFIO 102*20e6b156SSteve Sistare^^^^^^^^^^^^^^^ 103*20e6b156SSteve Sistare:: 104*20e6b156SSteve Sistare 105*20e6b156SSteve Sistare # qemu-kvm -monitor stdio 106*20e6b156SSteve Sistare -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m 4G 107*20e6b156SSteve Sistare -device vfio-pci, ... 108*20e6b156SSteve Sistare -chardev socket,id=qga0,path=qga.sock,server=on,wait=off 109*20e6b156SSteve Sistare -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 110*20e6b156SSteve Sistare ... 111*20e6b156SSteve Sistare 112*20e6b156SSteve Sistare (qemu) info status 113*20e6b156SSteve Sistare VM status: running 114*20e6b156SSteve Sistare 115*20e6b156SSteve Sistare # echo '{"execute":"guest-suspend-ram"}' | ncat --send-only -U qga.sock 116*20e6b156SSteve Sistare 117*20e6b156SSteve Sistare (qemu) info status 118*20e6b156SSteve Sistare VM status: paused (suspended) 119*20e6b156SSteve Sistare (qemu) migrate_set_parameter mode cpr-reboot 120*20e6b156SSteve Sistare (qemu) migrate_set_capability x-ignore-shared on 121*20e6b156SSteve Sistare (qemu) migrate -d file:vm.state 122*20e6b156SSteve Sistare (qemu) info status 123*20e6b156SSteve Sistare VM status: paused (postmigrate) 124*20e6b156SSteve Sistare (qemu) quit 125*20e6b156SSteve Sistare 126*20e6b156SSteve Sistare ### optionally update kernel and reboot 127*20e6b156SSteve Sistare # systemctl kexec 128*20e6b156SSteve Sistare kexec_core: Starting new kernel 129*20e6b156SSteve Sistare ... 130*20e6b156SSteve Sistare 131*20e6b156SSteve Sistare # qemu-kvm ... -incoming defer 132*20e6b156SSteve Sistare (qemu) info status 133*20e6b156SSteve Sistare VM status: paused (inmigrate) 134*20e6b156SSteve Sistare (qemu) migrate_set_parameter mode cpr-reboot 135*20e6b156SSteve Sistare (qemu) migrate_set_capability x-ignore-shared on 136*20e6b156SSteve Sistare (qemu) migrate_incoming file:vm.state 137*20e6b156SSteve Sistare (qemu) info status 138*20e6b156SSteve Sistare VM status: paused (suspended) 139*20e6b156SSteve Sistare (qemu) system_wakeup 140*20e6b156SSteve Sistare (qemu) info status 141*20e6b156SSteve Sistare VM status: running 142*20e6b156SSteve Sistare 143*20e6b156SSteve SistareCaveats 144*20e6b156SSteve Sistare^^^^^^^ 145*20e6b156SSteve Sistare 146*20e6b156SSteve Sistarecpr-reboot mode may not be used with postcopy, background-snapshot, 147*20e6b156SSteve Sistareor COLO. 148