1*9cd9313fSDavid HildenbrandQEMU VM templating 2*9cd9313fSDavid Hildenbrand================== 3*9cd9313fSDavid Hildenbrand 4*9cd9313fSDavid HildenbrandThis document explains how to use VM templating in QEMU. 5*9cd9313fSDavid Hildenbrand 6*9cd9313fSDavid HildenbrandFor now, the focus is on VM memory aspects, and not about how to save and 7*9cd9313fSDavid Hildenbrandrestore other VM state (i.e., migrate-to-file with ``x-ignore-shared``). 8*9cd9313fSDavid Hildenbrand 9*9cd9313fSDavid HildenbrandOverview 10*9cd9313fSDavid Hildenbrand-------- 11*9cd9313fSDavid Hildenbrand 12*9cd9313fSDavid HildenbrandWith VM templating, a single template VM serves as the starting point for 13*9cd9313fSDavid Hildenbrandnew VMs. This allows for fast and efficient replication of VMs, resulting 14*9cd9313fSDavid Hildenbrandin fast startup times and reduced memory consumption. 15*9cd9313fSDavid Hildenbrand 16*9cd9313fSDavid HildenbrandConceptually, the VM state is frozen, to then be used as a basis for new 17*9cd9313fSDavid HildenbrandVMs. The Copy-On-Write mechanism in the operating systems makes sure that 18*9cd9313fSDavid Hildenbrandnew VMs are able to read template VM memory; however, any modifications 19*9cd9313fSDavid Hildenbrandstay private and don't modify the original template VM or any other 20*9cd9313fSDavid Hildenbrandcreated VM. 21*9cd9313fSDavid Hildenbrand 22*9cd9313fSDavid Hildenbrand!!! Security Alert !!! 23*9cd9313fSDavid Hildenbrand---------------------- 24*9cd9313fSDavid Hildenbrand 25*9cd9313fSDavid HildenbrandWhen effectively cloning VMs by VM templating, hardware identifiers 26*9cd9313fSDavid Hildenbrand(such as UUIDs and NIC MAC addresses), and similar data in the guest OS 27*9cd9313fSDavid Hildenbrand(such as machine IDs, SSH keys, certificates) that are supposed to be 28*9cd9313fSDavid Hildenbrand*unique* are no longer unique, which can be a security concern. 29*9cd9313fSDavid Hildenbrand 30*9cd9313fSDavid HildenbrandPlease be aware of these implications and how to mitigate them for your 31*9cd9313fSDavid Hildenbranduse case, which might involve vmgenid, hot(un)plug of NIC, etc.. 32*9cd9313fSDavid Hildenbrand 33*9cd9313fSDavid HildenbrandMemory configuration 34*9cd9313fSDavid Hildenbrand-------------------- 35*9cd9313fSDavid Hildenbrand 36*9cd9313fSDavid HildenbrandIn order to create the template VM, we have to make sure that VM memory 37*9cd9313fSDavid Hildenbrandends up in a file, from where it can be reused for the new VMs: 38*9cd9313fSDavid Hildenbrand 39*9cd9313fSDavid HildenbrandSupply VM RAM via memory-backend-file, with ``share=on`` (modifications go 40*9cd9313fSDavid Hildenbrandto the file) and ``readonly=off`` (open the file writable). Note that 41*9cd9313fSDavid Hildenbrand``readonly=off`` is implicit. 42*9cd9313fSDavid Hildenbrand 43*9cd9313fSDavid HildenbrandIn the following command-line example, a 2GB VM is created, whereby VM RAM 44*9cd9313fSDavid Hildenbrandis to be stored in the ``template`` file. 45*9cd9313fSDavid Hildenbrand 46*9cd9313fSDavid Hildenbrand.. parsed-literal:: 47*9cd9313fSDavid Hildenbrand 48*9cd9313fSDavid Hildenbrand |qemu_system| [...] -m 2g \\ 49*9cd9313fSDavid Hildenbrand -object memory-backend-file,id=pc.ram,mem-path=template,size=2g,share=on,... \\ 50*9cd9313fSDavid Hildenbrand -machine q35,memory-backend=pc.ram 51*9cd9313fSDavid Hildenbrand 52*9cd9313fSDavid HildenbrandIf multiple memory backends are used (vNUMA, DIMMs), configure all 53*9cd9313fSDavid Hildenbrandmemory backends accordingly. 54*9cd9313fSDavid Hildenbrand 55*9cd9313fSDavid HildenbrandOnce the VM is in the desired state, stop the VM and save other VM state, 56*9cd9313fSDavid Hildenbrandleaving the current state of VM RAM reside in the file. 57*9cd9313fSDavid Hildenbrand 58*9cd9313fSDavid HildenbrandIn order to have a new VM be based on a template VM, we have to 59*9cd9313fSDavid Hildenbrandconfigure VM RAM to be based on a template VM RAM file; however, the VM 60*9cd9313fSDavid Hildenbrandshould not be able to modify file content. 61*9cd9313fSDavid Hildenbrand 62*9cd9313fSDavid HildenbrandSupply VM RAM via memory-backend-file, with ``share=off`` (modifications 63*9cd9313fSDavid Hildenbrandstay private), ``readonly=on`` (open the file readonly) and ``rom=off`` 64*9cd9313fSDavid Hildenbrand(don't make the memory readonly for the VM). Note that ``share=off`` is 65*9cd9313fSDavid Hildenbrandimplicit and that other VM state has to be restored separately. 66*9cd9313fSDavid Hildenbrand 67*9cd9313fSDavid HildenbrandIn the following command-line example, a 2GB VM is created based on the 68*9cd9313fSDavid Hildenbrandexisting 2GB file ``template``. 69*9cd9313fSDavid Hildenbrand 70*9cd9313fSDavid Hildenbrand.. parsed-literal:: 71*9cd9313fSDavid Hildenbrand 72*9cd9313fSDavid Hildenbrand |qemu_system| [...] -m 2g \\ 73*9cd9313fSDavid Hildenbrand -object memory-backend-file,id=pc.ram,mem-path=template,size=2g,readonly=on,rom=off,... \\ 74*9cd9313fSDavid Hildenbrand -machine q35,memory-backend=pc.ram 75*9cd9313fSDavid Hildenbrand 76*9cd9313fSDavid HildenbrandIf multiple memory backends are used (vNUMA, DIMMs), configure all 77*9cd9313fSDavid Hildenbrandmemory backends accordingly. 78*9cd9313fSDavid Hildenbrand 79*9cd9313fSDavid HildenbrandNote that ``-mem-path`` cannot be used for VM templating when creating the 80*9cd9313fSDavid Hildenbrandtemplate VM or when starting new VMs based on a template VM. 81*9cd9313fSDavid Hildenbrand 82*9cd9313fSDavid HildenbrandIncompatible features 83*9cd9313fSDavid Hildenbrand--------------------- 84*9cd9313fSDavid Hildenbrand 85*9cd9313fSDavid HildenbrandSome features are incompatible with VM templating, as the underlying file 86*9cd9313fSDavid Hildenbrandcannot be modified to discard VM RAM, or to actually share memory with 87*9cd9313fSDavid Hildenbrandanother process. 88*9cd9313fSDavid Hildenbrand 89*9cd9313fSDavid Hildenbrandvhost-user and multi-process QEMU 90*9cd9313fSDavid Hildenbrand~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 91*9cd9313fSDavid Hildenbrand 92*9cd9313fSDavid Hildenbrandvhost-user and multi-process QEMU are incompatible with VM templating. 93*9cd9313fSDavid HildenbrandThese technologies rely on shared memory, however, the template VMs 94*9cd9313fSDavid Hildenbranddon't actually share memory (``share=off``), even though they are 95*9cd9313fSDavid Hildenbrandfile-based. 96*9cd9313fSDavid Hildenbrand 97*9cd9313fSDavid Hildenbrandvirtio-balloon 98*9cd9313fSDavid Hildenbrand~~~~~~~~~~~~~~ 99*9cd9313fSDavid Hildenbrand 100*9cd9313fSDavid Hildenbrandvirtio-balloon inflation and "free page reporting" cannot discard VM RAM 101*9cd9313fSDavid Hildenbrandand will repeatedly report errors. While virtio-balloon can be used 102*9cd9313fSDavid Hildenbrandfor template VMs (e.g., report VM RAM stats), "free page reporting" 103*9cd9313fSDavid Hildenbrandshould be disabled and the balloon should not be inflated. 104*9cd9313fSDavid Hildenbrand 105*9cd9313fSDavid Hildenbrandvirtio-mem 106*9cd9313fSDavid Hildenbrand~~~~~~~~~~ 107*9cd9313fSDavid Hildenbrand 108*9cd9313fSDavid Hildenbrandvirtio-mem cannot discard VM RAM that is managed by the virtio-mem 109*9cd9313fSDavid Hildenbranddevice. virtio-mem will fail early when realizing the device. To use 110*9cd9313fSDavid HildenbrandVM templating with virtio-mem, either hotplug virtio-mem devices to the 111*9cd9313fSDavid Hildenbrandnew VM, or don't supply any memory to the template VM using virtio-mem 112*9cd9313fSDavid Hildenbrand(requested-size=0), not using a template VM file as memory backend for the 113*9cd9313fSDavid Hildenbrandvirtio-mem device. 114*9cd9313fSDavid Hildenbrand 115*9cd9313fSDavid HildenbrandVM migration 116*9cd9313fSDavid Hildenbrand~~~~~~~~~~~~ 117*9cd9313fSDavid Hildenbrand 118*9cd9313fSDavid HildenbrandFor VM migration, "x-release-ram" similarly relies on discarding of VM 119*9cd9313fSDavid HildenbrandRAM on the migration source to free up migrated RAM, and will 120*9cd9313fSDavid Hildenbrandrepeatedly report errors. 121*9cd9313fSDavid Hildenbrand 122*9cd9313fSDavid HildenbrandPostcopy live migration fails discarding VM RAM on the migration 123*9cd9313fSDavid Hildenbranddestination early and refuses to activate postcopy live migration. Note 124*9cd9313fSDavid Hildenbrandthat postcopy live migration usually only works on selected filesystems 125*9cd9313fSDavid Hildenbrand(shmem/tmpfs, hugetlbfs) either way. 126