1Mapped-ram 2========== 3 4Mapped-ram is a new stream format for the RAM section designed to 5supplement the existing ``file:`` migration and make it compatible 6with ``multifd``. This enables parallel migration of a guest's RAM to 7a file. 8 9The core of the feature is to ensure that RAM pages are mapped 10directly to offsets in the resulting migration file. This enables the 11``multifd`` threads to write exclusively to those offsets even if the 12guest is constantly dirtying pages (i.e. live migration). Another 13benefit is that the resulting file will have a bounded size, since 14pages which are dirtied multiple times will always go to a fixed 15location in the file, rather than constantly being added to a 16sequential stream. Having the pages at fixed offsets also allows the 17usage of O_DIRECT for save/restore of the migration stream as the 18pages are ensured to be written respecting O_DIRECT alignment 19restrictions (direct-io support not yet implemented). 20 21Usage 22----- 23 24On both source and destination, enable the ``multifd`` and 25``mapped-ram`` capabilities: 26 27 ``migrate_set_capability multifd on`` 28 29 ``migrate_set_capability mapped-ram on`` 30 31Use a ``file:`` URL for migration: 32 33 ``migrate file:/path/to/migration/file`` 34 35Mapped-ram migration is best done non-live, i.e. by stopping the VM on 36the source side before migrating. 37 38Use-cases 39--------- 40 41The mapped-ram feature was designed for use cases where the migration 42stream will be directed to a file in the filesystem and not 43immediately restored on the destination VM [#]_. These could be 44thought of as snapshots. We can further categorize them into live and 45non-live. 46 47- Non-live snapshot 48 49If the use case requires a VM to be stopped before taking a snapshot, 50that's the ideal scenario for mapped-ram migration. Not having to 51track dirty pages, the migration will write the RAM pages to the disk 52as fast as it can. 53 54Note: if a snapshot is taken of a running VM, but the VM will be 55stopped after the snapshot by the admin, then consider stopping it 56right before the snapshot to take benefit of the performance gains 57mentioned above. 58 59- Live snapshot 60 61If the use case requires that the VM keeps running during and after 62the snapshot operation, then mapped-ram migration can still be used, 63but will be less performant. Other strategies such as 64background-snapshot should be evaluated as well. One benefit of 65mapped-ram in this scenario is portability since background-snapshot 66depends on async dirty tracking (KVM_GET_DIRTY_LOG) which is not 67supported outside of Linux. 68 69.. [#] While this same effect could be obtained with the usage of 70 snapshots or the ``file:`` migration alone, mapped-ram provides 71 a performance increase for VMs with larger RAM sizes (10s to 72 100s of GiBs), specially if the VM has been stopped beforehand. 73 74RAM section format 75------------------ 76 77Instead of having a sequential stream of pages that follow the 78RAMBlock headers, the dirty pages for a RAMBlock follow its header 79instead. This ensures that each RAM page has a fixed offset in the 80resulting migration file. 81 82A bitmap is introduced to track which pages have been written in the 83migration file. Pages are written at a fixed location for every 84ramblock. Zero pages are ignored as they'd be zero in the destination 85migration as well. 86 87:: 88 89 Without mapped-ram: With mapped-ram: 90 91 --------------------- -------------------------------- 92 | ramblock 1 header | | ramblock 1 header | 93 --------------------- -------------------------------- 94 | ramblock 2 header | | ramblock 1 mapped-ram header | 95 --------------------- -------------------------------- 96 | ... | | padding to next 1MB boundary | 97 --------------------- | ... | 98 | ramblock n header | -------------------------------- 99 --------------------- | ramblock 1 pages | 100 | RAM_SAVE_FLAG_EOS | | ... | 101 --------------------- -------------------------------- 102 | stream of pages | | ramblock 2 header | 103 | (iter 1) | -------------------------------- 104 | ... | | ramblock 2 mapped-ram header | 105 --------------------- -------------------------------- 106 | RAM_SAVE_FLAG_EOS | | padding to next 1MB boundary | 107 --------------------- | ... | 108 | stream of pages | -------------------------------- 109 | (iter 2) | | ramblock 2 pages | 110 | ... | | ... | 111 --------------------- -------------------------------- 112 | ... | | ... | 113 --------------------- -------------------------------- 114 | RAM_SAVE_FLAG_EOS | 115 -------------------------------- 116 | ... | 117 -------------------------------- 118 119where: 120 - ramblock header: the generic information for a ramblock, such as 121 idstr, used_len, etc. 122 123 - ramblock mapped-ram header: the information added by this feature: 124 bitmap of pages written, bitmap size and offset of pages in the 125 migration file. 126 127Restrictions 128------------ 129 130Since pages are written to their relative offsets and out of order 131(due to the memory dirtying patterns), streaming channels such as 132sockets are not supported. A seekable channel such as a file is 133required. This can be verified in the QIOChannel by the presence of 134the QIO_CHANNEL_FEATURE_SEEKABLE. 135 136The improvements brought by this feature apply only to guest physical 137RAM. Other types of memory such as VRAM are migrated as part of device 138states. 139