17f65ce83SAlberto Garciaqcow2 L2/refcount cache configuration 27f65ce83SAlberto Garcia===================================== 3*be820971SAlberto GarciaCopyright (C) 2015, 2018 Igalia, S.L. 47f65ce83SAlberto GarciaAuthor: Alberto Garcia <berto@igalia.com> 57f65ce83SAlberto Garcia 67f65ce83SAlberto GarciaThis work is licensed under the terms of the GNU GPL, version 2 or 77f65ce83SAlberto Garcialater. See the COPYING file in the top-level directory. 87f65ce83SAlberto Garcia 97f65ce83SAlberto GarciaIntroduction 107f65ce83SAlberto Garcia------------ 117f65ce83SAlberto GarciaThe QEMU qcow2 driver has two caches that can improve the I/O 127f65ce83SAlberto Garciaperformance significantly. However, setting the right cache sizes is 137f65ce83SAlberto Garcianot a straightforward operation. 147f65ce83SAlberto Garcia 157f65ce83SAlberto GarciaThis document attempts to give an overview of the L2 and refcount 167f65ce83SAlberto Garciacaches, and how to configure them. 177f65ce83SAlberto Garcia 18f3fdeb9cSPhilippe Mathieu-DaudéPlease refer to the docs/interop/qcow2.txt file for an in-depth 197f65ce83SAlberto Garciatechnical description of the qcow2 file format. 207f65ce83SAlberto Garcia 217f65ce83SAlberto Garcia 227f65ce83SAlberto GarciaClusters 237f65ce83SAlberto Garcia-------- 247f65ce83SAlberto GarciaA qcow2 file is organized in units of constant size called clusters. 257f65ce83SAlberto Garcia 267f65ce83SAlberto GarciaThe cluster size is configurable, but it must be a power of two and 277f65ce83SAlberto Garciaits value 512 bytes or higher. QEMU currently defaults to 64 KB 287f65ce83SAlberto Garciaclusters, and it does not support sizes larger than 2MB. 297f65ce83SAlberto Garcia 307f65ce83SAlberto GarciaThe 'qemu-img create' command supports specifying the size using the 317f65ce83SAlberto Garciacluster_size option: 327f65ce83SAlberto Garcia 337f65ce83SAlberto Garcia qemu-img create -f qcow2 -o cluster_size=128K hd.qcow2 4G 347f65ce83SAlberto Garcia 357f65ce83SAlberto Garcia 367f65ce83SAlberto GarciaThe L2 tables 377f65ce83SAlberto Garcia------------- 387f65ce83SAlberto GarciaThe qcow2 format uses a two-level structure to map the virtual disk as 397f65ce83SAlberto Garciaseen by the guest to the disk image in the host. These structures are 407f65ce83SAlberto Garciacalled the L1 and L2 tables. 417f65ce83SAlberto Garcia 427f65ce83SAlberto GarciaThere is one single L1 table per disk image. The table is small and is 437f65ce83SAlberto Garciaalways kept in memory. 447f65ce83SAlberto Garcia 457f65ce83SAlberto GarciaThere can be many L2 tables, depending on how much space has been 467f65ce83SAlberto Garciaallocated in the image. Each table is one cluster in size. In order to 477f65ce83SAlberto Garciaread or write data from the virtual disk, QEMU needs to read its 487f65ce83SAlberto Garciacorresponding L2 table to find out where that data is located. Since 497f65ce83SAlberto Garciareading the table for each I/O operation can be expensive, QEMU keeps 507f65ce83SAlberto Garciaan L2 cache in memory to speed up disk access. 517f65ce83SAlberto Garcia 527f65ce83SAlberto GarciaThe size of the L2 cache can be configured, and setting the right 537f65ce83SAlberto Garciavalue can improve the I/O performance significantly. 547f65ce83SAlberto Garcia 557f65ce83SAlberto Garcia 567f65ce83SAlberto GarciaThe refcount blocks 577f65ce83SAlberto Garcia------------------- 587f65ce83SAlberto GarciaThe qcow2 format also mantains a reference count for each cluster. 597f65ce83SAlberto GarciaReference counts are used for cluster allocation and internal 607f65ce83SAlberto Garciasnapshots. The data is stored in a two-level structure similar to the 617f65ce83SAlberto GarciaL1/L2 tables described above. 627f65ce83SAlberto Garcia 637f65ce83SAlberto GarciaThe second level structures are called refcount blocks, are also one 647f65ce83SAlberto Garciacluster in size and the number is also variable and dependent on the 657f65ce83SAlberto Garciaamount of allocated space. 667f65ce83SAlberto Garcia 677f65ce83SAlberto GarciaEach block contains a number of refcount entries. Their size (in bits) 687f65ce83SAlberto Garciais a power of two and must not be higher than 64. It defaults to 16 697f65ce83SAlberto Garciabits, but a different value can be set using the refcount_bits option: 707f65ce83SAlberto Garcia 717f65ce83SAlberto Garcia qemu-img create -f qcow2 -o refcount_bits=8 hd.qcow2 4G 727f65ce83SAlberto Garcia 737f65ce83SAlberto GarciaQEMU keeps a refcount cache to speed up I/O much like the 747f65ce83SAlberto Garciaaforementioned L2 cache, and its size can also be configured. 757f65ce83SAlberto Garcia 767f65ce83SAlberto Garcia 777f65ce83SAlberto GarciaChoosing the right cache sizes 787f65ce83SAlberto Garcia------------------------------ 797f65ce83SAlberto GarciaIn order to choose the cache sizes we need to know how they relate to 807f65ce83SAlberto Garciathe amount of allocated space. 817f65ce83SAlberto Garcia 827f65ce83SAlberto GarciaThe amount of virtual disk that can be mapped by the L2 and refcount 837f65ce83SAlberto Garciacaches (in bytes) is: 847f65ce83SAlberto Garcia 857f65ce83SAlberto Garcia disk_size = l2_cache_size * cluster_size / 8 867f65ce83SAlberto Garcia disk_size = refcount_cache_size * cluster_size * 8 / refcount_bits 877f65ce83SAlberto Garcia 887f65ce83SAlberto GarciaWith the default values for cluster_size (64KB) and refcount_bits 897f65ce83SAlberto Garcia(16), that is 907f65ce83SAlberto Garcia 917f65ce83SAlberto Garcia disk_size = l2_cache_size * 8192 927f65ce83SAlberto Garcia disk_size = refcount_cache_size * 32768 937f65ce83SAlberto Garcia 947f65ce83SAlberto GarciaSo in order to cover n GB of disk space with the default values we 957f65ce83SAlberto Garcianeed: 967f65ce83SAlberto Garcia 977f65ce83SAlberto Garcia l2_cache_size = disk_size_GB * 131072 987f65ce83SAlberto Garcia refcount_cache_size = disk_size_GB * 32768 997f65ce83SAlberto Garcia 1007f65ce83SAlberto GarciaQEMU has a default L2 cache of 1MB (1048576 bytes) and a refcount 1017f65ce83SAlberto Garciacache of 256KB (262144 bytes), so using the formulas we've just seen 1027f65ce83SAlberto Garciawe have 1037f65ce83SAlberto Garcia 1047f65ce83SAlberto Garcia 1048576 / 131072 = 8 GB of virtual disk covered by that cache 1057f65ce83SAlberto Garcia 262144 / 32768 = 8 GB 1067f65ce83SAlberto Garcia 1077f65ce83SAlberto Garcia 1087f65ce83SAlberto GarciaHow to configure the cache sizes 1097f65ce83SAlberto Garcia-------------------------------- 1107f65ce83SAlberto GarciaCache sizes can be configured using the -drive option in the 1117f65ce83SAlberto Garciacommand-line, or the 'blockdev-add' QMP command. 1127f65ce83SAlberto Garcia 1137f65ce83SAlberto GarciaThere are three options available, and all of them take bytes: 1147f65ce83SAlberto Garcia 1157f65ce83SAlberto Garcia"l2-cache-size": maximum size of the L2 table cache 1167f65ce83SAlberto Garcia"refcount-cache-size": maximum size of the refcount block cache 1177f65ce83SAlberto Garcia"cache-size": maximum size of both caches combined 1187f65ce83SAlberto Garcia 1197f65ce83SAlberto GarciaThere are two things that need to be taken into account: 1207f65ce83SAlberto Garcia 121*be820971SAlberto Garcia - Both caches must have a size that is a multiple of the cluster size 122*be820971SAlberto Garcia (or the cache entry size: see "Using smaller cache sizes" below). 1237f65ce83SAlberto Garcia 1247f65ce83SAlberto Garcia - If you only set one of the options above, QEMU will automatically 1257f65ce83SAlberto Garcia adjust the others so that the L2 cache is 4 times bigger than the 1267f65ce83SAlberto Garcia refcount cache. 1277f65ce83SAlberto Garcia 1287f65ce83SAlberto GarciaThis means that these options are equivalent: 1297f65ce83SAlberto Garcia 1307f65ce83SAlberto Garcia -drive file=hd.qcow2,l2-cache-size=2097152 1317f65ce83SAlberto Garcia -drive file=hd.qcow2,refcount-cache-size=524288 1327f65ce83SAlberto Garcia -drive file=hd.qcow2,cache-size=2621440 1337f65ce83SAlberto Garcia 1347f65ce83SAlberto GarciaThe reason for this 1/4 ratio is to ensure that both caches cover the 1357f65ce83SAlberto Garciasame amount of disk space. Note however that this is only valid with 1367f65ce83SAlberto Garciathe default value of refcount_bits (16). If you are using a different 1377f65ce83SAlberto Garciavalue you might want to calculate both cache sizes yourself since QEMU 1387f65ce83SAlberto Garciawill always use the same 1/4 ratio. 1397f65ce83SAlberto Garcia 1407f65ce83SAlberto GarciaIt's also worth mentioning that there's no strict need for both caches 1417f65ce83SAlberto Garciato cover the same amount of disk space. The refcount cache is used 1427f65ce83SAlberto Garciamuch less often than the L2 cache, so it's perfectly reasonable to 1437f65ce83SAlberto Garciakeep it small. 1447f65ce83SAlberto Garcia 1457f65ce83SAlberto Garcia 146*be820971SAlberto GarciaUsing smaller cache entries 147*be820971SAlberto Garcia--------------------------- 148*be820971SAlberto GarciaThe qcow2 L2 cache stores complete tables by default. This means that 149*be820971SAlberto Garciaif QEMU needs an entry from an L2 table then the whole table is read 150*be820971SAlberto Garciafrom disk and is kept in the cache. If the cache is full then a 151*be820971SAlberto Garciacomplete table needs to be evicted first. 152*be820971SAlberto Garcia 153*be820971SAlberto GarciaThis can be inefficient with large cluster sizes since it results in 154*be820971SAlberto Garciamore disk I/O and wastes more cache memory. 155*be820971SAlberto Garcia 156*be820971SAlberto GarciaSince QEMU 2.12 you can change the size of the L2 cache entry and make 157*be820971SAlberto Garciait smaller than the cluster size. This can be configured using the 158*be820971SAlberto Garcia"l2-cache-entry-size" parameter: 159*be820971SAlberto Garcia 160*be820971SAlberto Garcia -drive file=hd.qcow2,l2-cache-size=2097152,l2-cache-entry-size=4096 161*be820971SAlberto Garcia 162*be820971SAlberto GarciaSome things to take into account: 163*be820971SAlberto Garcia 164*be820971SAlberto Garcia - The L2 cache entry size has the same restrictions as the cluster 165*be820971SAlberto Garcia size (power of two, at least 512 bytes). 166*be820971SAlberto Garcia 167*be820971SAlberto Garcia - Smaller entry sizes generally improve the cache efficiency and make 168*be820971SAlberto Garcia disk I/O faster. This is particularly true with solid state drives 169*be820971SAlberto Garcia so it's a good idea to reduce the entry size in those cases. With 170*be820971SAlberto Garcia rotating hard drives the situation is a bit more complicated so you 171*be820971SAlberto Garcia should test it first and stay with the default size if unsure. 172*be820971SAlberto Garcia 173*be820971SAlberto Garcia - Try different entry sizes to see which one gives faster performance 174*be820971SAlberto Garcia in your case. The block size of the host filesystem is generally a 175*be820971SAlberto Garcia good default (usually 4096 bytes in the case of ext4). 176*be820971SAlberto Garcia 177*be820971SAlberto Garcia - Only the L2 cache can be configured this way. The refcount cache 178*be820971SAlberto Garcia always uses the cluster size as the entry size. 179*be820971SAlberto Garcia 180*be820971SAlberto Garcia - If the L2 cache is big enough to hold all of the image's L2 tables 181*be820971SAlberto Garcia (as explained in the "Choosing the right cache sizes" section 182*be820971SAlberto Garcia earlier in this document) then none of this is necessary and you 183*be820971SAlberto Garcia can omit the "l2-cache-entry-size" parameter altogether. 184*be820971SAlberto Garcia 185*be820971SAlberto Garcia 1867f65ce83SAlberto GarciaReducing the memory usage 1877f65ce83SAlberto Garcia------------------------- 1887f65ce83SAlberto GarciaIt is possible to clean unused cache entries in order to reduce the 1897f65ce83SAlberto Garciamemory usage during periods of low I/O activity. 1907f65ce83SAlberto Garcia 1917f65ce83SAlberto GarciaThe parameter "cache-clean-interval" defines an interval (in seconds). 1927f65ce83SAlberto GarciaAll cache entries that haven't been accessed during that interval are 1937f65ce83SAlberto Garciaremoved from memory. 1947f65ce83SAlberto Garcia 1957f65ce83SAlberto GarciaThis example removes all unused cache entries every 15 minutes: 1967f65ce83SAlberto Garcia 1977f65ce83SAlberto Garcia -drive file=hd.qcow2,cache-clean-interval=900 1987f65ce83SAlberto Garcia 1997f65ce83SAlberto GarciaIf unset, the default value for this parameter is 0 and it disables 2007f65ce83SAlberto Garciathis feature. 2017f65ce83SAlberto Garcia 2027f65ce83SAlberto GarciaNote that this functionality currently relies on the MADV_DONTNEED 2038f577583SAlberto Garciaargument for madvise() to actually free the memory. This is a 2048f577583SAlberto GarciaLinux-specific feature, so cache-clean-interval is not supported in 2058f577583SAlberto Garciaother systems. 206