10d177cddSPierre Morel.. _cpu-topology-s390x: 20d177cddSPierre Morel 30d177cddSPierre MorelCPU topology on s390x 40d177cddSPierre Morel===================== 50d177cddSPierre Morel 60d177cddSPierre MorelSince QEMU 8.2, CPU topology on s390x provides up to 3 levels of 70d177cddSPierre Moreltopology containers: drawers, books and sockets. They define a 80d177cddSPierre Moreltree-shaped hierarchy. 90d177cddSPierre Morel 100d177cddSPierre MorelThe socket container has one or more CPU entries. 110d177cddSPierre MorelEach of these CPU entries consists of a bitmap and three CPU attributes: 120d177cddSPierre Morel 130d177cddSPierre Morel- CPU type 140d177cddSPierre Morel- entitlement 150d177cddSPierre Morel- dedication 160d177cddSPierre Morel 170d177cddSPierre MorelEach bit set in the bitmap correspond to a core-id of a vCPU with matching 180d177cddSPierre Morelattributes. 190d177cddSPierre Morel 200d177cddSPierre MorelThis documentation provides general information on S390 CPU topology, 210d177cddSPierre Morelhow to enable it and explains the new CPU attributes. 220d177cddSPierre MorelFor information on how to modify the S390 CPU topology and how to 230d177cddSPierre Morelmonitor polarization changes, see ``docs/devel/s390-cpu-topology.rst``. 240d177cddSPierre Morel 250d177cddSPierre MorelPrerequisites 260d177cddSPierre Morel------------- 270d177cddSPierre Morel 28*b1769345SClaudio FontanaTo use the CPU topology, you currently need to choose the KVM accelerator. 29*b1769345SClaudio FontanaSee :ref:`Accelerators` for more details about accelerators and how to select them. 30*b1769345SClaudio Fontana 31*b1769345SClaudio FontanaThe s390x host needs to use a Linux kernel v6.0 or newer (which provides the so-called 320d177cddSPierre Morel``KVM_CAP_S390_CPU_TOPOLOGY`` capability that allows QEMU to signal the 330d177cddSPierre MorelCPU topology facility via the so-called STFLE bit 11 to the VM). 340d177cddSPierre Morel 350d177cddSPierre MorelEnabling CPU topology 360d177cddSPierre Morel--------------------- 370d177cddSPierre Morel 38*b1769345SClaudio FontanaCurrently, CPU topology is enabled by default only in the "host" CPU model. 390d177cddSPierre Morel 40*b1769345SClaudio FontanaEnabling CPU topology in another CPU model is done by setting the CPU flag 410d177cddSPierre Morel``ctop`` to ``on`` as in: 420d177cddSPierre Morel 430d177cddSPierre Morel.. code-block:: bash 440d177cddSPierre Morel 450d177cddSPierre Morel -cpu gen16b,ctop=on 460d177cddSPierre Morel 470d177cddSPierre MorelHaving the topology disabled by default allows migration between 480d177cddSPierre Morelold and new QEMU without adding new flags. 490d177cddSPierre Morel 500d177cddSPierre MorelDefault topology usage 510d177cddSPierre Morel---------------------- 520d177cddSPierre Morel 530d177cddSPierre MorelThe CPU topology can be specified on the QEMU command line 540d177cddSPierre Morelwith the ``-smp`` or the ``-device`` QEMU command arguments. 550d177cddSPierre Morel 560d177cddSPierre MorelNote also that since 7.2 threads are no longer supported in the topology 570d177cddSPierre Moreland the ``-smp`` command line argument accepts only ``threads=1``. 580d177cddSPierre Morel 590d177cddSPierre MorelIf none of the containers attributes (drawers, books, sockets) are 600d177cddSPierre Morelspecified for the ``-smp`` flag, the number of these containers 610d177cddSPierre Morelis 1. 620d177cddSPierre Morel 630d177cddSPierre MorelThus the following two options will result in the same topology: 640d177cddSPierre Morel 650d177cddSPierre Morel.. code-block:: bash 660d177cddSPierre Morel 670d177cddSPierre Morel -smp cpus=5,drawer=1,books=1,sockets=8,cores=4,maxcpus=32 680d177cddSPierre Morel 690d177cddSPierre Moreland 700d177cddSPierre Morel 710d177cddSPierre Morel.. code-block:: bash 720d177cddSPierre Morel 730d177cddSPierre Morel -smp cpus=5,sockets=8,cores=4,maxcpus=32 740d177cddSPierre Morel 750d177cddSPierre MorelWhen a CPU is defined by the ``-smp`` command argument, its position 760d177cddSPierre Morelinside the topology is calculated by adding the CPUs to the topology 770d177cddSPierre Morelbased on the core-id starting with core-0 at position 0 of socket-0, 780d177cddSPierre Morelbook-0, drawer-0 and filling all CPUs of socket-0 before filling socket-1 790d177cddSPierre Morelof book-0 and so on up to the last socket of the last book of the last 800d177cddSPierre Moreldrawer. 810d177cddSPierre Morel 820d177cddSPierre MorelWhen a CPU is defined by the ``-device`` command argument, the 830d177cddSPierre Moreltree topology attributes must all be defined or all not defined. 840d177cddSPierre Morel 850d177cddSPierre Morel.. code-block:: bash 860d177cddSPierre Morel 870d177cddSPierre Morel -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 880d177cddSPierre Morel 890d177cddSPierre Morelor 900d177cddSPierre Morel 910d177cddSPierre Morel.. code-block:: bash 920d177cddSPierre Morel 930d177cddSPierre Morel -device gen16b-s390x-cpu,core-id=1,dedicated=true 940d177cddSPierre Morel 950d177cddSPierre MorelIf none of the tree attributes (drawer, book, sockets), are specified 960d177cddSPierre Morelfor the ``-device`` argument, like for all CPUs defined with the ``-smp`` 970d177cddSPierre Morelcommand argument the topology tree attributes will be set by simply 980d177cddSPierre Moreladding the CPUs to the topology based on the core-id. 990d177cddSPierre Morel 1000d177cddSPierre MorelQEMU will not try to resolve collisions and will report an error if the 1010d177cddSPierre MorelCPU topology defined explicitly or implicitly on a ``-device`` 1020d177cddSPierre Morelargument collides with the definition of a CPU implicitly defined 1030d177cddSPierre Morelon the ``-smp`` argument. 1040d177cddSPierre Morel 1050d177cddSPierre MorelWhen the topology modifier attributes are not defined for the 1060d177cddSPierre Morel``-device`` command argument they takes following default values: 1070d177cddSPierre Morel 1080d177cddSPierre Morel- dedicated: ``false`` 1090d177cddSPierre Morel- entitlement: ``medium`` 1100d177cddSPierre Morel 1110d177cddSPierre Morel 1120d177cddSPierre MorelHot plug 1130d177cddSPierre Morel++++++++ 1140d177cddSPierre Morel 1150d177cddSPierre MorelNew CPUs can be plugged using the device_add hmp command as in: 1160d177cddSPierre Morel 1170d177cddSPierre Morel.. code-block:: bash 1180d177cddSPierre Morel 1190d177cddSPierre Morel (qemu) device_add gen16b-s390x-cpu,core-id=9 1200d177cddSPierre Morel 1210d177cddSPierre MorelThe placement of the CPU is derived from the core-id as described above. 1220d177cddSPierre Morel 1230d177cddSPierre MorelThe topology can of course also be fully defined: 1240d177cddSPierre Morel 1250d177cddSPierre Morel.. code-block:: bash 1260d177cddSPierre Morel 1270d177cddSPierre Morel (qemu) device_add gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 1280d177cddSPierre Morel 1290d177cddSPierre Morel 1300d177cddSPierre MorelExamples 1310d177cddSPierre Morel++++++++ 1320d177cddSPierre Morel 1330d177cddSPierre MorelIn the following machine we define 8 sockets with 4 cores each. 1340d177cddSPierre Morel 1350d177cddSPierre Morel.. code-block:: bash 1360d177cddSPierre Morel 137*b1769345SClaudio Fontana $ qemu-system-s390x -accel kvm -m 2G \ 1380d177cddSPierre Morel -cpu gen16b,ctop=on \ 1390d177cddSPierre Morel -smp cpus=5,sockets=8,cores=4,maxcpus=32 \ 1400d177cddSPierre Morel -device host-s390x-cpu,core-id=14 \ 1410d177cddSPierre Morel 1420d177cddSPierre MorelA new CPUs can be plugged using the device_add hmp command as before: 1430d177cddSPierre Morel 1440d177cddSPierre Morel.. code-block:: bash 1450d177cddSPierre Morel 1460d177cddSPierre Morel (qemu) device_add gen16b-s390x-cpu,core-id=9 1470d177cddSPierre Morel 1480d177cddSPierre MorelThe core-id defines the placement of the core in the topology by 1490d177cddSPierre Morelstarting with core 0 in socket 0 up to maxcpus. 1500d177cddSPierre Morel 1510d177cddSPierre MorelIn the example above: 1520d177cddSPierre Morel 1530d177cddSPierre Morel* There are 5 CPUs provided to the guest with the ``-smp`` command line 1540d177cddSPierre Morel They will take the core-ids 0,1,2,3,4 1550d177cddSPierre Morel As we have 4 cores in a socket, we have 4 CPUs provided 1560d177cddSPierre Morel to the guest in socket 0, with core-ids 0,1,2,3. 1570d177cddSPierre Morel The last CPU, with core-id 4, will be on socket 1. 1580d177cddSPierre Morel 1590d177cddSPierre Morel* the core with ID 14 provided by the ``-device`` command line will 1600d177cddSPierre Morel be placed in socket 3, with core-id 14 1610d177cddSPierre Morel 1620d177cddSPierre Morel* the core with ID 9 provided by the ``device_add`` qmp command will 1630d177cddSPierre Morel be placed in socket 2, with core-id 9 1640d177cddSPierre Morel 1650d177cddSPierre Morel 1660d177cddSPierre MorelPolarization, entitlement and dedication 1670d177cddSPierre Morel---------------------------------------- 1680d177cddSPierre Morel 1690d177cddSPierre MorelPolarization 1700d177cddSPierre Morel++++++++++++ 1710d177cddSPierre Morel 1720d177cddSPierre MorelThe polarization affects how the CPUs of a shared host are utilized/distributed 1730d177cddSPierre Morelamong guests. 1740d177cddSPierre MorelThe guest determines the polarization by using the PTF instruction. 1750d177cddSPierre Morel 1760d177cddSPierre MorelPolarization defines two models of CPU provisioning: horizontal 1770d177cddSPierre Moreland vertical. 1780d177cddSPierre Morel 1790d177cddSPierre MorelThe horizontal polarization is the default model on boot and after 1800d177cddSPierre Morelsubsystem reset. When horizontal polarization is in effect all vCPUs should 1810d177cddSPierre Morelhave about equal resource provisioning. 1820d177cddSPierre Morel 1830d177cddSPierre MorelIn the vertical polarization model vCPUs are unequal, but overall more resources 1840d177cddSPierre Morelmight be available. 1850d177cddSPierre MorelThe guest can make use of the vCPU entitlement information provided by the host 1860d177cddSPierre Morelto optimize kernel thread scheduling. 1870d177cddSPierre Morel 1880d177cddSPierre MorelA subsystem reset puts all vCPU of the configuration into the 1890d177cddSPierre Morelhorizontal polarization. 1900d177cddSPierre Morel 1910d177cddSPierre MorelEntitlement 1920d177cddSPierre Morel+++++++++++ 1930d177cddSPierre Morel 1940d177cddSPierre MorelThe vertical polarization specifies that the guest's vCPU can get 1950d177cddSPierre Moreldifferent real CPU provisioning: 1960d177cddSPierre Morel 1970d177cddSPierre Morel- a vCPU with vertical high entitlement specifies that this 1980d177cddSPierre Morel vCPU gets 100% of the real CPU provisioning. 1990d177cddSPierre Morel 2000d177cddSPierre Morel- a vCPU with vertical medium entitlement specifies that this 2010d177cddSPierre Morel vCPU shares the real CPU with other vCPUs. 2020d177cddSPierre Morel 2030d177cddSPierre Morel- a vCPU with vertical low entitlement specifies that this 2040d177cddSPierre Morel vCPU only gets real CPU provisioning when no other vCPUs needs it. 2050d177cddSPierre Morel 2060d177cddSPierre MorelIn the case a vCPU with vertical high entitlement does not use 2070d177cddSPierre Morelthe real CPU, the unused "slack" can be dispatched to other vCPU 2080d177cddSPierre Morelwith medium or low entitlement. 2090d177cddSPierre Morel 2100d177cddSPierre MorelA vCPU can be "dedicated" in which case the vCPU is fully dedicated to a single 2110d177cddSPierre Morelreal CPU. 2120d177cddSPierre Morel 2130d177cddSPierre MorelThe dedicated bit is an indication of affinity of a vCPU for a real CPU 2140d177cddSPierre Morelwhile the entitlement indicates the sharing or exclusivity of use. 2150d177cddSPierre Morel 2160d177cddSPierre MorelDefining the topology on the command line 2170d177cddSPierre Morel----------------------------------------- 2180d177cddSPierre Morel 2190d177cddSPierre MorelThe topology can entirely be defined using -device cpu statements, 2200d177cddSPierre Morelwith the exception of CPU 0 which must be defined with the -smp 2210d177cddSPierre Morelargument. 2220d177cddSPierre Morel 2230d177cddSPierre MorelFor example, here we set the position of the cores 1,2,3 to 2240d177cddSPierre Moreldrawer 1, book 1, socket 2 and cores 0,9 and 14 to drawer 0, 2250d177cddSPierre Morelbook 0, socket 0 without defining entitlement or dedication. 2260d177cddSPierre MorelCore 4 will be set on its default position on socket 1 2270d177cddSPierre Morel(since we have 4 core per socket) and we define it as dedicated and 2280d177cddSPierre Morelwith vertical high entitlement. 2290d177cddSPierre Morel 2300d177cddSPierre Morel.. code-block:: bash 2310d177cddSPierre Morel 232*b1769345SClaudio Fontana $ qemu-system-s390x -accel kvm -m 2G \ 2330d177cddSPierre Morel -cpu gen16b,ctop=on \ 2340d177cddSPierre Morel -smp cpus=1,sockets=8,cores=4,maxcpus=32 \ 2350d177cddSPierre Morel \ 2360d177cddSPierre Morel -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 \ 2370d177cddSPierre Morel -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=2 \ 2380d177cddSPierre Morel -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=3 \ 2390d177cddSPierre Morel \ 2400d177cddSPierre Morel -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=9 \ 2410d177cddSPierre Morel -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=14 \ 2420d177cddSPierre Morel \ 2430d177cddSPierre Morel -device gen16b-s390x-cpu,core-id=4,dedicated=on,entitlement=high 2440d177cddSPierre Morel 2450d177cddSPierre MorelThe entitlement defined for the CPU 4 will only be used after the guest 2460d177cddSPierre Morelsuccessfully enables vertical polarization by using the PTF instruction. 247