1pSeries family boards (``pseries``) 2=================================== 3 4The Power machine para-virtualized environment described by the `Linux on Power 5Architecture Reference document (LoPAR) 6<https://openpowerfoundation.org/wp-content/uploads/2020/07/LoPAR-20200812.pdf>`_ 7is called pSeries. This environment is also known as sPAPR, System p guests, or 8simply Power Linux guests (although it is capable of running other operating 9systems, such as AIX). 10 11Even though pSeries is designed to behave as a guest environment, it is also 12capable of acting as a hypervisor OS, providing, on that role, nested 13virtualization capabilities. 14 15Supported devices 16----------------- 17 18 * Multi processor support for many Power processors generations: POWER7, 19 POWER7+, POWER8, POWER8NVL, POWER9, and Power10. Support for POWER5+ exists, 20 but its state is unknown. 21 * Interrupt Controller, XICS (POWER8) and XIVE (POWER9 and Power10) 22 * vPHB PCIe Host bridge. 23 * vscsi and vnet devices, compatible with the same devices available on a 24 PowerVM hypervisor with VIOS managing LPARs. 25 * Virtio based devices. 26 * PCIe device pass through. 27 28Missing devices 29--------------- 30 31 * SPICE support. 32 33Firmware 34-------- 35 36`SLOF <https://github.com/aik/SLOF>`_ (Slimline Open Firmware) is an 37implementation of the `IEEE 1275-1994, Standard for Boot (Initialization 38Configuration) Firmware: Core Requirements and Practices 39<https://standards.ieee.org/standard/1275-1994.html>`_. 40 41QEMU includes a prebuilt image of SLOF which is updated when a more recent 42version is required. 43 44Build directions 45---------------- 46 47.. code-block:: bash 48 49 ./configure --target-list=ppc64-softmmu && make 50 51Running instructions 52-------------------- 53 54Someone can select the pSeries machine type by running QEMU with the following 55options: 56 57.. code-block:: bash 58 59 qemu-system-ppc64 -M pseries <other QEMU arguments> 60 61sPAPR devices 62------------- 63 64The sPAPR specification defines a set of para-virtualized devices, which are 65also supported by the pSeries machine in QEMU and can be instantiated with the 66``-device`` option: 67 68* ``spapr-vlan`` : a virtual network interface. 69* ``spapr-vscsi`` : a virtual SCSI disk interface. 70* ``spapr-rng`` : a pseudo-device for passing random number generator data to the 71 guest (see the `H_RANDOM hypercall feature 72 <https://wiki.qemu.org/Features/HRandomHypercall>`_ for details). 73* ``spapr-vty``: a virtual teletype. 74* ``spapr-pci-host-bridge``: a PCI host bridge. 75* ``tpm-spapr``: a Trusted Platform Module (TPM). 76* ``spapr-tpm-proxy``: a TPM proxy. 77 78These are compatible with the devices historically available for use when 79running the IBM PowerVM hypervisor with LPARs. 80 81However, since these devices have originally been specified with another 82hypervisor and non-Linux guests in mind, you should use the virtio counterparts 83(virtio-net, virtio-blk/scsi and virtio-rng for instance) if possible instead, 84since they will most probably give you better performance with Linux guests in a 85QEMU environment. 86 87The pSeries machine in QEMU is always instantiated with the following devices: 88 89* A NVRAM device (``spapr-nvram``). 90* A virtual teletype (``spapr-vty``). 91* A PCI host bridge (``spapr-pci-host-bridge``). 92 93Hence, it is not needed to add them manually, unless you use the ``-nodefaults`` 94command line option in QEMU. 95 96In the case of the default ``spapr-nvram`` device, if someone wants to make the 97contents of the NVRAM device persistent, they will need to specify a PFLASH 98device when starting QEMU, i.e. either use 99``-drive if=pflash,file=<filename>,format=raw`` to set the default PFLASH 100device, or specify one with an ID 101(``-drive if=none,file=<filename>,format=raw,id=pfid``) and pass that ID to the 102NVRAM device with ``-global spapr-nvram.drive=pfid``. 103 104sPAPR specification 105^^^^^^^^^^^^^^^^^^^ 106 107The main source of documentation on the sPAPR standard is the `Linux on Power 108Architecture Reference document (LoPAR) 109<https://openpowerfoundation.org/wp-content/uploads/2020/07/LoPAR-20200812.pdf>`_. 110However, documentation specific to QEMU's implementation of the specification 111can also be found in QEMU documentation: 112 113.. toctree:: 114 :maxdepth: 1 115 116 ../../specs/ppc-spapr-hcalls.rst 117 ../../specs/ppc-spapr-numa.rst 118 ../../specs/ppc-spapr-xive.rst 119 120Other documentation available in QEMU docs directory: 121 122* Hot plug (``/docs/specs/ppc-spapr-hotplug.txt``). 123* Hypervisor calls needed by the Ultravisor 124 (``/docs/specs/ppc-spapr-uv-hcalls.txt``). 125 126Switching between the KVM-PR and KVM-HV kernel module 127----------------------------------------------------- 128 129Currently, there are two implementations of KVM on Power, ``kvm_hv.ko`` and 130``kvm_pr.ko``. 131 132 133If a host supports both KVM modes, and both KVM kernel modules are loaded, it is 134possible to switch between the two modes with the ``kvm-type`` parameter: 135 136* Use ``qemu-system-ppc64 -M pseries,accel=kvm,kvm-type=PR`` to use the 137 ``kvm_pr.ko`` kernel module. 138* Use ``qemu-system-ppc64 -M pseries,accel=kvm,kvm-type=HV`` to use ``kvm_hv.ko`` 139 instead. 140 141KVM-PR 142^^^^^^ 143 144KVM-PR uses the so-called **PR**\ oblem state of the PPC CPUs to run the guests, 145i.e. the virtual machine is run in user mode and all privileged instructions 146trap and have to be emulated by the host. That means you can run KVM-PR inside 147a pSeries guest (or a PowerVM LPAR for that matter), and that is where it has 148originated, as historically (prior to POWER7) it was not possible to run Linux 149on hypervisor mode on a Power processor (this function was restricted to 150PowerVM, the IBM proprietary hypervisor). 151 152Because all privileged instructions are trapped, guests that use a lot of 153privileged instructions run quite slow with KVM-PR. On the other hand, because 154of that, this kernel module can run on pretty much every PPC hardware, and is 155able to emulate a lot of guests CPUs. This module can even be used to run other 156PowerPC guests like an emulated PowerMac. 157 158As KVM-PR can be run inside a pSeries guest, it can also provide nested 159virtualization capabilities (i.e. running a guest from within a guest). 160 161It is important to notice that, as KVM-HV provides a much better execution 162performance, maintenance work has been much more focused on it in the past 163years. Maintenance for KVM-PR has been minimal. 164 165In order to run KVM-PR guests with POWER9 processors, someone will need to start 166QEMU with ``kernel_irqchip=off`` command line option. 167 168KVM-HV 169^^^^^^ 170 171KVM-HV uses the hypervisor mode of more recent Power processors, that allow 172access to the bare metal hardware directly. Although POWER7 had this capability, 173it was only starting with POWER8 that this was officially supported by IBM. 174 175Originally, KVM-HV was only available when running on a PowerNV platform (a.k.a. 176Power bare metal). Although it runs on a PowerNV platform, it can only be used 177to start pSeries guests. As the pSeries guest doesn't have access to the 178hypervisor mode of the Power CPU, it wasn't possible to run KVM-HV on a guest. 179This limitation has been lifted, and now it is possible to run KVM-HV inside 180pSeries guests as well, making nested virtualization possible with KVM-HV. 181 182As KVM-HV has access to privileged instructions, guests that use a lot of these 183can run much faster than with KVM-PR. On the other hand, the guest CPU has to be 184of the same type as the host CPU this way, e.g. it is not possible to specify an 185embedded PPC CPU for the guest with KVM-HV. However, there is at least the 186possibility to run the guest in a backward-compatibility mode of the previous 187CPUs generations, e.g. you can run a POWER7 guest on a POWER8 host by using 188``-cpu POWER8,compat=power7`` as parameter to QEMU. 189 190Modules support 191--------------- 192 193As noticed in the sections above, each module can run in a different 194environment. The following table shows with which environment each module can 195run. As long as you are in a supported environment, you can run KVM-PR or KVM-HV 196nested. Combinations not shown in the table are not available. 197 198+--------------+------------+------+-------------------+----------+--------+ 199| Platform | Host type | Bits | Page table format | KVM-HV | KVM-PR | 200+==============+============+======+===================+==========+========+ 201| PowerNV | bare metal | 32 | hash | no | yes | 202| | | +-------------------+----------+--------+ 203| | | | radix | N/A | N/A | 204| | +------+-------------------+----------+--------+ 205| | | 64 | hash | yes | yes | 206| | | +-------------------+----------+--------+ 207| | | | radix | yes | no | 208+--------------+------------+------+-------------------+----------+--------+ 209| pSeries [1]_ | PowerNV | 32 | hash | no | yes | 210| | | +-------------------+----------+--------+ 211| | | | radix | N/A | N/A | 212| | +------+-------------------+----------+--------+ 213| | | 64 | hash | no | yes | 214| | | +-------------------+----------+--------+ 215| | | | radix | yes [2]_ | no | 216| +------------+------+-------------------+----------+--------+ 217| | PowerVM | 32 | hash | no | yes | 218| | | +-------------------+----------+--------+ 219| | | | radix | N/A | N/A | 220| | +------+-------------------+----------+--------+ 221| | | 64 | hash | no | yes | 222| | | +-------------------+----------+--------+ 223| | | | radix [3]_ | no | yes | 224+--------------+------------+------+-------------------+----------+--------+ 225 226.. [1] On POWER9 DD2.1 processors, the page table format on the host and guest 227 must be the same. 228 229.. [2] KVM-HV cannot run nested on POWER8 machines. 230 231.. [3] Introduced on Power10 machines. 232 233Maintainer contact information 234------------------------------ 235 236Cédric Le Goater <clg@kaod.org> 237 238Daniel Henrique Barboza <danielhb413@gmail.com> 239