1Adjunct Processor (AP) Device
2=============================
3
4.. contents::
5
6Introduction
7------------
8
9The IBM Adjunct Processor (AP) Cryptographic Facility is comprised
10of three AP instructions and from 1 to 256 PCIe cryptographic adapter cards.
11These AP devices provide cryptographic functions to all CPUs assigned to a
12linux system running in an IBM Z system LPAR.
13
14On s390x, AP adapter cards are exposed via the AP bus. This document
15describes how those cards may be made available to KVM guests using the
16VFIO mediated device framework.
17
18AP Architectural Overview
19-------------------------
20
21In order understand the terminology used in the rest of this document, let's
22start with some definitions:
23
24* AP adapter
25
26  An AP adapter is an IBM Z adapter card that can perform cryptographic
27  functions. There can be from 0 to 256 adapters assigned to an LPAR depending
28  on the machine model. Adapters assigned to the LPAR in which a linux host is
29  running will be available to the linux host. Each adapter is identified by a
30  number from 0 to 255; however, the maximum adapter number allowed is
31  determined by machine model. When installed, an AP adapter is accessed by
32  AP instructions executed by any CPU.
33
34* AP domain
35
36  An adapter is partitioned into domains. Each domain can be thought of as
37  a set of hardware registers for processing AP instructions. An adapter can
38  hold up to 256 domains; however, the maximum domain number allowed is
39  determined by machine model. Each domain is identified by a number from 0 to
40  255. Domains can be further classified into two types:
41
42    * Usage domains are domains that can be accessed directly to process AP
43      commands
44
45    * Control domains are domains that are accessed indirectly by AP
46      commands sent to a usage domain to control or change the domain; for
47      example, to set a secure private key for the domain.
48
49* AP Queue
50
51  An AP queue is the means by which an AP command-request message is sent to an
52  AP usage domain inside a specific AP. An AP queue is identified by a tuple
53  comprised of an AP adapter ID (APID) and an AP queue index (APQI). The
54  APQI corresponds to a given usage domain number within the adapter. This tuple
55  forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP
56  instructions include a field containing the APQN to identify the AP queue to
57  which the AP command-request message is to be sent for processing.
58
59* AP Instructions:
60
61  There are three AP instructions:
62
63  * NQAP: to enqueue an AP command-request message to a queue
64  * DQAP: to dequeue an AP command-reply message from a queue
65  * PQAP: to administer the queues
66
67  AP instructions identify the domain that is targeted to process the AP
68  command; this must be one of the usage domains. An AP command may modify a
69  domain that is not one of the usage domains, but the modified domain
70  must be one of the control domains.
71
72Start Interpretive Execution (SIE) Instruction
73----------------------------------------------
74
75A KVM guest is started by executing the Start Interpretive Execution (SIE)
76instruction. The SIE state description is a control block that contains the
77state information for a KVM guest and is supplied as input to the SIE
78instruction. The SIE state description contains a satellite control block called
79the Crypto Control Block (CRYCB). The CRYCB contains three fields to identify
80the adapters, usage domains and control domains assigned to the KVM guest:
81
82* The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned
83  to the KVM guest. Each bit in the mask, from left to right, corresponds to
84  an APID from 0-255. If a bit is set, the corresponding adapter is valid for
85  use by the KVM guest.
86
87* The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains
88  assigned to the KVM guest. Each bit in the mask, from left to right,
89  corresponds to  an AP queue index (APQI) from 0-255. If a bit is set, the
90  corresponding queue is valid for use by the KVM guest.
91
92* The AP Domain Mask field is a bit mask that identifies the AP control domains
93  assigned to the KVM guest. The ADM bit mask controls which domains can be
94  changed by an AP command-request message sent to a usage domain from the
95  guest. Each bit in the mask, from left to right, corresponds to a domain from
96  0-255. If a bit is set, the corresponding domain can be modified by an AP
97  command-request message sent to a usage domain.
98
99If you recall from the description of an AP Queue, AP instructions include
100an APQN to identify the AP adapter and AP queue to which an AP command-request
101message is to be sent (NQAP and PQAP instructions), or from which a
102command-reply message is to be received (DQAP instruction). The validity of an
103APQN is defined by the matrix calculated from the APM and AQM; it is the
104cross product of all assigned adapter numbers (APM) with all assigned queue
105indexes (AQM). For example, if adapters 1 and 2 and usage domains 5 and 6 are
106assigned to a guest, the APQNs (1,5), (1,6), (2,5) and (2,6) will be valid for
107the guest.
108
109The APQNs can provide secure key functionality - i.e., a private key is stored
110on the adapter card for each of its domains - so each APQN must be assigned to
111at most one guest or the linux host.
112
113Example 1: Valid configuration
114~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
115
116+----------+--------+--------+
117|          | Guest1 | Guest2 |
118+==========+========+========+
119| adapters |  1, 2  |  1, 2  |
120+----------+--------+--------+
121| domains  |  5, 6  |  7     |
122+----------+--------+--------+
123
124This is valid because both guests have a unique set of APQNs:
125
126* Guest1 has APQNs (1,5), (1,6), (2,5) and (2,6);
127* Guest2 has APQNs (1,7) and (2,7).
128
129Example 2: Valid configuration
130~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
131
132+----------+--------+--------+
133|          | Guest1 | Guest2 |
134+==========+========+========+
135| adapters |  1, 2  |  3, 4  |
136+----------+--------+--------+
137| domains  |  5, 6  |  5, 6  |
138+----------+--------+--------+
139
140This is also valid because both guests have a unique set of APQNs:
141
142* Guest1 has APQNs (1,5), (1,6), (2,5), (2,6);
143* Guest2 has APQNs (3,5), (3,6), (4,5), (4,6)
144
145Example 3: Invalid configuration
146~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
147
148+----------+--------+--------+
149|          | Guest1 | Guest2 |
150+==========+========+========+
151| adapters |  1, 2  |  1     |
152+----------+--------+--------+
153| domains  |  5, 6  |  6, 7  |
154+----------+--------+--------+
155
156This is an invalid configuration because both guests have access to
157APQN (1,6).
158
159AP Matrix Configuration on Linux Host
160-------------------------------------
161
162A linux system is a guest of the LPAR in which it is running and has access to
163the AP resources configured for the LPAR. The LPAR's AP matrix is
164configured via its Activation Profile which can be edited on the HMC. When the
165linux system is started, the AP bus will detect the AP devices assigned to the
166LPAR and create the following in sysfs::
167
168  /sys/bus/ap
169  ... [devices]
170  ...... xx.yyyy
171  ...... ...
172  ...... cardxx
173  ...... ...
174
175Where:
176
177``cardxx``
178  is AP adapter number xx (in hex)
179
180``xx.yyyy``
181  is an APQN with xx specifying the APID and yyyy specifying the APQI
182
183For example, if AP adapters 5 and 6 and domains 4, 71 (0x47), 171 (0xab) and
184255 (0xff) are configured for the LPAR, the sysfs representation on the linux
185host system would look like this::
186
187  /sys/bus/ap
188  ... [devices]
189  ...... 05.0004
190  ...... 05.0047
191  ...... 05.00ab
192  ...... 05.00ff
193  ...... 06.0004
194  ...... 06.0047
195  ...... 06.00ab
196  ...... 06.00ff
197  ...... card05
198  ...... card06
199
200A set of default device drivers are also created to control each type of AP
201device that can be assigned to the LPAR on which a linux host is running::
202
203  /sys/bus/ap
204  ... [drivers]
205  ...... [cex2acard]        for Crypto Express 2/3 accelerator cards
206  ...... [cex2aqueue]       for AP queues served by Crypto Express 2/3
207                            accelerator cards
208  ...... [cex4card]         for Crypto Express 4/5/6 accelerator and coprocessor
209                            cards
210  ...... [cex4queue]        for AP queues served by Crypto Express 4/5/6
211                            accelerator and coprocessor cards
212  ...... [pcixcccard]       for Crypto Express 2/3 coprocessor cards
213  ...... [pcixccqueue]      for AP queues served by Crypto Express 2/3
214                            coprocessor cards
215
216Binding AP devices to device drivers
217~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
218
219There are two sysfs files that specify bitmasks marking a subset of the APQN
220range as 'usable by the default AP queue device drivers' or 'not usable by the
221default device drivers' and thus available for use by the alternate device
222driver(s). The sysfs locations of the masks are::
223
224   /sys/bus/ap/apmask
225   /sys/bus/ap/aqmask
226
227The ``apmask`` is a 256-bit mask that identifies a set of AP adapter IDs
228(APID). Each bit in the mask, from left to right (i.e., from most significant
229to least significant bit in big endian order), corresponds to an APID from
2300-255. If a bit is set, the APID is marked as usable only by the default AP
231queue device drivers; otherwise, the APID is usable by the vfio_ap
232device driver.
233
234The ``aqmask`` is a 256-bit mask that identifies a set of AP queue indexes
235(APQI). Each bit in the mask, from left to right (i.e., from most significant
236to least significant bit in big endian order), corresponds to an APQI from
2370-255. If a bit is set, the APQI is marked as usable only by the default AP
238queue device drivers; otherwise, the APQI is usable by the vfio_ap device
239driver.
240
241Take, for example, the following mask::
242
243      0x7dffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
244
245It indicates:
246
247      1, 2, 3, 4, 5, and 7-255 belong to the default drivers' pool, and 0 and 6
248      belong to the vfio_ap device driver's pool.
249
250The APQN of each AP queue device assigned to the linux host is checked by the
251AP bus against the set of APQNs derived from the cross product of APIDs
252and APQIs marked as usable only by the default AP queue device drivers. If a
253match is detected,  only the default AP queue device drivers will be probed;
254otherwise, the vfio_ap device driver will be probed.
255
256By default, the two masks are set to reserve all APQNs for use by the default
257AP queue device drivers. There are two ways the default masks can be changed:
258
259 1. The sysfs mask files can be edited by echoing a string into the
260    respective sysfs mask file in one of two formats:
261
262    * An absolute hex string starting with 0x - like "0x12345678" - sets
263      the mask. If the given string is shorter than the mask, it is padded
264      with 0s on the right; for example, specifying a mask value of 0x41 is
265      the same as specifying::
266
267           0x4100000000000000000000000000000000000000000000000000000000000000
268
269      Keep in mind that the mask reads from left to right (i.e., most
270      significant to least significant bit in big endian order), so the mask
271      above identifies device numbers 1 and 7 (``01000001``).
272
273      If the string is longer than the mask, the operation is terminated with
274      an error (EINVAL).
275
276    * Individual bits in the mask can be switched on and off by specifying
277      each bit number to be switched in a comma separated list. Each bit
278      number string must be prepended with a (``+``) or minus (``-``) to indicate
279      the corresponding bit is to be switched on (``+``) or off (``-``). Some
280      valid values are::
281
282           "+0"    switches bit 0 on
283           "-13"   switches bit 13 off
284           "+0x41" switches bit 65 on
285           "-0xff" switches bit 255 off
286
287      The following example::
288
289              +0,-6,+0x47,-0xf0
290
291      Switches bits 0 and 71 (0x47) on
292      Switches bits 6 and 240 (0xf0) off
293
294      Note that the bits not specified in the list remain as they were before
295      the operation.
296
297 2. The masks can also be changed at boot time via parameters on the kernel
298    command line like this::
299
300         ap.apmask=0xffff ap.aqmask=0x40
301
302    This would create the following masks:
303
304    apmask::
305
306            0xffff000000000000000000000000000000000000000000000000000000000000
307
308    aqmask::
309
310            0x4000000000000000000000000000000000000000000000000000000000000000
311
312    Resulting in these two pools::
313
314            default drivers pool:    adapter 0-15, domain 1
315            alternate drivers pool:  adapter 16-255, domains 0, 2-255
316
317Configuring an AP matrix for a linux guest
318~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
319
320The sysfs interfaces for configuring an AP matrix for a guest are built on the
321VFIO mediated device framework. To configure an AP matrix for a guest, a
322mediated matrix device must first be created for the ``/sys/devices/vfio_ap/matrix``
323device. When the vfio_ap device driver is loaded, it registers with the VFIO
324mediated device framework. When the driver registers, the sysfs interfaces for
325creating mediated matrix devices is created::
326
327  /sys/devices
328  ... [vfio_ap]
329  ......[matrix]
330  ......... [mdev_supported_types]
331  ............ [vfio_ap-passthrough]
332  ............... create
333  ............... [devices]
334
335A mediated AP matrix device is created by writing a UUID to the attribute file
336named ``create``, for example::
337
338   uuidgen > create
339
340or
341
342::
343
344   echo $uuid > create
345
346When a mediated AP matrix device is created, a sysfs directory named after
347the UUID is created in the ``devices`` subdirectory::
348
349  /sys/devices
350  ... [vfio_ap]
351  ......[matrix]
352  ......... [mdev_supported_types]
353  ............ [vfio_ap-passthrough]
354  ............... create
355  ............... [devices]
356  .................. [$uuid]
357
358There will also be three sets of attribute files created in the mediated
359matrix device's sysfs directory to configure an AP matrix for the
360KVM guest::
361
362  /sys/devices
363  ... [vfio_ap]
364  ......[matrix]
365  ......... [mdev_supported_types]
366  ............ [vfio_ap-passthrough]
367  ............... create
368  ............... [devices]
369  .................. [$uuid]
370  ..................... assign_adapter
371  ..................... assign_control_domain
372  ..................... assign_domain
373  ..................... matrix
374  ..................... unassign_adapter
375  ..................... unassign_control_domain
376  ..................... unassign_domain
377
378``assign_adapter``
379   To assign an AP adapter to the mediated matrix device, its APID is written
380   to the ``assign_adapter`` file. This may be done multiple times to assign more
381   than one adapter. The APID may be specified using conventional semantics
382   as a decimal, hexadecimal, or octal number. For example, to assign adapters
383   4, 5 and 16 to a mediated matrix device in decimal, hexadecimal and octal
384   respectively::
385
386       echo 4 > assign_adapter
387       echo 0x5 > assign_adapter
388       echo 020 > assign_adapter
389
390   In order to successfully assign an adapter:
391
392   * The adapter number specified must represent a value from 0 up to the
393     maximum adapter number allowed by the machine model. If an adapter number
394     higher than the maximum is specified, the operation will terminate with
395     an error (ENODEV).
396
397   * All APQNs that can be derived from the adapter ID being assigned and the
398     IDs of the previously assigned domains must be bound to the vfio_ap device
399     driver. If no domains have yet been assigned, then there must be at least
400     one APQN with the specified APID bound to the vfio_ap driver. If no such
401     APQNs are bound to the driver, the operation will terminate with an
402     error (EADDRNOTAVAIL).
403
404   * No APQN that can be derived from the adapter ID and the IDs of the
405     previously assigned domains can be assigned to another mediated matrix
406     device. If an APQN is assigned to another mediated matrix device, the
407     operation will terminate with an error (EADDRINUSE).
408
409``unassign_adapter``
410   To unassign an AP adapter, its APID is written to the ``unassign_adapter``
411   file. This may also be done multiple times to unassign more than one adapter.
412
413``assign_domain``
414   To assign a usage domain, the domain number is written into the
415   ``assign_domain`` file. This may be done multiple times to assign more than one
416   usage domain. The domain number is specified using conventional semantics as
417   a decimal, hexadecimal, or octal number. For example, to assign usage domains
418   4, 8, and 71 to a mediated matrix device in decimal, hexadecimal and octal
419   respectively::
420
421      echo 4 > assign_domain
422      echo 0x8 > assign_domain
423      echo 0107 > assign_domain
424
425   In order to successfully assign a domain:
426
427   * The domain number specified must represent a value from 0 up to the
428     maximum domain number allowed by the machine model. If a domain number
429     higher than the maximum is specified, the operation will terminate with
430     an error (ENODEV).
431
432   * All APQNs that can be derived from the domain ID being assigned and the IDs
433     of the previously assigned adapters must be bound to the vfio_ap device
434     driver. If no domains have yet been assigned, then there must be at least
435     one APQN with the specified APQI bound to the vfio_ap driver. If no such
436     APQNs are bound to the driver, the operation will terminate with an
437     error (EADDRNOTAVAIL).
438
439   * No APQN that can be derived from the domain ID being assigned and the IDs
440     of the previously assigned adapters can be assigned to another mediated
441     matrix device. If an APQN is assigned to another mediated matrix device,
442     the operation will terminate with an error (EADDRINUSE).
443
444``unassign_domain``
445   To unassign a usage domain, the domain number is written into the
446   ``unassign_domain`` file. This may be done multiple times to unassign more than
447   one usage domain.
448
449``assign_control_domain``
450   To assign a control domain, the domain number is written into the
451   ``assign_control_domain`` file. This may be done multiple times to
452   assign more than one control domain. The domain number may be specified using
453   conventional semantics as a decimal, hexadecimal, or octal number. For
454   example, to assign  control domains 4, 8, and 71 to  a mediated matrix device
455   in decimal, hexadecimal and octal respectively::
456
457      echo 4 > assign_domain
458      echo 0x8 > assign_domain
459      echo 0107 > assign_domain
460
461   In order to successfully assign a control domain, the domain number
462   specified must represent a value from 0 up to the maximum domain number
463   allowed by the machine model. If a control domain number higher than the
464   maximum is specified, the operation will terminate with an error (ENODEV).
465
466``unassign_control_domain``
467   To unassign a control domain, the domain number is written into the
468   ``unassign_domain`` file. This may be done multiple times to unassign more than
469   one control domain.
470
471Notes: No changes to the AP matrix will be allowed while a guest using
472the mediated matrix device is running. Attempts to assign an adapter,
473domain or control domain will be rejected and an error (EBUSY) returned.
474
475Starting a Linux Guest Configured with an AP Matrix
476~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
477
478To provide a mediated matrix device for use by a guest, the following option
479must be specified on the QEMU command line::
480
481   -device vfio_ap,sysfsdev=$path-to-mdev
482
483The sysfsdev parameter specifies the path to the mediated matrix device.
484There are a number of ways to specify this path::
485
486  /sys/devices/vfio_ap/matrix/$uuid
487  /sys/bus/mdev/devices/$uuid
488  /sys/bus/mdev/drivers/vfio_mdev/$uuid
489  /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/devices/$uuid
490
491When the linux guest is started, the guest will open the mediated
492matrix device's file descriptor to get information about the mediated matrix
493device. The ``vfio_ap`` device driver will update the APM, AQM, and ADM fields in
494the guest's CRYCB with the adapter, usage domain and control domains assigned
495via the mediated matrix device's sysfs attribute files. Programs running on the
496linux guest will then:
497
4981. Have direct access to the APQNs derived from the cross product of the AP
499   adapter numbers (APID) and queue indexes (APQI) specified in the APM and AQM
500   fields of the guests's CRYCB respectively. These APQNs identify the AP queues
501   that are valid for use by the guest; meaning, AP commands can be sent by the
502   guest to any of these queues for processing.
503
5042. Have authorization to process AP commands to change a control domain
505   identified in the ADM field of the guest's CRYCB. The AP command must be sent
506   to a valid APQN (see 1 above).
507
508CPU model features:
509
510Three CPU model features are available for controlling guest access to AP
511facilities:
512
5131. AP facilities feature
514
515   The AP facilities feature indicates that AP facilities are installed on the
516   guest. This feature will be exposed for use only if the AP facilities
517   are installed on the host system. The feature is s390-specific and is
518   represented as a parameter of the -cpu option on the QEMU command line::
519
520      qemu-system-s390x -cpu $model,ap=on|off
521
522   Where:
523
524      ``$model``
525        is the CPU model defined for the guest (defaults to the model of
526        the host system if not specified).
527
528      ``ap=on|off``
529        indicates whether AP facilities are installed (on) or not
530        (off). The default for CPU models zEC12 or newer
531        is ``ap=on``. AP facilities must be installed on the guest if a
532        vfio-ap device (``-device vfio-ap,sysfsdev=$path``) is configured
533        for the guest, or the guest will fail to start.
534
5352. Query Configuration Information (QCI) facility
536
537   The QCI facility is used by the AP bus running on the guest to query the
538   configuration of the AP facilities. This facility will be available
539   only if the QCI facility is installed on the host system. The feature is
540   s390-specific and is represented as a parameter of the -cpu option on the
541   QEMU command line::
542
543      qemu-system-s390x -cpu $model,apqci=on|off
544
545   Where:
546
547      ``$model``
548        is the CPU model defined for the guest
549
550      ``apqci=on|off``
551        indicates whether the QCI facility is installed (on) or
552        not (off). The default for CPU models zEC12 or newer
553        is ``apqci=on``; for older models, QCI will not be installed.
554
555        If QCI is installed (``apqci=on``) but AP facilities are not
556        (``ap=off``), an error message will be logged, but the guest
557        will be allowed to start. It makes no sense to have QCI
558        installed if the AP facilities are not; this is considered
559        an invalid configuration.
560
561        If the QCI facility is not installed, APQNs with an APQI
562        greater than 15 will not be detected by the AP bus
563        running on the guest.
564
5653. Adjunct Process Facility Test (APFT) facility
566
567   The APFT facility is used by the AP bus running on the guest to test the
568   AP facilities available for a given AP queue. This facility will be available
569   only if the APFT facility is installed on the host system. The feature is
570   s390-specific and is represented as a parameter of the -cpu option on the
571   QEMU command line::
572
573      qemu-system-s390x -cpu $model,apft=on|off
574
575   Where:
576
577      ``$model``
578        is the CPU model defined for the guest (defaults to the model of
579        the host system if not specified).
580
581      ``apft=on|off``
582        indicates whether the APFT facility is installed (on) or
583        not (off). The default for CPU models zEC12 and
584        newer is ``apft=on`` for older models, APFT will not be
585        installed.
586
587        If APFT is installed (``apft=on``) but AP facilities are not
588        (``ap=off``), an error message will be logged, but the guest
589        will be allowed to start. It makes no sense to have APFT
590        installed if the AP facilities are not; this is considered
591        an invalid configuration.
592
593        It also makes no sense to turn APFT off because the AP bus
594        running on the guest will not detect CEX4 and newer devices
595        without it. Since only CEX4 and newer devices are supported
596        for guest usage, no AP devices can be made accessible to a
597        guest started without APFT installed.
598
599Hot plug a vfio-ap device into a running guest
600~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
601
602Only one vfio-ap device can be attached to the virtual machine's ap-bus, so a
603vfio-ap device can be hot plugged if and only if no vfio-ap device is attached
604to the bus already, whether via the QEMU command line or a prior hot plug
605action.
606
607To hot plug a vfio-ap device, use the QEMU ``device_add`` command::
608
609    (qemu) device_add vfio-ap,sysfsdev="$path-to-mdev"
610
611Where the ``$path-to-mdev`` value specifies the absolute path to a mediated
612device to which AP resources to be used by the guest have been assigned.
613
614Note that on Linux guests, the AP devices will be created in the
615``/sys/bus/ap/devices`` directory when the AP bus subsequently performs its periodic
616scan, so there may be a short delay before the AP devices are accessible on the
617guest.
618
619The command will fail if:
620
621* A vfio-ap device has already been attached to the virtual machine's ap-bus.
622
623* The CPU model features for controlling guest access to AP facilities are not
624  enabled (see 'CPU model features' subsection in the previous section).
625
626Hot unplug a vfio-ap device from a running guest
627~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
628
629A vfio-ap device can be unplugged from a running KVM guest if a vfio-ap device
630has been attached to the virtual machine's ap-bus via the QEMU command line
631or a prior hot plug action.
632
633To hot unplug a vfio-ap device, use the QEMU ``device_del`` command::
634
635    (qemu) device_del vfio-ap,sysfsdev="$path-to-mdev"
636
637Where ``$path-to-mdev`` is the same as the path specified when the vfio-ap
638device was attached to the virtual machine's ap-bus.
639
640On a Linux guest, the AP devices will be removed from the ``/sys/bus/ap/devices``
641directory on the guest when the AP bus subsequently performs its periodic scan,
642so there may be a short delay before the AP devices are no longer accessible by
643the guest.
644
645The command will fail if the ``$path-to-mdev`` specified on the ``device_del`` command
646does not match the value specified when the vfio-ap device was attached to
647the virtual machine's ap-bus.
648
649Example: Configure AP Matrices for Three Linux Guests
650-----------------------------------------------------
651
652Let's now provide an example to illustrate how KVM guests may be given
653access to AP facilities. For this example, we will show how to configure
654three guests such that executing the lszcrypt command on the guests would
655look like this:
656
657Guest1::
658
659  CARD.DOMAIN TYPE  MODE
660  ------------------------------
661  05          CEX5C CCA-Coproc
662  05.0004     CEX5C CCA-Coproc
663  05.00ab     CEX5C CCA-Coproc
664  06          CEX5A Accelerator
665  06.0004     CEX5A Accelerator
666  06.00ab     CEX5C CCA-Coproc
667
668Guest2::
669
670  CARD.DOMAIN TYPE  MODE
671  ------------------------------
672  05          CEX5A Accelerator
673  05.0047     CEX5A Accelerator
674  05.00ff     CEX5A Accelerator
675
676Guest3::
677
678  CARD.DOMAIN TYPE  MODE
679  ------------------------------
680  06          CEX5A Accelerator
681  06.0047     CEX5A Accelerator
682  06.00ff     CEX5A Accelerator
683
684These are the steps:
685
6861. Install the vfio_ap module on the linux host. The dependency chain for the
687   vfio_ap module is:
688
689   * iommu
690   * s390
691   * zcrypt
692   * vfio
693   * vfio_mdev
694   * vfio_mdev_device
695   * KVM
696
697   To build the vfio_ap module, the kernel build must be configured with the
698   following Kconfig elements selected:
699
700   * IOMMU_SUPPORT
701   * S390
702   * ZCRYPT
703   * S390_AP_IOMMU
704   * VFIO
705   * VFIO_MDEV
706   * VFIO_MDEV_DEVICE
707   * KVM
708
709   If using make menuconfig select the following to build the vfio_ap module::
710     -> Device Drivers
711        -> IOMMU Hardware Support
712           select S390 AP IOMMU Support
713        -> VFIO Non-Privileged userspace driver framework
714           -> Mediated device driver framework
715              -> VFIO driver for Mediated devices
716     -> I/O subsystem
717        -> VFIO support for AP devices
718
7192. Secure the AP queues to be used by the three guests so that the host can not
720   access them. To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff,
721   06.0004, 06.0047, 06.00ab, and 06.00ff for use by the vfio_ap device driver,
722   the corresponding APQNs must be removed from the default queue drivers pool
723   as follows::
724
725      echo -5,-6 > /sys/bus/ap/apmask
726
727      echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask
728
729   This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004,
730   06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The
731   sysfs directory for the vfio_ap device driver will now contain symbolic links
732   to the AP queue devices bound to it::
733
734     /sys/bus/ap
735     ... [drivers]
736     ...... [vfio_ap]
737     ......... [05.0004]
738     ......... [05.0047]
739     ......... [05.00ab]
740     ......... [05.00ff]
741     ......... [06.0004]
742     ......... [06.0047]
743     ......... [06.00ab]
744     ......... [06.00ff]
745
746   Keep in mind that only type 10 and newer adapters (i.e., CEX4 and later)
747   can be bound to the vfio_ap device driver. The reason for this is to
748   simplify the implementation by not needlessly complicating the design by
749   supporting older devices that will go out of service in the relatively near
750   future, and for which there are few older systems on which to test.
751
752   The administrator, therefore, must take care to secure only AP queues that
753   can be bound to the vfio_ap device driver. The device type for a given AP
754   queue device can be read from the parent card's sysfs directory. For example,
755   to see the hardware type of the queue 05.0004::
756
757     cat /sys/bus/ap/devices/card05/hwtype
758
759   The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the
760   vfio_ap device driver.
761
7623. Create the mediated devices needed to configure the AP matrixes for the
763   three guests and to provide an interface to the vfio_ap driver for
764   use by the guests::
765
766     /sys/devices/vfio_ap/matrix/
767     ... [mdev_supported_types]
768     ...... [vfio_ap-passthrough] (passthrough mediated matrix device type)
769     ......... create
770     ......... [devices]
771
772   To create the mediated devices for the three guests::
773
774       uuidgen > create
775       uuidgen > create
776       uuidgen > create
777
778   or
779
780   ::
781
782       echo $uuid1 > create
783       echo $uuid2 > create
784       echo $uuid3 > create
785
786   This will create three mediated devices in the [devices] subdirectory named
787   after the UUID used to create the mediated device. We'll call them $uuid1,
788   $uuid2 and $uuid3 and this is the sysfs directory structure after creation::
789
790     /sys/devices/vfio_ap/matrix/
791     ... [mdev_supported_types]
792     ...... [vfio_ap-passthrough]
793     ......... [devices]
794     ............ [$uuid1]
795     ............... assign_adapter
796     ............... assign_control_domain
797     ............... assign_domain
798     ............... matrix
799     ............... unassign_adapter
800     ............... unassign_control_domain
801     ............... unassign_domain
802
803     ............ [$uuid2]
804     ............... assign_adapter
805     ............... assign_control_domain
806     ............... assign_domain
807     ............... matrix
808     ............... unassign_adapter
809     ............... unassign_control_domain
810     ............... unassign_domain
811
812     ............ [$uuid3]
813     ............... assign_adapter
814     ............... assign_control_domain
815     ............... assign_domain
816     ............... matrix
817     ............... unassign_adapter
818     ............... unassign_control_domain
819     ............... unassign_domain
820
8214. The administrator now needs to configure the matrixes for the mediated
822   devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3).
823
824   This is how the matrix is configured for Guest1::
825
826      echo 5 > assign_adapter
827      echo 6 > assign_adapter
828      echo 4 > assign_domain
829      echo 0xab > assign_domain
830
831   Control domains can similarly be assigned using the assign_control_domain
832   sysfs file.
833
834   If a mistake is made configuring an adapter, domain or control domain,
835   you can use the ``unassign_xxx`` interfaces to unassign the adapter, domain or
836   control domain.
837
838   To display the matrix configuration for Guest1::
839
840         cat matrix
841
842   The output will display the APQNs in the format ``xx.yyyy``, where xx is
843   the adapter number and yyyy is the domain number. The output for Guest1
844   will look like this::
845
846         05.0004
847         05.00ab
848         06.0004
849         06.00ab
850
851   This is how the matrix is configured for Guest2::
852
853      echo 5 > assign_adapter
854      echo 0x47 > assign_domain
855      echo 0xff > assign_domain
856
857   This is how the matrix is configured for Guest3::
858
859      echo 6 > assign_adapter
860      echo 0x47 > assign_domain
861      echo 0xff > assign_domain
862
8635. Start Guest1::
864
865   /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...
866
8677. Start Guest2::
868
869   /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...
870
8717. Start Guest3::
872
873   /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid3 ...
874
875When the guest is shut down, the mediated matrix devices may be removed.
876
877Using our example again, to remove the mediated matrix device $uuid1::
878
879   /sys/devices/vfio_ap/matrix/
880   ... [mdev_supported_types]
881   ...... [vfio_ap-passthrough]
882   ......... [devices]
883   ............ [$uuid1]
884   ............... remove
885
886
887   echo 1 > remove
888
889This will remove all of the mdev matrix device's sysfs structures including
890the mdev device itself. To recreate and reconfigure the mdev matrix device,
891all of the steps starting with step 3 will have to be performed again. Note
892that the remove will fail if a guest using the mdev is still running.
893
894It is not necessary to remove an mdev matrix device, but one may want to
895remove it if no guest will use it during the remaining lifetime of the linux
896host. If the mdev matrix device is removed, one may want to also reconfigure
897the pool of adapters and queues reserved for use by the default drivers.
898
899Limitations
900-----------
901
902* The KVM/kernel interfaces do not provide a way to prevent restoring an APQN
903  to the default drivers pool of a queue that is still assigned to a mediated
904  device in use by a guest. It is incumbent upon the administrator to
905  ensure there is no mediated device in use by a guest to which the APQN is
906  assigned lest the host be given access to the private data of the AP queue
907  device, such as a private key configured specifically for the guest.
908
909* Dynamically assigning AP resources to or unassigning AP resources from a
910  mediated matrix device - see `Configuring an AP matrix for a linux guest`_
911  section above - while a running guest is using it is currently not supported.
912
913* Live guest migration is not supported for guests using AP devices. If a guest
914  is using AP devices, the vfio-ap device configured for the guest must be
915  unplugged before migrating the guest (see `Hot unplug a vfio-ap device from a
916  running guest`_ section above.)
917