1=============== 2Testing in QEMU 3=============== 4 5This document describes the testing infrastructure in QEMU. 6 7Testing with "make check" 8========================= 9 10The "make check" testing family includes most of the C based tests in QEMU. For 11a quick help, run ``make check-help`` from the source tree. 12 13The usual way to run these tests is: 14 15.. code:: 16 17 make check 18 19which includes QAPI schema tests, unit tests, and QTests. Different sub-types 20of "make check" tests will be explained below. 21 22Before running tests, it is best to build QEMU programs first. Some tests 23expect the executables to exist and will fail with obscure messages if they 24cannot find them. 25 26Unit tests 27---------- 28 29Unit tests, which can be invoked with ``make check-unit``, are simple C tests 30that typically link to individual QEMU object files and exercise them by 31calling exported functions. 32 33If you are writing new code in QEMU, consider adding a unit test, especially 34for utility modules that are relatively stateless or have few dependencies. To 35add a new unit test: 36 371. Create a new source file. For example, ``tests/foo-test.c``. 38 392. Write the test. Normally you would include the header file which exports 40 the module API, then verify the interface behaves as expected from your 41 test. The test code should be organized with the glib testing framework. 42 Copying and modifying an existing test is usually a good idea. 43 443. Add the test to ``tests/Makefile.include``. First, name the unit test 45 program and add it to ``$(check-unit-y)``; then add a rule to build the 46 executable. Optionally, you can add a magical variable to support ``gcov``. 47 For example: 48 49.. code:: 50 51 check-unit-y += tests/foo-test$(EXESUF) 52 tests/foo-test$(EXESUF): tests/foo-test.o $(test-util-obj-y) 53 ... 54 gcov-files-foo-test-y = util/foo.c 55 56Since unit tests don't require environment variables, the simplest way to debug 57a unit test failure is often directly invoking it or even running it under 58``gdb``. However there can still be differences in behavior between ``make`` 59invocations and your manual run, due to ``$MALLOC_PERTURB_`` environment 60variable (which affects memory reclamation and catches invalid pointers better) 61and gtester options. If necessary, you can run 62 63.. code:: 64 make check-unit V=1 65 66and copy the actual command line which executes the unit test, then run 67it from the command line. 68 69QTest 70----- 71 72QTest is a device emulation testing framework. It can be very useful to test 73device models; it could also control certain aspects of QEMU (such as virtual 74clock stepping), with a special purpose "qtest" protocol. Refer to the 75documentation in ``qtest.c`` for more details of the protocol. 76 77QTest cases can be executed with 78 79.. code:: 80 81 make check-qtest 82 83The QTest library is implemented by ``tests/libqtest.c`` and the API is defined 84in ``tests/libqtest.h``. 85 86Consider adding a new QTest case when you are introducing a new virtual 87hardware, or extending one if you are adding functionalities to an existing 88virtual device. 89 90On top of libqtest, a higher level library, ``libqos``, was created to 91encapsulate common tasks of device drivers, such as memory management and 92communicating with system buses or devices. Many virtual device tests use 93libqos instead of directly calling into libqtest. 94 95Steps to add a new QTest case are: 96 971. Create a new source file for the test. (More than one file can be added as 98 necessary.) For example, ``tests/test-foo-device.c``. 99 1002. Write the test code with the glib and libqtest/libqos API. See also existing 101 tests and the library headers for reference. 102 1033. Register the new test in ``tests/Makefile.include``. Add the test executable 104 name to an appropriate ``check-qtest-*-y`` variable. For example: 105 106 ``check-qtest-generic-y = tests/test-foo-device$(EXESUF)`` 107 1084. Add object dependencies of the executable in the Makefile, including the 109 test source file(s) and other interesting objects. For example: 110 111 ``tests/test-foo-device$(EXESUF): tests/test-foo-device.o $(libqos-obj-y)`` 112 113Debugging a QTest failure is slightly harder than the unit test because the 114tests look up QEMU program names in the environment variables, such as 115``QTEST_QEMU_BINARY`` and ``QTEST_QEMU_IMG``, and also because it is not easy 116to attach gdb to the QEMU process spawned from the test. But manual invoking 117and using gdb on the test is still simple to do: find out the actual command 118from the output of 119 120.. code:: 121 make check-qtest V=1 122 123which you can run manually. 124 125QAPI schema tests 126----------------- 127 128The QAPI schema tests validate the QAPI parser used by QMP, by feeding 129predefined input to the parser and comparing the result with the reference 130output. 131 132The input/output data is managed under the ``tests/qapi-schema`` directory. 133Each test case includes four files that have a common base name: 134 135 * ``${casename}.json`` - the file contains the JSON input for feeding the 136 parser 137 * ``${casename}.out`` - the file contains the expected stdout from the parser 138 * ``${casename}.err`` - the file contains the expected stderr from the parser 139 * ``${casename}.exit`` - the expected error code 140 141Consider adding a new QAPI schema test when you are making a change on the QAPI 142parser (either fixing a bug or extending/modifying the syntax). To do this: 143 1441. Add four files for the new case as explained above. For example: 145 146 ``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``. 147 1482. Add the new test in ``tests/Makefile.include``. For example: 149 150 ``qapi-schema += foo.json`` 151 152check-block 153----------- 154 155``make check-block`` is a legacy command to invoke block layer iotests and is 156rarely used. See "QEMU iotests" section below for more information. 157 158GCC gcov support 159---------------- 160 161``gcov`` is a GCC tool to analyze the testing coverage by 162instrumenting the tested code. To use it, configure QEMU with 163``--enable-gcov`` option and build. Then run ``make check`` as usual. 164 165If you want to gather coverage information on a single test the ``make 166clean-coverage`` target can be used to delete any existing coverage 167information before running a single test. 168 169You can generate a HTML coverage report by executing ``make 170coverage-report`` which will create 171./reports/coverage/coverage-report.html. If you want to create it 172elsewhere simply execute ``make /foo/bar/baz/coverage-report.html``. 173 174Further analysis can be conducted by running the ``gcov`` command 175directly on the various .gcda output files. Please read the ``gcov`` 176documentation for more information. 177 178QEMU iotests 179============ 180 181QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing 182framework widely used to test block layer related features. It is higher level 183than "make check" tests and 99% of the code is written in bash or Python 184scripts. The testing success criteria is golden output comparison, and the 185test files are named with numbers. 186 187To run iotests, make sure QEMU is built successfully, then switch to the 188``tests/qemu-iotests`` directory under the build directory, and run ``./check`` 189with desired arguments from there. 190 191By default, "raw" format and "file" protocol is used; all tests will be 192executed, except the unsupported ones. You can override the format and protocol 193with arguments: 194 195.. code:: 196 197 # test with qcow2 format 198 ./check -qcow2 199 # or test a different protocol 200 ./check -nbd 201 202It's also possible to list test numbers explicitly: 203 204.. code:: 205 206 # run selected cases with qcow2 format 207 ./check -qcow2 001 030 153 208 209Cache mode can be selected with the "-c" option, which may help reveal bugs 210that are specific to certain cache mode. 211 212More options are supported by the ``./check`` script, run ``./check -h`` for 213help. 214 215Writing a new test case 216----------------------- 217 218Consider writing a tests case when you are making any changes to the block 219layer. An iotest case is usually the choice for that. There are already many 220test cases, so it is possible that extending one of them may achieve the goal 221and save the boilerplate to create one. (Unfortunately, there isn't a 100% 222reliable way to find a related one out of hundreds of tests. One approach is 223using ``git grep``.) 224 225Usually an iotest case consists of two files. One is an executable that 226produces output to stdout and stderr, the other is the expected reference 227output. They are given the same number in file names. E.g. Test script ``055`` 228and reference output ``055.out``. 229 230In rare cases, when outputs differ between cache mode ``none`` and others, a 231``.out.nocache`` file is added. In other cases, when outputs differ between 232image formats, more than one ``.out`` files are created ending with the 233respective format names, e.g. ``178.out.qcow2`` and ``178.out.raw``. 234 235There isn't a hard rule about how to write a test script, but a new test is 236usually a (copy and) modification of an existing case. There are a few 237commonly used ways to create a test: 238 239* A Bash script. It will make use of several environmental variables related 240 to the testing procedure, and could source a group of ``common.*`` libraries 241 for some common helper routines. 242 243* A Python unittest script. Import ``iotests`` and create a subclass of 244 ``iotests.QMPTestCase``, then call ``iotests.main`` method. The downside of 245 this approach is that the output is too scarce, and the script is considered 246 harder to debug. 247 248* A simple Python script without using unittest module. This could also import 249 ``iotests`` for launching QEMU and utilities etc, but it doesn't inherit 250 from ``iotests.QMPTestCase`` therefore doesn't use the Python unittest 251 execution. This is a combination of 1 and 2. 252 253Pick the language per your preference since both Bash and Python have 254comparable library support for invoking and interacting with QEMU programs. If 255you opt for Python, it is strongly recommended to write Python 3 compatible 256code. 257 258Docker based tests 259================== 260 261Introduction 262------------ 263 264The Docker testing framework in QEMU utilizes public Docker images to build and 265test QEMU in predefined and widely accessible Linux environments. This makes 266it possible to expand the test coverage across distros, toolchain flavors and 267library versions. 268 269Prerequisites 270------------- 271 272Install "docker" with the system package manager and start the Docker service 273on your development machine, then make sure you have the privilege to run 274Docker commands. Typically it means setting up passwordless ``sudo docker`` 275command or login as root. For example: 276 277.. code:: 278 279 $ sudo yum install docker 280 $ # or `apt-get install docker` for Ubuntu, etc. 281 $ sudo systemctl start docker 282 $ sudo docker ps 283 284The last command should print an empty table, to verify the system is ready. 285 286An alternative method to set up permissions is by adding the current user to 287"docker" group and making the docker daemon socket file (by default 288``/var/run/docker.sock``) accessible to the group: 289 290.. code:: 291 292 $ sudo groupadd docker 293 $ sudo usermod $USER -G docker 294 $ sudo chown :docker /var/run/docker.sock 295 296Note that any one of above configurations makes it possible for the user to 297exploit the whole host with Docker bind mounting or other privileged 298operations. So only do it on development machines. 299 300Quickstart 301---------- 302 303From source tree, type ``make docker`` to see the help. Testing can be started 304without configuring or building QEMU (``configure`` and ``make`` are done in 305the container, with parameters defined by the make target): 306 307.. code:: 308 309 make docker-test-build@min-glib 310 311This will create a container instance using the ``min-glib`` image (the image 312is downloaded and initialized automatically), in which the ``test-build`` job 313is executed. 314 315Images 316------ 317 318Along with many other images, the ``min-glib`` image is defined in a Dockerfile 319in ``tests/docker/dockefiles/``, called ``min-glib.docker``. ``make docker`` 320command will list all the available images. 321 322To add a new image, simply create a new ``.docker`` file under the 323``tests/docker/dockerfiles/`` directory. 324 325A ``.pre`` script can be added beside the ``.docker`` file, which will be 326executed before building the image under the build context directory. This is 327mainly used to do necessary host side setup. One such setup is ``binfmt_misc``, 328for example, to make qemu-user powered cross build containers work. 329 330Tests 331----- 332 333Different tests are added to cover various configurations to build and test 334QEMU. Docker tests are the executables under ``tests/docker`` named 335``test-*``. They are typically shell scripts and are built on top of a shell 336library, ``tests/docker/common.rc``, which provides helpers to find the QEMU 337source and build it. 338 339The full list of tests is printed in the ``make docker`` help. 340 341Tools 342----- 343 344There are executables that are created to run in a specific Docker environment. 345This makes it easy to write scripts that have heavy or special dependencies, 346but are still very easy to use. 347 348Currently the only tool is ``travis``, which mimics the Travis-CI tests in a 349container. It runs in the ``travis`` image: 350 351.. code:: 352 353 make docker-travis@travis 354 355Debugging a Docker test failure 356------------------------------- 357 358When CI tasks, maintainers or yourself report a Docker test failure, follow the 359below steps to debug it: 360 3611. Locally reproduce the failure with the reported command line. E.g. run 362 ``make docker-test-mingw@fedora J=8``. 3632. Add "V=1" to the command line, try again, to see the verbose output. 3643. Further add "DEBUG=1" to the command line. This will pause in a shell prompt 365 in the container right before testing starts. You could either manually 366 build QEMU and run tests from there, or press Ctrl-D to let the Docker 367 testing continue. 3684. If you press Ctrl-D, the same building and testing procedure will begin, and 369 will hopefully run into the error again. After that, you will be dropped to 370 the prompt for debug. 371 372Options 373------- 374 375Various options can be used to affect how Docker tests are done. The full 376list is in the ``make docker`` help text. The frequently used ones are: 377 378* ``V=1``: the same as in top level ``make``. It will be propagated to the 379 container and enable verbose output. 380* ``J=$N``: the number of parallel tasks in make commands in the container, 381 similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in 382 top level ``make`` will not be propagated into the container.) 383* ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test 384 failure" section. 385 386VM testing 387========== 388 389This test suite contains scripts that bootstrap various guest images that have 390necessary packages to build QEMU. The basic usage is documented in ``Makefile`` 391help which is displayed with ``make vm-test``. 392 393Quickstart 394---------- 395 396Run ``make vm-test`` to list available make targets. Invoke a specific make 397command to run build test in an image. For example, ``make vm-build-freebsd`` 398will build the source tree in the FreeBSD image. The command can be executed 399from either the source tree or the build dir; if the former, ``./configure`` is 400not needed. The command will then generate the test image in ``./tests/vm/`` 401under the working directory. 402 403Note: images created by the scripts accept a well-known RSA key pair for SSH 404access, so they SHOULD NOT be exposed to external interfaces if you are 405concerned about attackers taking control of the guest and potentially 406exploiting a QEMU security bug to compromise the host. 407 408QEMU binary 409----------- 410 411By default, qemu-system-x86_64 is searched in $PATH to run the guest. If there 412isn't one, or if it is older than 2.10, the test won't work. In this case, 413provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``. 414 415Make jobs 416--------- 417 418The ``-j$X`` option in the make command line is not propagated into the VM, 419specify ``J=$X`` to control the make jobs in the guest. 420 421Debugging 422--------- 423 424Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive 425debugging and verbose output. If this is not enough, see the next section. 426 427Manual invocation 428----------------- 429 430Each guest script is an executable script with the same command line options. 431For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``: 432 433.. code:: 434 435 $ cd $QEMU_SRC/tests/vm 436 437 # To bootstrap the image 438 $ ./netbsd --build-image --image /var/tmp/netbsd.img 439 <...> 440 441 # To run an arbitrary command in guest (the output will not be echoed unless 442 # --debug is added) 443 $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a 444 445 # To build QEMU in guest 446 $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC 447 448 # To get to an interactive shell 449 $ ./netbsd --interactive --image /var/tmp/netbsd.img sh 450 451Adding new guests 452----------------- 453 454Please look at existing guest scripts for how to add new guests. 455 456Most importantly, create a subclass of BaseVM and implement ``build_image()`` 457method and define ``BUILD_SCRIPT``, then finally call ``basevm.main()`` from 458the script's ``main()``. 459 460* Usually in ``build_image()``, a template image is downloaded from a 461 predefined URL. ``BaseVM._download_with_cache()`` takes care of the cache and 462 the checksum, so consider using it. 463 464* Once the image is downloaded, users, SSH server and QEMU build deps should 465 be set up: 466 467 - Root password set to ``BaseVM.ROOT_PASS`` 468 - User ``BaseVM.GUEST_USER`` is created, and password set to 469 ``BaseVM.GUEST_PASS`` 470 - SSH service is enabled and started on boot, 471 ``$QEMU_SRC/tests/keys/id_rsa.pub`` is added to ssh's ``authorized_keys`` 472 file of both root and the normal user 473 - DHCP client service is enabled and started on boot, so that it can 474 automatically configure the virtio-net-pci NIC and communicate with QEMU 475 user net (10.0.2.2) 476 - Necessary packages are installed to untar the source tarball and build 477 QEMU 478 479* Write a proper ``BUILD_SCRIPT`` template, which should be a shell script that 480 untars a raw virtio-blk block device, which is the tarball data blob of the 481 QEMU source tree, then configure/build it. Running "make check" is also 482 recommended. 483 484Image fuzzer testing 485==================== 486 487An image fuzzer was added to exercise format drivers. Currently only qcow2 is 488supported. To start the fuzzer, run 489 490.. code:: 491 492 tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2 493 494Alternatively, some command different from "qemu-img info" can be tested, by 495changing the ``-c`` option. 496 497Acceptance tests using the Avocado Framework 498============================================ 499 500The ``tests/acceptance`` directory hosts functional tests, also known 501as acceptance level tests. They're usually higher level tests, and 502may interact with external resources and with various guest operating 503systems. 504 505These tests are written using the Avocado Testing Framework (which must 506be installed separately) in conjunction with a the ``avocado_qemu.Test`` 507class, implemented at ``tests/acceptance/avocado_qemu``. 508 509Tests based on ``avocado_qemu.Test`` can easily: 510 511 * Customize the command line arguments given to the convenience 512 ``self.vm`` attribute (a QEMUMachine instance) 513 514 * Interact with the QEMU monitor, send QMP commands and check 515 their results 516 517 * Interact with the guest OS, using the convenience console device 518 (which may be useful to assert the effectiveness and correctness of 519 command line arguments or QMP commands) 520 521 * Interact with external data files that accompany the test itself 522 (see ``self.get_data()``) 523 524 * Download (and cache) remote data files, such as firmware and kernel 525 images 526 527 * Have access to a library of guest OS images (by means of the 528 ``avocado.utils.vmimage`` library) 529 530 * Make use of various other test related utilities available at the 531 test class itself and at the utility library: 532 533 - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test 534 - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html 535 536Installation 537------------ 538 539To install Avocado and its dependencies, run: 540 541.. code:: 542 543 pip install --user avocado-framework 544 545Alternatively, follow the instructions on this link: 546 547 http://avocado-framework.readthedocs.io/en/latest/GetStartedGuide.html#installing-avocado 548 549Overview 550-------- 551 552This directory provides the ``avocado_qemu`` Python module, containing 553the ``avocado_qemu.Test`` class. Here's a simple usage example: 554 555.. code:: 556 557 from avocado_qemu import Test 558 559 560 class Version(Test): 561 """ 562 :avocado: enable 563 :avocado: tags=quick 564 """ 565 def test_qmp_human_info_version(self): 566 self.vm.launch() 567 res = self.vm.command('human-monitor-command', 568 command_line='info version') 569 self.assertRegexpMatches(res, r'^(\d+\.\d+\.\d)') 570 571To execute your test, run: 572 573.. code:: 574 575 avocado run version.py 576 577Tests may be classified according to a convention by using docstring 578directives such as ``:avocado: tags=TAG1,TAG2``. To run all tests 579in the current directory, tagged as "quick", run: 580 581.. code:: 582 583 avocado run -t quick . 584 585The ``avocado_qemu.Test`` base test class 586----------------------------------------- 587 588The ``avocado_qemu.Test`` class has a number of characteristics that 589are worth being mentioned right away. 590 591First of all, it attempts to give each test a ready to use QEMUMachine 592instance, available at ``self.vm``. Because many tests will tweak the 593QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``) 594is left to the test writer. 595 596At test "tear down", ``avocado_qemu.Test`` handles the QEMUMachine 597shutdown. 598 599QEMUMachine 600~~~~~~~~~~~ 601 602The QEMUMachine API is already widely used in the Python iotests, 603device-crash-test and other Python scripts. It's a wrapper around the 604execution of a QEMU binary, giving its users: 605 606 * the ability to set command line arguments to be given to the QEMU 607 binary 608 609 * a ready to use QMP connection and interface, which can be used to 610 send commands and inspect its results, as well as asynchronous 611 events 612 613 * convenience methods to set commonly used command line arguments in 614 a more succinct and intuitive way 615 616QEMU binary selection 617~~~~~~~~~~~~~~~~~~~~~ 618 619The QEMU binary used for the ``self.vm`` QEMUMachine instance will 620primarily depend on the value of the ``qemu_bin`` parameter. If it's 621not explicitly set, its default value will be the result of a dynamic 622probe in the same source tree. A suitable binary will be one that 623targets the architecture matching host machine. 624 625Based on this description, test writers will usually rely on one of 626the following approaches: 627 6281) Set ``qemu_bin``, and use the given binary 629 6302) Do not set ``qemu_bin``, and use a QEMU binary named like 631 "${arch}-softmmu/qemu-system-${arch}", either in the current 632 working directory, or in the current source tree. 633 634The resulting ``qemu_bin`` value will be preserved in the 635``avocado_qemu.Test`` as an attribute with the same name. 636 637Attribute reference 638------------------- 639 640Besides the attributes and methods that are part of the base 641``avocado.Test`` class, the following attributes are available on any 642``avocado_qemu.Test`` instance. 643 644vm 645~~ 646 647A QEMUMachine instance, initially configured according to the given 648``qemu_bin`` parameter. 649 650qemu_bin 651~~~~~~~~ 652 653The preserved value of the ``qemu_bin`` parameter or the result of the 654dynamic probe for a QEMU binary in the current working directory or 655source tree. 656 657Parameter reference 658------------------- 659 660To understand how Avocado parameters are accessed by tests, and how 661they can be passed to tests, please refer to:: 662 663 http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#accessing-test-parameters 664 665Parameter values can be easily seen in the log files, and will look 666like the following: 667 668.. code:: 669 670 PARAMS (key=qemu_bin, path=*, default=x86_64-softmmu/qemu-system-x86_64) => 'x86_64-softmmu/qemu-system-x86_64 671 672qemu_bin 673~~~~~~~~ 674 675The exact QEMU binary to be used on QEMUMachine. 676 677Uninstalling Avocado 678-------------------- 679 680If you've followed the installation instructions above, you can easily 681uninstall Avocado. Start by listing the packages you have installed:: 682 683 pip list --user 684 685And remove any package you want with:: 686 687 pip uninstall <package_name> 688