1integration-aliases 2=================== 3 4Integration tests are executed by ``ansible-test`` and reside in directories under ``test/integration/targets/``. 5Each test MUST have an ``aliases`` file to control test execution. 6 7Aliases are explained in the following sections. Each alias must be on a separate line in an ``aliases`` file. 8 9Groups 10------ 11 12Tests must be configured to run in exactly one group. This is done by adding the appropriate group to the ``aliases`` file. 13 14The following are examples of some of the available groups: 15 16- ``shippable/posix/group1`` 17- ``shippable/windows/group2`` 18- ``shippable/azure/group3`` 19- ``shippable/aws/group1`` 20- ``shippable/cloud/group1`` 21 22Groups are used to balance tests across multiple CI jobs to minimize test run time. 23They also improve efficiency by keeping tests with similar requirements running together. 24 25When selecting a group for a new test, use the same group as existing tests similar to the one being added. 26If more than one group is available, select one randomly. 27 28Setup 29----- 30 31Aliases can be used to execute setup targets before running tests: 32 33- ``setup/once/TARGET`` - Run the target ``TARGET`` before the first target that requires it. 34- ``setup/always/TARGET`` - Run the target ``TARGET`` before each target that requires it. 35 36Requirements 37------------ 38 39Aliases can be used to express some test requirements: 40 41- ``needs/privileged`` - Requires ``--docker-privileged`` when running tests with ``--docker``. 42- ``needs/root`` - Requires running tests as ``root`` or with ``--docker``. 43- ``needs/ssh`` - Requires SSH connections to localhost (or the test container with ``--docker``) without a password. 44- ``needs/httptester`` - Requires use of the http-test-container to run tests. 45 46Dependencies 47------------ 48 49Some test dependencies are automatically discovered: 50 51- Ansible role dependencies defined in ``meta/main.yml`` files. 52- Setup targets defined with ``setup/*`` aliases. 53- Symbolic links from one target to a file in another target. 54 55Aliases can be used to declare dependencies that are not handled automatically: 56 57- ``needs/target/TARGET`` - Requires use of the test target ``TARGET``. 58- ``needs/file/PATH`` - Requires use of the file ``PATH`` relative to the git root. 59 60Skipping 61-------- 62 63Aliases can be used to skip platforms using one of the following: 64 65- ``skip/freebsd`` - Skip tests on FreeBSD. 66- ``skip/osx`` - Skip tests on macOS. 67- ``skip/rhel`` - Skip tests on RHEL. 68- ``skip/docker`` - Skip tests when running in a Docker container. 69 70Platform versions, as specified using the ``--remote`` option with ``/`` removed, can also be skipped: 71 72- ``skip/freebsd11.1`` - Skip tests on FreeBSD 11.1. 73- ``skip/rhel7.6`` - Skip tests on RHEL 7.6. 74 75Windows versions, as specified using the ``--windows`` option can also be skipped: 76 77- ``skip/windows/2008`` - Skip tests on Windows Server 2008. 78- ``skip/windows/2012-R2`` - Skip tests on Windows Server 2012 R2. 79 80Aliases can be used to skip Python major versions using one of the following: 81 82- ``skip/python2`` - Skip tests on Python 2.x. 83- ``skip/python3`` - Skip tests on Python 3.x. 84 85For more fine grained skipping, use conditionals in integration test playbooks, such as: 86 87.. code-block:: yaml 88 89 when: ansible_distribution in ('Ubuntu') 90 91 92Miscellaneous 93------------- 94 95There are several other aliases available as well: 96 97- ``destructive`` - Requires ``--allow-destructive`` to run without ``--docker`` or ``--remote``. 98- ``hidden`` - Target is ignored. Usable as a dependency. Automatic for ``setup_`` and ``prepare_`` prefixed targets. 99 100Unstable 101-------- 102 103Tests which fail sometimes should be marked with the ``unstable`` alias until the instability has been fixed. 104These tests will continue to run for pull requests which modify the test or the module under test. 105 106This avoids unnecessary test failures for other pull requests, as well as tests on merge runs and nightly CI jobs. 107 108There are two ways to run unstable tests manually: 109 110- Use the ``--allow-unstable`` option for ``ansible-test`` 111- Prefix the test name with ``unstable/`` when passing it to ``ansible-test``. 112 113Tests will be marked as unstable by a member of the Ansible Core Team. 114GitHub issues_ will be created to track each unstable test. 115 116Disabled 117-------- 118 119Tests which always fail should be marked with the ``disabled`` alias until they can be fixed. 120 121Disabled tests are automatically skipped. 122 123There are two ways to run disabled tests manually: 124 125- Use the ``--allow-disabled`` option for ``ansible-test`` 126- Prefix the test name with ``disabled/`` when passing it to ``ansible-test``. 127 128Tests will be marked as disabled by a member of the Ansible Core Team. 129GitHub issues_ will be created to track each disabled test. 130 131Unsupported 132----------- 133 134Tests which cannot be run in CI should be marked with the ``unsupported`` alias. 135Most tests can be supported through the use of simulators and/or cloud plugins. 136 137However, if that is not possible then marking a test as unsupported will prevent it from running in CI. 138 139There are two ways to run unsupported tests manually: 140 141* Use the ``--allow-unsupported`` option for ``ansible-test`` 142* Prefix the test name with ``unsupported/`` when passing it to ``ansible-test``. 143 144Tests will be marked as unsupported by the contributor of the test. 145 146Cloud 147----- 148 149Tests for cloud services and other modules that require access to external APIs usually require special support for testing in CI. 150 151These require an additional alias to indicate the required test plugin. 152 153Some of the available aliases are: 154 155- ``cloud/aws`` 156- ``cloud/azure`` 157- ``cloud/cs`` 158- ``cloud/foreman`` 159- ``cloud/openshift`` 160- ``cloud/tower`` 161- ``cloud/vcenter`` 162 163Untested 164-------- 165 166Every module and plugin should have integration tests, even if the tests cannot be run in CI. 167 168Issues 169------ 170 171Tests that are marked as unstable_ or disabled_ will have an issue created to track the status of the test. 172Each issue will be assigned to one of the following projects: 173 174- `AWS <https://github.com/ansible/ansible/projects/21>`_ 175- `Azure <https://github.com/ansible/ansible/projects/22>`_ 176- `Windows <https://github.com/ansible/ansible/projects/23>`_ 177- `General <https://github.com/ansible/ansible/projects/25>`_ 178 179Questions 180--------- 181 182For questions about integration tests reach out to @mattclay or @gundalow on GitHub or ``#ansible-devel`` on IRC. 183