51ab5f8b | 15-Mar-2023 |
Daniel P. Berrangé <berrange@redhat.com> |
iotests: register each I/O test separately with meson
Currently meson registers a single test that invokes an entire group of I/O tests, hiding the test granularity from meson. There are various dow
iotests: register each I/O test separately with meson
Currently meson registers a single test that invokes an entire group of I/O tests, hiding the test granularity from meson. There are various downsides of doing this
* You cannot ask 'meson test' to invoke a single I/O test * The meson test timeout can't be applied to the individual tests * Meson only gets a pass/fail for the overall I/O test group not individual tests * If a CI job gets killed by the GitLab timeout, we don't get visibility into how far through the I/O tests execution got.
This switches meson to perform test discovery by invoking 'check' in dry-run mode. It then registers one meson test case for each I/O test. Parallel execution remains disabled since the I/O tests do not use self contained execution environments and thus conflict with each other.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Acked-by: Hanna Czenczek <hreitz@redhat.com> Tested-by: Thomas Huth <thuth@redhat.com> Message-Id: <20230303160727.3977246-8-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230315174331.2959-25-alex.bennee@linaro.org>
show more ...
|
5ba7db09 | 15-Mar-2023 |
Daniel P. Berrangé <berrange@redhat.com> |
iotests: always use a unique sub-directory per test
The current test runner is only safe against parallel execution within a single instance of the 'check' process, and only if -j is given a value g
iotests: always use a unique sub-directory per test
The current test runner is only safe against parallel execution within a single instance of the 'check' process, and only if -j is given a value greater than 2. This prevents running multiple copies of the 'check' process for different test scenarios.
This change switches the output / socket directories to always include the test name, image format and image protocol. This should allow full parallelism of all distinct test scenarios. eg running both qcow2 and raw tests at the same time, or both file and nbd tests at the same time.
It would be possible to allow for parallelism of the same test scenario by including the pid, but that would potentially let many directories accumulate over time on failures, so is not done.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Acked-by: Hanna Czenczek <hreitz@redhat.com> Tested-by: Thomas Huth <thuth@redhat.com> Message-Id: <20230303160727.3977246-7-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230315174331.2959-24-alex.bennee@linaro.org>
show more ...
|
cb845eaa | 15-Mar-2023 |
Daniel P. Berrangé <berrange@redhat.com> |
iotests: connect stdin to /dev/null when running tests
Currently the tests have their stdin inherited from the test harness, meaning they are connected to a TTY. The QEMU processes spawned by certai
iotests: connect stdin to /dev/null when running tests
Currently the tests have their stdin inherited from the test harness, meaning they are connected to a TTY. The QEMU processes spawned by certain tests, however, modify TTY settings and if the test exits abnormally the settings might not be restored.
The python test harness thus has some logic which will capture the initial TTY settings and restore them once all tests are finished.
This does not, however, take into account the possibility of many copies of the 'check' program running in parallel. With parallel execution, a later invokation may save the TTY state that QEMU has already modified, and thus restore bad state leaving the TTY non-functional.
None of the I/O tests shnould actually be interactive requiring user input and so they should not require a TTY at all. To avoid this while TTY save/restore complexity we can connect the test stdin to /dev/null instead.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Acked-by: Hanna Czenczek <hreitz@redhat.com> Tested-by: Thomas Huth <thuth@redhat.com> Message-Id: <20230303160727.3977246-6-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230315174331.2959-23-alex.bennee@linaro.org>
show more ...
|
6e5792a1 | 15-Mar-2023 |
Daniel P. Berrangé <berrange@redhat.com> |
iotests: print TAP protocol version when reporting tests
Recently meson started complaining that TAP test reports don't include the TAP protocol version. While this warning is bogus and has since be
iotests: print TAP protocol version when reporting tests
Recently meson started complaining that TAP test reports don't include the TAP protocol version. While this warning is bogus and has since been removed from Meson, it looks like good practice to include this header going forward. The GLib library test harness has started unconditionally printing the version, so this brings the I/O tests into line.
Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Acked-by: Hanna Czenczek <hreitz@redhat.com> Tested-by: Thomas Huth <thuth@redhat.com> Message-Id: <20230303160727.3977246-5-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230315174331.2959-22-alex.bennee@linaro.org>
show more ...
|
663755b0 | 15-Mar-2023 |
Daniel P. Berrangé <berrange@redhat.com> |
iotests: strip subdir path when listing tests
When asking 'check' to list individual tests by invoking it in dry run mode, it prints the paths to the tests relative to the base of the I/O test direc
iotests: strip subdir path when listing tests
When asking 'check' to list individual tests by invoking it in dry run mode, it prints the paths to the tests relative to the base of the I/O test directory.
When asking 'check' to run an individual test, however, it mandates that only the unqualified test name is given, without any path prefix. This inconsistency makes it harder to ask for a list of tests and then invoke each one.
Thus the test listing code is change to flatten the test names, by printing only the base name, which can be directly invoked.
Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Acked-by: Hanna Czenczek <hreitz@redhat.com> Tested-by: Thomas Huth <thuth@redhat.com> Message-Id: <20230303160727.3977246-4-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230315174331.2959-21-alex.bennee@linaro.org>
show more ...
|
a9e21786 | 15-Mar-2023 |
Daniel P. Berrangé <berrange@redhat.com> |
iotests: allow test discovery before building
The 'check' script can be invoked in "dry run" mode, in which case it merely does test discovery and prints out all their names. Despite only doing test
iotests: allow test discovery before building
The 'check' script can be invoked in "dry run" mode, in which case it merely does test discovery and prints out all their names. Despite only doing test discovery it still validates that the various QEMU binaries can be found. This makes it impossible todo test discovery prior to building QEMU. This is a desirable feature to support, because it will let meson discover tests.
Fortunately the code in the TestEnv constructor is ordered in a way that makes this fairly trivial to achieve. We can just short circuit the constructor after the basic directory paths have been set.
Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Acked-by: Hanna Czenczek <hreitz@redhat.com> Tested-by: Thomas Huth <thuth@redhat.com> Message-Id: <20230303160727.3977246-3-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230315174331.2959-20-alex.bennee@linaro.org>
show more ...
|