mtest2make: stop disabling meson test timeouts

The mtest2make.py script passes the arg '-t 0' to 'meson test' which
disables all test timeouts. This is a major source of pain when running
in GitLab

mtest2make: stop disabling meson test timeouts

The mtest2make.py script passes the arg '-t 0' to 'meson test' which
disables all test timeouts. This is a major source of pain when running
in GitLab CI and a test gets stuck. It will stall until GitLab kills the
CI job. This leaves us with little easily consumable information about
the stalled test. The TAP format doesn't show the test name until it is
completed, and TAP output from multiple tests it interleaved. So we
have to analyse the log to figure out what tests had un-finished TAP
output present and thus infer which test case caused the hang. This is
very time consuming and error prone.

By allowing meson to kill stalled tests, we get a direct display of what
test program got stuck, which lets us more directly focus in on what
specific test case within the test program hung.

The other issue with disabling meson test timeouts by default is that it
makes it more likely that maintainers inadvertantly introduce slowdowns.
For example the recent-ish change that accidentally made migrate-test
take 15-20 minutes instead of around 1 minute.

The main risk of this change is that the individual test timeouts might
be too short to allow completion in high load scenarios. Thus, there is
likely to be some short term pain where we have to bump the timeouts for
certain tests to make them reliable enough. The preceeding few patches
raised the timeouts for all failures that were immediately apparent
in GitLab CI.

Even with the possible short term instability, this should still be a
net win for debuggability of failed CI pipelines over the long term.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20230717182859.707658-13-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20231215070357.10888-17-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

show more ...

mtest2make: stop disabling meson test timeouts

The mtest2make.py script passes the arg '-t 0' to 'meson test' which
disables all test timeouts. This is a major source of pain when running
in GitLab

mtest2make: stop disabling meson test timeouts

The mtest2make.py script passes the arg '-t 0' to 'meson test' which
disables all test timeouts. This is a major source of pain when running
in GitLab CI and a test gets stuck. It will stall until GitLab kills the
CI job. This leaves us with little easily consumable information about
the stalled test. The TAP format doesn't show the test name until it is
completed, and TAP output from multiple tests it interleaved. So we
have to analyse the log to figure out what tests had un-finished TAP
output present and thus infer which test case caused the hang. This is
very time consuming and error prone.

By allowing meson to kill stalled tests, we get a direct display of what
test program got stuck, which lets us more directly focus in on what
specific test case within the test program hung.

The other issue with disabling meson test timeouts by default is that it
makes it more likely that maintainers inadvertantly introduce slowdowns.
For example the recent-ish change that accidentally made migrate-test
take 15-20 minutes instead of around 1 minute.

The main risk of this change is that the individual test timeouts might
be too short to allow completion in high load scenarios. Thus, there is
likely to be some short term pain where we have to bump the timeouts for
certain tests to make them reliable enough. The preceeding few patches
raised the timeouts for all failures that were immediately apparent
in GitLab CI.

Even with the possible short term instability, this should still be a
net win for debuggability of failed CI pipelines over the long term.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20230717182859.707658-13-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20231215070357.10888-17-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

show more ...

mtest2make: stop disabling meson test timeouts

The mtest2make.py script passes the arg '-t 0' to 'meson test' which
disables all test timeouts. This is a major source of pain when running
in GitLab

mtest2make: stop disabling meson test timeouts

The mtest2make.py script passes the arg '-t 0' to 'meson test' which
disables all test timeouts. This is a major source of pain when running
in GitLab CI and a test gets stuck. It will stall until GitLab kills the
CI job. This leaves us with little easily consumable information about
the stalled test. The TAP format doesn't show the test name until it is
completed, and TAP output from multiple tests it interleaved. So we
have to analyse the log to figure out what tests had un-finished TAP
output present and thus infer which test case caused the hang. This is
very time consuming and error prone.

By allowing meson to kill stalled tests, we get a direct display of what
test program got stuck, which lets us more directly focus in on what
specific test case within the test program hung.

The other issue with disabling meson test timeouts by default is that it
makes it more likely that maintainers inadvertantly introduce slowdowns.
For example the recent-ish change that accidentally made migrate-test
take 15-20 minutes instead of around 1 minute.

The main risk of this change is that the individual test timeouts might
be too short to allow completion in high load scenarios. Thus, there is
likely to be some short term pain where we have to bump the timeouts for
certain tests to make them reliable enough. The preceeding few patches
raised the timeouts for all failures that were immediately apparent
in GitLab CI.

Even with the possible short term instability, this should still be a
net win for debuggability of failed CI pipelines over the long term.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20230717182859.707658-13-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20231215070357.10888-17-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

show more ...

mtest2make.py: teach suite name that are just "PROJECT"

A subproject test may be simply in the "PROJECT" suite (such as
"qemu-common" with the following patches)

Signed-off-by: Marc-André Lureau <m

mtest2make.py: teach suite name that are just "PROJECT"

A subproject test may be simply in the "PROJECT" suite (such as
"qemu-common" with the following patches)

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20230302131848.1527460-2-marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

mtest2make.py: teach suite name that are just "PROJECT"

A subproject test may be simply in the "PROJECT" suite (such as
"qemu-common" with the following patches)

Signed-off-by: Marc-André Lureau <m

mtest2make.py: teach suite name that are just "PROJECT"

A subproject test may be simply in the "PROJECT" suite (such as
"qemu-common" with the following patches)

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20230302131848.1527460-2-marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

mtest2make.py: teach suite name that are just "PROJECT"

A subproject test may be simply in the "PROJECT" suite (such as
"qemu-common" with the following patches)

Signed-off-by: Marc-André Lureau <m

mtest2make.py: teach suite name that are just "PROJECT"

A subproject test may be simply in the "PROJECT" suite (such as
"qemu-common" with the following patches)

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20230302131848.1527460-2-marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

mtest2make.py: teach suite name that are just "PROJECT"

A subproject test may be simply in the "PROJECT" suite (such as
"qemu-common" with the following patches)

Signed-off-by: Marc-André Lureau <m

mtest2make.py: teach suite name that are just "PROJECT"

A subproject test may be simply in the "PROJECT" suite (such as
"qemu-common" with the following patches)

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20230302131848.1527460-2-marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

mtest2make.py: teach suite name that are just "PROJECT"

A subproject test may be simply in the "PROJECT" suite (such as
"qemu-common" with the following patches)

Signed-off-by: Marc-André Lureau <m

mtest2make.py: teach suite name that are just "PROJECT"

A subproject test may be simply in the "PROJECT" suite (such as
"qemu-common" with the following patches)

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20230302131848.1527460-2-marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

build: add a more generic way to specify make->ninja dependencies

Let any make target specify ninja goals that needs to be built for it
(though selecting the goals is _not_ recursive on depending ta

build: add a more generic way to specify make->ninja dependencies

Let any make target specify ninja goals that needs to be built for it
(though selecting the goals is _not_ recursive on depending targets)
instead of having a custom mechanism only for "make check" and "make
bench".

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220517092616.1272238-4-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-15-alex.bennee@linaro.org>

show more ...

build: add a more generic way to specify make->ninja dependencies

Let any make target specify ninja goals that needs to be built for it
(though selecting the goals is _not_ recursive on depending ta

build: add a more generic way to specify make->ninja dependencies

Let any make target specify ninja goals that needs to be built for it
(though selecting the goals is _not_ recursive on depending targets)
instead of having a custom mechanism only for "make check" and "make
bench".

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220517092616.1272238-4-pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-15-alex.bennee@linaro.org>

show more ...

tests: Do not treat the iotests as separate meson test target anymore

If there is a failing iotest, the output is currently not logged to
the console anymore. To get this working again, we need to r

tests: Do not treat the iotests as separate meson test target anymore

If there is a failing iotest, the output is currently not logged to
the console anymore. To get this working again, we need to run the
meson test runner with "--print-errorlogs" (and without "--verbose"
due to a current meson bug that will be fixed here:
https://github.com/mesonbuild/meson/commit/c3f145ca2b9f5.patch ).
We could update the "meson test" call in tests/Makefile.include,
but actually it's nicer and easier if we simply do not treat the
iotests as separate test target anymore and integrate them along
with the other test suites. This has the disadvantage of not getting
the detailed progress indication there anymore, but since that was
only working right in single-threaded "make -j1" mode anyway, it's
not a huge loss right now.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220310075048.2303495-1-thuth@redhat.com>
Tested-by: Hanna Reitz <hreitz@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>

show more ...

build: make check-block a meson test

"meson test" can be asked to run tests verbosely; this makes it usable
also for qemu-iotests's own harness, and it lets "make check-block"
reuse mtest2make.py's

build: make check-block a meson test

"meson test" can be asked to run tests verbosely; this makes it usable
also for qemu-iotests's own harness, and it lets "make check-block"
reuse mtest2make.py's infrastructure to find and build test dependencies.

Adjust check-block.sh to use the standard exit code that reports a test
as skipped. Alternatively, in the future we could make it produce TAP
output, which is consistent with all other "make check" tests.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

scripts/mtest2make: add support for SPEED=thorough

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

build: use "meson test" as the test harness

"meson test" starting with version 0.57 is just as capable and easy to
use as QEMU's own TAP driver. All existing options for "make check"
work. The onl

build: use "meson test" as the test harness

"meson test" starting with version 0.57 is just as capable and easy to
use as QEMU's own TAP driver. All existing options for "make check"
work. The only required code change involves how to mark "slow" tests;
they need to belong to an additional "slow" suite.

The rules for .tap output are replaced by JUnit XML; GitLab is able
to parse that output and present it in the CI pipeline report.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

meson: switch minimum meson version to 0.58.2, minimum recommended to 0.59.2

Meson 0.58.2 does not need b_staticpic=$pie anymore, and has
stabilized the keyval module. Remove the workaround and use

meson: switch minimum meson version to 0.58.2, minimum recommended to 0.59.2

Meson 0.58.2 does not need b_staticpic=$pie anymore, and has
stabilized the keyval module. Remove the workaround and use a few
replacements for features deprecated in the 0.57.0 release cycle.

One feature that we would like to use is passing dependencies to
summary. However, that was broken in 0.59.0 and 0.59.1. Therefore,
use the embedded Meson if the host has anything older than 0.59.2,
but allow --meson= to use 0.58.2.

Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

meson: Re-enable the possibility to run "make check SPEED=slow"

"make check SPEED=slow" got lost in the conversion of the build
system to meson - the tests were always running in "quick" mode.
Fix i

meson: Re-enable the possibility to run "make check SPEED=slow"

"make check SPEED=slow" got lost in the conversion of the build
system to meson - the tests were always running in "quick" mode.
Fix it by passing the "-m" parameter to the test harness at the
right spot in scripts/mtest2make.py.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20210218172313.2217440-1-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>

show more ...

scripts/mtest2make.py: export all-%s-targets variable and use it

There are some places where the conditional makefile support is the
simplest solution. Now we don't expose CONFIG_TCG as a variable c

scripts/mtest2make.py: export all-%s-targets variable and use it

There are some places where the conditional makefile support is the
simplest solution. Now we don't expose CONFIG_TCG as a variable create
a new one that can be checked for the check-help output.

As check-tcg is a PHONY target we re-use check-softfloat to gate that
as well.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20210202134001.25738-13-alex.bennee@linaro.org>

show more ...

build: replace ninjatool with ninja

Now that the build is done entirely by Meson, there is no need
to keep the Makefile conversion. Instead, we can ask Ninja about
the targets it exposes and forwar

build: replace ninjatool with ninja

Now that the build is done entirely by Meson, there is no need
to keep the Makefile conversion. Instead, we can ask Ninja about
the targets it exposes and forward them.

The main advantages are, from smallest to largest:

- reducing the possible namespace pollution within the Makefile

- removal of a relatively large Python program

- faster build because parsing Makefile.ninja is slower than
parsing build.ninja; and faster build after Meson runs because
we do not have to generate Makefile.ninja.

- tracking of command lines, which provides more accurate rebuilds

In addition the change removes the requirement for GNU make 3.82, which
was annoying on Mac, and avoids bugs on Windows due to ninjatool not
knowing how to convert Windows escapes to POSIX escapes.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

mtest2make: add support for introspected test dependencies

Right now all "make check" targets depend blindly on "all". If Meson
is 0.56.0 or newer, we can use the correct dependencies using the new

mtest2make: add support for introspected test dependencies

Right now all "make check" targets depend blindly on "all". If Meson
is 0.56.0 or newer, we can use the correct dependencies using the new
"depends" entry in "meson introspect --tests".

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

meson: convert the speed tests

Use meson benchmark() for them, adjust mtest2make.py for that.
A new target "make bench" can be used to run all benchmarks.

Signed-off-by: Marc-André Lureau <marcandr

meson: convert the speed tests

Use meson benchmark() for them, adjust mtest2make.py for that.
A new target "make bench" can be used to run all benchmarks.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20200828110734.1638685-14-marcandre.lureau@redhat.com>
[Rewrite mtest2make part. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

mtest2make: unify tests that appear in multiple suites

Whenever a test appears in multiple suites, the rules generated
by mtest2make are currently running it twice. Instead, after
this patch we gen

mtest2make: unify tests that appear in multiple suites

Whenever a test appears in multiple suites, the rules generated
by mtest2make are currently running it twice. Instead, after
this patch we generate a phony target for each test and we have
a generic "run-tests" target depend on all the tests that were
chosen on the command line. Tests that appear in multiple suites
will be added to the prerequisites just once.

This has other advantages: it removes the handling of -k and
it increases parallelism.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

mtest2make: hide output of successful tests

The softfloat tests are quite noisy; before the Meson conversion
they buffered the output in a file and emitted the output only
if the test failed. Tweak

mtest2make: hide output of successful tests

The softfloat tests are quite noisy; before the Meson conversion
they buffered the output in a file and emitted the output only
if the test failed. Tweak mtest2make.py so that the courtesy
is extended to all non-TAP tests.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

mtest2make: split working directory from test command

Pass the working directory and test command in separate macro arguments,
so that we will be able to insert a test driver in the next patch.

Sig

mtest2make: split working directory from test command

Pass the working directory and test command in separate macro arguments,
so that we will be able to insert a test driver in the next patch.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

mtest2make: split environment from test command

Pass the environment and test command in separate macro arguments,
so that we will be able to insert a test driver in the next patch.

Signed-off-by:

mtest2make: split environment from test command

Pass the environment and test command in separate macro arguments,
so that we will be able to insert a test driver in the next patch.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

meson: fix relpath failure on Win32

On win32, os.path.relpath can raise an exception when computing
for example C:/msys64/mingw64/x.exe relative to E:/path/qemu-build.
Use try...except to avoid this

meson: fix relpath failure on Win32

On win32, os.path.relpath can raise an exception when computing
for example C:/msys64/mingw64/x.exe relative to E:/path/qemu-build.
Use try...except to avoid this, just using an absolute path in
this case.

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...