tests: respect --enable/--disable-download for Avocado

Pass the content of $mkvenv_flags (which is either "--online"
or empty) down to tests/Makefile.include.

Signed-off-by: Paolo Bonzini <pbonzini

tests: respect --enable/--disable-download for Avocado

Pass the content of $mkvenv_flags (which is either "--online"
or empty) down to tests/Makefile.include.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

Makefile: build plugins before running TCG tests

Add back test-plugins and, after making sure it is always defined,
do so unconditionally.

Reported-by: Alex Bennée <alex.bennee@linaro.org>
Fixes: 2

Makefile: build plugins before running TCG tests

Add back test-plugins and, after making sure it is always defined,
do so unconditionally.

Reported-by: Alex Bennée <alex.bennee@linaro.org>
Fixes: 2c13c574418 ("configure, meson: move --enable-plugins to meson", 2023-09-07)
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

Makefile: build plugins before running TCG tests

Add back test-plugins and, after making sure it is always defined,
do so unconditionally.

Reported-by: Alex Bennée <alex.bennee@linaro.org>
Fixes: 2

Makefile: build plugins before running TCG tests

Add back test-plugins and, after making sure it is always defined,
do so unconditionally.

Reported-by: Alex Bennée <alex.bennee@linaro.org>
Fixes: 2c13c574418 ("configure, meson: move --enable-plugins to meson", 2023-09-07)
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

Makefile: build plugins before running TCG tests

Add back test-plugins and, after making sure it is always defined,
do so unconditionally.

Reported-by: Alex Bennée <alex.bennee@linaro.org>
Fixes: 2

Makefile: build plugins before running TCG tests

Add back test-plugins and, after making sure it is always defined,
do so unconditionally.

Reported-by: Alex Bennée <alex.bennee@linaro.org>
Fixes: 2c13c574418 ("configure, meson: move --enable-plugins to meson", 2023-09-07)
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

configure, meson: move --enable-plugins to meson

While the option still needs to be parsed in the configure script
(it's needed by tests/tcg, and also to decide about recursing
into contrib/plugins)

configure, meson: move --enable-plugins to meson

While the option still needs to be parsed in the configure script
(it's needed by tests/tcg, and also to decide about recursing
into contrib/plugins), passing it to Meson can be done with -D
instead of using config-host.mak.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

Revert "tests: Use separate virtual environment for avocado"

This reverts commit e8e4298feadae7924cf7600bb3bcc5b0a8d7cbe9.

ensuregroup allows to specify both the acceptable versions of avocado,
and

Revert "tests: Use separate virtual environment for avocado"

This reverts commit e8e4298feadae7924cf7600bb3bcc5b0a8d7cbe9.

ensuregroup allows to specify both the acceptable versions of avocado,
and a locked version to be used when avocado is not installed as a system
pacakge. This lets us install avocado in pyvenv/ using "mkvenv.py" and
reuse the distro package on Fedora and CentOS Stream (the only distros
where it's available).

ensuregroup's usage of "(>=..., <=...)" constraints when evaluating
the distro package, and "==" constraints when installing it from PyPI,
makes it possible to avoid conflicts between the known-good version and
a package plugins included in the distro.

This is because package plugins have "==" constraints on the version
that is included in the distro, and, using "pip install avocado==88.1"
on a venv that includes system packages will result in an error:

avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.

But at the same time, if the venv does not include a system distribution
of avocado then we can install a known-good version and stick to LTS
releases.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1663
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

Revert "tests: Use separate virtual environment for avocado"

This reverts commit e8e4298feadae7924cf7600bb3bcc5b0a8d7cbe9.

ensuregroup allows to specify both the acceptable versions of avocado,
and

Revert "tests: Use separate virtual environment for avocado"

This reverts commit e8e4298feadae7924cf7600bb3bcc5b0a8d7cbe9.

ensuregroup allows to specify both the acceptable versions of avocado,
and a locked version to be used when avocado is not installed as a system
pacakge. This lets us install avocado in pyvenv/ using "mkvenv.py" and
reuse the distro package on Fedora and CentOS Stream (the only distros
where it's available).

ensuregroup's usage of "(>=..., <=...)" constraints when evaluating
the distro package, and "==" constraints when installing it from PyPI,
makes it possible to avoid conflicts between the known-good version and
a package plugins included in the distro.

This is because package plugins have "==" constraints on the version
that is included in the distro, and, using "pip install avocado==88.1"
on a venv that includes system packages will result in an error:

avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.

But at the same time, if the venv does not include a system distribution
of avocado then we can install a known-good version and stick to LTS
releases.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1663
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

tests: Use separate virtual environment for avocado

This reverts commits eea2d141179 ("Makefile: remove $(TESTS_PYTHON)",
2023-05-26) and 9c6692db550 ("tests: Use configure-provided pyvenv for
tests

tests: Use separate virtual environment for avocado

This reverts commits eea2d141179 ("Makefile: remove $(TESTS_PYTHON)",
2023-05-26) and 9c6692db550 ("tests: Use configure-provided pyvenv for
tests", 2023-05-18).

Right now, there is a conflict between wanting a ">=" constraint when
using a distro-provided package and wanting a "==" constraint when
installing Avocado from PyPI; this would provide the best of both worlds
in terms of resiliency for both distros that have required packages and
distros that don't.

The conflict is visible also for meson, where we would like to install
the latest 0.63.x version but also accept a distro 1.1.x version.
But it is worse for avocado, for two reasons:

1) we cannot use an "==" constraint to install avocado if the venv
includes a system avocado. The distro will package plugins that have
"==" constraints on the version that is included in the distro, and, using
"pip install avocado==88.1" on a venv that includes system packages will
result in this error:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
make[1]: Leaving directory '/home/berrange/src/virt/qemu/build'

2) we cannot use ">=" either if the venv does _not_ include a system
avocado, because that would result in the installation of v101.0 which
is the one we've just reverted.

So the idea is to encode the dependencies as an (acceptable, locked)
tuple, like this hypothetical TOML that would be committed inside
python/ and used by mkvenv.py:

[meson]
meson = { minimum = "0.63.0", install = "0.63.3", canary = "meson" }

[docs]
# 6.0 drops support for Python 3.7
sphinx = { minimum = "1.6", install = "<6.0", canary = "sphinx-build" }
sphinx_rtd_theme = { minimum = "0.5" }

[avocado]
avocado-framework = { minimum = "88.1", install = "88.1", canary = "avocado" }

Once this is implemented, it would also be possible to install avocado in
pyvenv/ using "mkvenv.py ensure", thus using the distro package on Fedora
and CentOS Stream (the only distros where it's available). But until
this is implemented, keep avocado in a separate venv. There is still the
benefit of using a single python for meson custom_targets and for sphinx.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

Revert "tests/requirements.txt: bump up avocado-framework version to 101.0"

This reverts commit ec5ffa0056389c3c10ea2de1e78366f66f4e5abc.

Bumping avocado to version 101 has two issues. First, ther

Revert "tests/requirements.txt: bump up avocado-framework version to 101.0"

This reverts commit ec5ffa0056389c3c10ea2de1e78366f66f4e5abc.

Bumping avocado to version 101 has two issues. First, there are problems
where Avocado is not logging of command lines or terminal output, and not
collecting Python logs outside the avocado namespace.

Second, the recent changes to Python handling mean that there is a single
virtual environment for all the build, instead of a separate one for testing.
Requiring a too-new version of avocado causes conflicts with any avocado
plugins installed on the host:

$ make check-venv
make[1]: Entering directory '/home/berrange/src/virt/qemu/build'
GIT ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
VENVPIP install -e /home/berrange/src/virt/qemu/python/
VENVPIP install -r /home/berrange/src/virt/qemu/tests/requirements.txt
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
make[1]: Leaving directory '/home/berrange/src/virt/qemu/build'

To avoid this issue, tests/requirements.txt should use a ">=" constraint
and the version of Avocado should be limited to what distros provide
in the system packages. Only Fedora has Avocado, and more specifically
version 92.0 (though 98.0 is also available as a module). As a first
step, this patch reverts the introduction of a too-new Avocado.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

Makefile: remove $(TESTS_PYTHON)

It is now the same as $(PYTHON), since the latter always points at pyvenv/bin/python3.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzin

Makefile: remove $(TESTS_PYTHON)

It is now the same as $(PYTHON), since the latter always points at pyvenv/bin/python3.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

Makefile: remove $(TESTS_PYTHON)

It is now the same as $(PYTHON), since the latter always points at pyvenv/bin/python3.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzin

Makefile: remove $(TESTS_PYTHON)

It is now the same as $(PYTHON), since the latter always points at pyvenv/bin/python3.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

Makefile: remove $(TESTS_PYTHON)

It is now the same as $(PYTHON), since the latter always points at pyvenv/bin/python3.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzin

Makefile: remove $(TESTS_PYTHON)

It is now the same as $(PYTHON), since the latter always points at pyvenv/bin/python3.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

tests: Use configure-provided pyvenv for tests

This patch changes how the avocado tests are provided, ever so
slightly. Instead of creating a new testing venv, use the
configure-provided 'pyvenv' in

tests: Use configure-provided pyvenv for tests

This patch changes how the avocado tests are provided, ever so
slightly. Instead of creating a new testing venv, use the
configure-provided 'pyvenv' instead, and install optional packages into
that.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-20-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

tests: Use configure-provided pyvenv for tests

This patch changes how the avocado tests are provided, ever so
slightly. Instead of creating a new testing venv, use the
configure-provided 'pyvenv' in

tests: Use configure-provided pyvenv for tests

This patch changes how the avocado tests are provided, ever so
slightly. Instead of creating a new testing venv, use the
configure-provided 'pyvenv' instead, and install optional packages into
that.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-20-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

tests/requirements.txt: bump up avocado-framework version to 101.0

Avocado version 101.0 has a fix to re-compute the checksum
of an asset file if the algorithm used in the *-CHECKSUM
file isn't the

tests/requirements.txt: bump up avocado-framework version to 101.0

Avocado version 101.0 has a fix to re-compute the checksum
of an asset file if the algorithm used in the *-CHECKSUM
file isn't the same as the one being passed to it by the
avocado user (i.e. the avocado_qemu python module).
In the earlier avocado versions this fix wasn't there due
to which if the checksum wouldn't match the earlier
checksum (calculated by a different algorithm), the avocado
code would start downloading a fresh image from the internet
URL thus making the test-cases take longer to execute.

Bump up the avocado-framework version to 101.0.

Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
Message-Id: <20230424092249.58552-2-alex.bennee@linaro.org>

show more ...

tests/requirements.txt: bump up avocado-framework version to 101.0

Avocado version 101.0 has a fix to re-compute the checksum
of an asset file if the algorithm used in the *-CHECKSUM
file isn't the

tests/requirements.txt: bump up avocado-framework version to 101.0

Avocado version 101.0 has a fix to re-compute the checksum
of an asset file if the algorithm used in the *-CHECKSUM
file isn't the same as the one being passed to it by the
avocado user (i.e. the avocado_qemu python module).
In the earlier avocado versions this fix wasn't there due
to which if the checksum wouldn't match the earlier
checksum (calculated by a different algorithm), the avocado
code would start downloading a fresh image from the internet
URL thus making the test-cases take longer to execute.

Bump up the avocado-framework version to 101.0.

Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
Message-Id: <20230424092249.58552-2-alex.bennee@linaro.org>

show more ...

tests/requirements.txt: bump up avocado-framework version to 101.0

Avocado version 101.0 has a fix to re-compute the checksum
of an asset file if the algorithm used in the *-CHECKSUM
file isn't the

tests/requirements.txt: bump up avocado-framework version to 101.0

Avocado version 101.0 has a fix to re-compute the checksum
of an asset file if the algorithm used in the *-CHECKSUM
file isn't the same as the one being passed to it by the
avocado user (i.e. the avocado_qemu python module).
In the earlier avocado versions this fix wasn't there due
to which if the checksum wouldn't match the earlier
checksum (calculated by a different algorithm), the avocado
code would start downloading a fresh image from the internet
URL thus making the test-cases take longer to execute.

Bump up the avocado-framework version to 101.0.

Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
Message-Id: <20230424092249.58552-2-alex.bennee@linaro.org>

show more ...

configure: move tests/tcg/Makefile.prereqs to root build directory

It will not be specific to tests/tcg anymore, since it will be possible to
build firmware using container-based cross compilers too

configure: move tests/tcg/Makefile.prereqs to root build directory

It will not be specific to tests/tcg anymore, since it will be possible to
build firmware using container-based cross compilers too.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220929114231.583801-26-alex.bennee@linaro.org>

show more ...

tests/tcg: add distclean rule

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220929114231.583801-22-alex.bennee@linaro.org>

tests/tcg: remove -f from Makefile invocation

Instead of linking tests/tcg/Makefile.target into the build tree, name
the symbolic link "Makefile" and create it in every target subdirectory.
This mak

tests/tcg: remove -f from Makefile invocation

Instead of linking tests/tcg/Makefile.target into the build tree, name
the symbolic link "Makefile" and create it in every target subdirectory.
This makes it possible to just invoke "make" in tests/tcg subdirectories.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220929114231.583801-21-alex.bennee@linaro.org>

show more ...

tests: simplify Makefile invocation for tests/tcg

Remove the DOCKER_SCRIPT and TARGET variable from the Makefile invocation
for tests/tcg. For DOCKER_SCRIPT, resolve the path to docker.py in config

tests: simplify Makefile invocation for tests/tcg

Remove the DOCKER_SCRIPT and TARGET variable from the Makefile invocation
for tests/tcg. For DOCKER_SCRIPT, resolve the path to docker.py in configure;
for TARGET, move it to config-$(TARGET).mak and use a symbolic link to break
the cycle.

The symbolic link is still needed because tests/tcg includes dummy config files
for targets that are not buildable. Once that is cleaned up, the symbolic link
will go away too.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220929114231.583801-20-alex.bennee@linaro.org>

show more ...

tests: install "qemu" namespace package into venv

This patch adds the "qemu" namespace package to the $build/tests/venv
directory. It does so in "editable" mode, which means that changes to
the sour

tests: install "qemu" namespace package into venv

This patch adds the "qemu" namespace package to the $build/tests/venv
directory. It does so in "editable" mode, which means that changes to
the source python directory will actively be reflected by the venv.

This patch also then removes any sys.path hacking from the avocado test
scripts directly. By doing this, the environment of where to find these
packages is managed entirely by the virtual environment and not by the
scripts themselves.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220526000921.1581503-7-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

tests: add quiet-venv-pip macro

Factor out the "test venv pip" macro; rewrite the "check-venv" rule to
be a little more compact. Replace the "PIP" pseudo-command output with
"VENVPIP" to make it 1%

tests: add quiet-venv-pip macro

Factor out the "test venv pip" macro; rewrite the "check-venv" rule to
be a little more compact. Replace the "PIP" pseudo-command output with
"VENVPIP" to make it 1% more clear that we are talking about using pip
to install something into a venv.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220526000921.1581503-6-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

tests: silence pip upgrade warnings during venv creation

Turn off the nag warning coaxing us to upgrade pip. It's not really that
interesting to see in CI logs, and as long as nothing is broken --
n

tests: silence pip upgrade warnings during venv creation

Turn off the nag warning coaxing us to upgrade pip. It's not really that
interesting to see in CI logs, and as long as nothing is broken --
nothing is broken.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220526000921.1581503-5-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

tests: use python3 as the python executable name

Use "python3" instead of "python" as per PEP0394:
https://peps.python.org/pep-0394/

This should always be defined (in a venv, at least!), matching t

tests: use python3 as the python executable name

Use "python3" instead of "python" as per PEP0394:
https://peps.python.org/pep-0394/

This should always be defined (in a venv, at least!), matching the
preferred python shebang of "#!/usr/bin/env python3".

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220526000921.1581503-4-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...