gitlab: aggressively avoid extra GIT data

This avoids fetching blobs and tree references for branches we are not
going to worry about. Also skip tag references which are similarly not
useful and kee

gitlab: aggressively avoid extra GIT data

This avoids fetching blobs and tree references for branches we are not
going to worry about. Also skip tag references which are similarly not
useful and keep the default --prune. This keeps the .git data to
around 100M rather than the ~400M even a shallow clone takes.

So we can check the savings we also run a quick du while setting up
the build.

We also have to have special settings of GIT_FETCH_EXTRA_FLAGS for the
Windows build, the migration legacy test and the custom runners. In
the case of the custom runners we also move the free floating variable
to the runner template.

Reviewed-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240312170011.1688444-1-alex.bennee@linaro.org>

show more ...

gitlab: custom-runners: preserve more artifacts for debugging

Since custom runners are not generally available, make it possible to
debug the differences between a successful and a failing build by

gitlab: custom-runners: preserve more artifacts for debugging

Since custom runners are not generally available, make it possible to
debug the differences between a successful and a failing build by
comparing the logs and the build.ninja rules.

Acked-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

gitlab: custom-runners: preserve more artifacts for debugging

Since custom runners are not generally available, make it possible to
debug the differences between a successful and a failing build by

gitlab: custom-runners: preserve more artifacts for debugging

Since custom runners are not generally available, make it possible to
debug the differences between a successful and a failing build by
comparing the logs and the build.ninja rules.

Acked-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

gitlab: extend custom runners with base_job_template

The base job template is responsible for controlling how we kick off
testing on our various branches. Rename and extend the
custom_runner_templat

gitlab: extend custom runners with base_job_template

The base job template is responsible for controlling how we kick off
testing on our various branches. Rename and extend the
custom_runner_template so we can take advantage of all that control.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20230228190653.1602033-7-alex.bennee@linaro.org>

show more ...

gitlab: extend custom runners with base_job_template

The base job template is responsible for controlling how we kick off
testing on our various branches. Rename and extend the
custom_runner_templat

gitlab: extend custom runners with base_job_template

The base job template is responsible for controlling how we kick off
testing on our various branches. Rename and extend the
custom_runner_template so we can take advantage of all that control.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20230228190653.1602033-7-alex.bennee@linaro.org>

show more ...

gitlab: extend custom runners with base_job_template

The base job template is responsible for controlling how we kick off
testing on our various branches. Rename and extend the
custom_runner_templat

gitlab: extend custom runners with base_job_template

The base job template is responsible for controlling how we kick off
testing on our various branches. Rename and extend the
custom_runner_template so we can take advantage of all that control.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20230228190653.1602033-7-alex.bennee@linaro.org>

show more ...

gitlab: wrap up test results for custom runners

Instead of spewing the whole log to stdout lets just define them as
build artefacts so we can examine them later. Where we are running
check-tcg run i

gitlab: wrap up test results for custom runners

Instead of spewing the whole log to stdout lets just define them as
build artefacts so we can examine them later. Where we are running
check-tcg run it first as those tests are yet to be integrated into
meson. To avoid confusion we don't run multiple check-tcg tests at
once.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230124180127.1881110-18-alex.bennee@linaro.org>

show more ...

gitlab: wrap up test results for custom runners

Instead of spewing the whole log to stdout lets just define them as
build artefacts so we can examine them later. Where we are running
check-tcg run i

gitlab: wrap up test results for custom runners

Instead of spewing the whole log to stdout lets just define them as
build artefacts so we can examine them later. Where we are running
check-tcg run it first as those tests are yet to be integrated into
meson. To avoid confusion we don't run multiple check-tcg tests at
once.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230124180127.1881110-18-alex.bennee@linaro.org>

show more ...

gitlab-ci: update aarch32/aarch64 custom runner jobs

The custom runner is now using 22.04 so we can drop our hacks to deal
with broken libssh and glusterfs. The provisioning scripts will be
updated

gitlab-ci: update aarch32/aarch64 custom runner jobs

The custom runner is now using 22.04 so we can drop our hacks to deal
with broken libssh and glusterfs. The provisioning scripts will be
updated in a separate commit.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220914155950.804707-21-alex.bennee@linaro.org>

show more ...

gitlab-ci: update aarch32/aarch64 custom runner jobs

The custom runner is now using 22.04 so we can drop our hacks to deal
with broken libssh and glusterfs. The provisioning scripts will be
updated

gitlab-ci: update aarch32/aarch64 custom runner jobs

The custom runner is now using 22.04 so we can drop our hacks to deal
with broken libssh and glusterfs. The provisioning scripts will be
updated in a separate commit.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220914155950.804707-21-alex.bennee@linaro.org>

show more ...

gitlab-ci: update aarch32/aarch64 custom runner jobs

The custom runner is now using 22.04 so we can drop our hacks to deal
with broken libssh and glusterfs. The provisioning scripts will be
updated

gitlab-ci: update aarch32/aarch64 custom runner jobs

The custom runner is now using 22.04 so we can drop our hacks to deal
with broken libssh and glusterfs. The provisioning scripts will be
updated in a separate commit.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220914155950.804707-21-alex.bennee@linaro.org>

show more ...

gitlab-ci: update aarch32/aarch64 custom runner jobs

The custom runner is now using 22.04 so we can drop our hacks to deal
with broken libssh and glusterfs. The provisioning scripts will be
updated

gitlab-ci: update aarch32/aarch64 custom runner jobs

The custom runner is now using 22.04 so we can drop our hacks to deal
with broken libssh and glusterfs. The provisioning scripts will be
updated in a separate commit.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220914155950.804707-21-alex.bennee@linaro.org>

show more ...

gitlab: include new aarch32 job in custom-runners

Without linking it in it won't be presented on the UI. Also while
doing that fix the misnamed job from 20.40 to 20.04.

Fixes: cc44a16002 ("gitlab:

gitlab: include new aarch32 job in custom-runners

Without linking it in it won't be presented on the UI. Also while
doing that fix the misnamed job from 20.40 to 20.04.

Fixes: cc44a16002 ("gitlab: add a new aarch32 custom runner definition")
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20220315121954.2283887-1-alex.bennee@linaro.org>

show more ...

gitlab: upgrade the job definition for s390x to 20.04

The new s390x machine has more of everything including the OS. As
18.04 will soon be going we might as well get onto something moderately
modern

gitlab: upgrade the job definition for s390x to 20.04

The new s390x machine has more of everything including the OS. As
18.04 will soon be going we might as well get onto something moderately
modern.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Christian Borntraeger <borntraeger@de.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Acked-by: Cornelia Huck <cohuck@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Cc: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220225172021.3493923-16-alex.bennee@linaro.org>

show more ...

gitlab-ci: Split custom-runners.yml in one file per runner

To ease maintenance, add the custom-runners/ directory and
split custom-runners.yml in 3 files, all included by the
current custom-runners.

gitlab-ci: Split custom-runners.yml in one file per runner

To ease maintenance, add the custom-runners/ directory and
split custom-runners.yml in 3 files, all included by the
current custom-runners.yml:
- ubuntu-18.04-s390x.yml
- ubuntu-20.04-aarch64.yml
- centos-stream-8-x86_64.yml

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20211115095608.2436223-1-philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20211115142915.3797652-7-alex.bennee@linaro.org>

show more ...

Jobs based on custom runners: add CentOS Stream 8

This introduces three different parts of a job designed to run
on a custom runner managed by Red Hat. The goals include:

a) propose a model for

Jobs based on custom runners: add CentOS Stream 8

This introduces three different parts of a job designed to run
on a custom runner managed by Red Hat. The goals include:

a) propose a model for other organizations that want to onboard
their own runners, with their specific platforms, build
configuration and tests.

b) bring awareness to the differences between upstream QEMU and the
version available under CentOS Stream, which is "A preview of
upcoming Red Hat Enterprise Linux minor and major releases".

c) because of b), it should be easier to identify and reduce the gap
between Red Hat's downstream and upstream QEMU.

The components of this custom job are:

I) OS build environment setup code:

- additions to the existing "build-environment.yml" playbook
that can be used to set up CentOS/EL 8 systems.

- a CentOS Stream 8 specific "build-environment.yml" playbook
that adds to the generic one.

II) QEMU build configuration: a script that will produce binaries with
features as similar as possible to the ones built and packaged on
CentOS stream 8.

III) Scripts that define the minimum amount of testing that the
binaries built with the given configuration (point II) under the
given OS build environment (point I) should be subjected to.

IV) Job definition: GitLab CI jobs that will dispatch the build/test
jobs (see points #II and #III) to the machine specifically
configured according to #I.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Tested-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20211111160501.862396-2-crosa@redhat.com>
Message-Id: <20211115142915.3797652-6-alex.bennee@linaro.org>

show more ...

gitlab-ci: Mark manual-only jobs as allow_failure

If a gitlab CI job is marked as manual-only but is not marked
as allow_failure, then gitlab considers that the pipeline is
"blocked" until the job h

gitlab-ci: Mark manual-only jobs as allow_failure

If a gitlab CI job is marked as manual-only but is not marked
as allow_failure, then gitlab considers that the pipeline is
"blocked" until the job has been manually triggered. We need
to mark these manual-only jobs as also allow_failure: true
so that gitlab doesn't insist that they have run before it
will consider the pipeline to be complete.

Fixes: 4c9af1ea1457782cf0adb29
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20210915123412.8232-1-peter.maydell@linaro.org
Acked-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>

show more ...

gitlab-ci: Make more custom runner jobs manual, and don't allow failure

Currently we define a lot of jobs for our custom runners:
for both aarch64 and s390x we have
- all-linux-static
- all
- all

gitlab-ci: Make more custom runner jobs manual, and don't allow failure

Currently we define a lot of jobs for our custom runners:
for both aarch64 and s390x we have
- all-linux-static
- all
- alldbg
- clang (manual)
- tci
- notcg (manual)

This is overkill. The main reason to run on these hosts is to get
coverage for the host architecture; we can leave the handling of
differences like debug vs non-debug to the x86 CI jobs.

The jobs are also generally running OK; they occasionally fail due to
timeouts, which is likely because we're overloading the machine by
asking it to run 4 CI jobs at once plus the ad-hoc CI.

Remove the 'allow_failure' tag from all these jobs, and switch the
s390x-alldbg, aarch64-all, s390x-tci and aarch64-tci jobs to manual.
(We keep -all on s390x and -alldbg on aarch64 just for diversity
of coverage.)

This will let us make the switch for s390x and aarch64 hosts from
the ad-hoc CI to gitlab.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-id: 20210913101948.12600-1-peter.maydell@linaro.org

show more ...

gitlab-ci: Fix ..._RUNNER_AVAILABLE variables and document them

The patch that recently introduced the S390X_RUNNER_AVAILABLE variable
in custom-runners.yml missed that the bottom half of the file i

gitlab-ci: Fix ..._RUNNER_AVAILABLE variables and document them

The patch that recently introduced the S390X_RUNNER_AVAILABLE variable
in custom-runners.yml missed that the bottom half of the file is rather
about aarch64 than s390x. Thus rename the S390X_RUNNER_AVAILABLE to
AARCH64_RUNNER_AVAILABLE in those jobs.

Finally mention both variables in our CI documentation, too.

Fixes: c5dd0f0342 ("Improve rules for the staging branch")
Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210730143809.717079-4-thuth@redhat.com>
[AJB: moved due to docu changes]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210806141015.2487502-5-alex.bennee@linaro.org>

show more ...

gitlab-ci.d/custom-runners: Improve rules for the staging branch

If maintainers are currently pushing to a branch called "staging"
in their repository, they are ending up with some stuck jobs - unle

gitlab-ci.d/custom-runners: Improve rules for the staging branch

If maintainers are currently pushing to a branch called "staging"
in their repository, they are ending up with some stuck jobs - unless
they have a s390x CI runner machine available. That's ugly, we should
make sure that the related jobs are really only started if such a
runner is available. So let's only run these jobs if it's the
"staging" branch of the main repository of the QEMU project (where
we can be sure that the s390x runner is available), or if the user
explicitly set a S390X_RUNNER_AVAILABLE variable in their CI configs
to declare that they have such a runner available, too.

Fixes: 4799c21023 ("Jobs based on custom runners: add job definitions ...")
Message-Id: <20210728173857.497523-1-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>

show more ...

Jobs based on custom runners: add job definitions for QEMU's machines

The QEMU project has two machines (aarch64 and s390x) that can be used
for jobs that do build and run tests. This introduces th

Jobs based on custom runners: add job definitions for QEMU's machines

The QEMU project has two machines (aarch64 and s390x) that can be used
for jobs that do build and run tests. This introduces those jobs,
which are a mapping of custom scripts used for the same purpose.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210630012619.115262-5-crosa@redhat.com>
Message-Id: <20210709143005.1554-5-alex.bennee@linaro.org>

show more ...

Jobs based on custom runners: documentation and configuration placeholder

As described in the included documentation, the "custom runner" jobs
extend the GitLab CI jobs already in place. One of the

Jobs based on custom runners: documentation and configuration placeholder

As described in the included documentation, the "custom runner" jobs
extend the GitLab CI jobs already in place. One of their primary
goals of catching and preventing regressions on a wider number of host
systems than the ones provided by GitLab's shared runners.

This sets the stage in which other community members can add their own
machine configuration documentation/scripts, and accompanying job
definitions. As a general rule, those newly added contributed jobs
should run as "non-gating", until their reliability is verified (AKA
"allow_failure: true").

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210630012619.115262-2-crosa@redhat.com>
Message-Id: <20210709143005.1554-2-alex.bennee@linaro.org>

show more ...