1--- 2stage: Verify 3group: Runner 4info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments 5--- 6 7# The Custom executor 8 9> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2885) in GitLab Runner 12.1 10 11GitLab Runner provides the Custom executor for environments that it 12doesn't support natively, for example, Podman or Libvirt. 13 14This gives you the control to create your own executor by configuring 15GitLab Runner to use some executable to provision, run, and clean up 16your environment. 17 18The scripts you configure for the custom executor are called `Drivers`. 19For example, you could create a Podman driver, an [LXD 20driver](custom_examples/lxd.md) or a [Libvirt 21driver](custom_examples/libvirt.md). 22 23## Limitations 24 25Below are some current limitations when using the Custom executor: 26 27- No [Interactive Web 28 Terminal](https://docs.gitlab.com/ee/ci/interactive_web_terminal/) support. 29 30## Configuration 31 32There are a few configuration keys that you can choose from. Some of them are optional. 33 34Below is an example of configuration for the Custom executor using all available 35configuration keys: 36 37```toml 38[[runners]] 39 name = "custom" 40 url = "https://gitlab.com" 41 token = "TOKEN" 42 executor = "custom" 43 builds_dir = "/builds" 44 cache_dir = "/cache" 45 [runners.custom] 46 config_exec = "/path/to/config.sh" 47 config_args = [ "SomeArg" ] 48 config_exec_timeout = 200 49 50 prepare_exec = "/path/to/script.sh" 51 prepare_args = [ "SomeArg" ] 52 prepare_exec_timeout = 200 53 54 run_exec = "/path/to/binary" 55 run_args = [ "SomeArg" ] 56 57 cleanup_exec = "/path/to/executable" 58 cleanup_args = [ "SomeArg" ] 59 cleanup_exec_timeout = 200 60 61 graceful_kill_timeout = 200 62 force_kill_timeout = 200 63``` 64 65For field definitions and which ones are required, see 66[`[runners.custom]` 67section](../configuration/advanced-configuration.md#the-runnerscustom-section) 68configuration. 69 70In addition both `builds_dir` and `cache_dir` inside of the 71[`[[runners]]`](../configuration/advanced-configuration.md#the-runners-section) 72are required fields. 73 74## Prerequisite software for running a Job 75 76The user must set up the environment, including the following that must 77be present in the `PATH`: 78 79- [Git](https://git-scm.com/download): Used to clone the repositories. 80- [Git LFS](https://git-lfs.github.com/): Pulls any LFS objects that 81 might be in the repository. 82- [GitLab Runner](../install/index.md): Used to 83 download/update artifacts and cache. 84 85## Stages 86 87The Custom executor provides the stages for you to configure some 88details of the job, prepare and cleanup the environment and run the job 89script within it. Each stage is responsible for specific things and has 90different things to keep in mind. 91 92Each stage executed by the Custom executor is executed at the time 93a builtin GitLab Runner executor would execute them. 94 95For each step that will be executed, specific environment variables are 96exposed to the executable, which can be used to get information about 97the specific Job that is running. All stages will have the following 98environment variables available to them: 99 100- Standard CI/CD [environment 101 variables](https://docs.gitlab.com/ee/ci/variables/), including 102 [predefined 103 variables](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html). 104- All environment variables provided by the Custom executor Runner host system. 105- All services and their [available settings](https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#available-settings-for-services). 106 Exposed in JSON format as `CUSTOM_ENV_CI_JOB_SERVICES`. 107 108Both CI/CD environment variables and predefined variables are prefixed 109with `CUSTOM_ENV_` to prevent conflicts with system environment 110variables. For example, `CI_BUILDS_DIR` will be available as 111`CUSTOM_ENV_CI_BUILDS_DIR`. 112 113The stages run in the following sequence: 114 1151. `config_exec` 1161. `prepare_exec` 1171. `run_exec` 1181. `cleanup_exec` 119 120### Services 121 122> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4358) in GitLab Runner 13.6 123 124[Services](https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#what-is-a-service) are exposed as a JSON array 125as `CUSTOM_ENV_CI_JOB_SERVICES`. 126 127Example: 128 129```yaml 130custom: 131 script: 132 - echo $CUSTOM_ENV_CI_JOB_SERVICES 133 services: 134 - redis:latest 135 - name: my-postgres:9.4 136 alias: pg 137 entrypoint: ["path", "to", "entrypoint"] 138 command: ["path", "to", "cmd"] 139``` 140 141The example above will set `CUSTOM_ENV_CI_JOB_SERVICES` environment variable with the following value: 142 143```json 144[{"name":"redis:latest","alias":"","entrypoint":null,"command":null},{"name":"my-postgres:9.4","alias":"pg","entrypoint":["path","to","entrypoint"],"command":["path","to","cmd"]}] 145``` 146 147### Config 148 149The Config stage is executed by `config_exec`. 150 151Sometimes you might want to set some settings during execution time. For 152example settings a build directory depending on the project ID. 153`config_exec` reads from STDOUT and expects a valid JSON string with 154specific keys. 155 156For example: 157 158```shell 159#!/usr/bin/env bash 160 161cat << EOS 162{ 163 "builds_dir": "/builds/${CUSTOM_ENV_CI_CONCURRENT_PROJECT_ID}/${CUSTOM_ENV_CI_PROJECT_PATH_SLUG}", 164 "cache_dir": "/cache/${CUSTOM_ENV_CI_CONCURRENT_PROJECT_ID}/${CUSTOM_ENV_CI_PROJECT_PATH_SLUG}", 165 "builds_dir_is_shared": true, 166 "hostname": "custom-hostname", 167 "driver": { 168 "name": "test driver", 169 "version": "v0.0.1" 170 }, 171 "job_env" : { 172 "CUSTOM_ENVIRONMENT": "example" 173 } 174} 175EOS 176``` 177 178Any additional keys inside of the JSON string will be ignored. If it's 179not a valid JSON string the stage will fail and be retried two more 180times. 181 182| Parameter | Type | Required | Allowed empty | Description | 183|-----------|------|----------|---------------|-------------| 184| `builds_dir` | string | ✗ | ✗ | The base directory where the working directory of the job will be created. | 185| `cache_dir` | string | ✗ | ✗ | The base directory where local cache will be stored. | 186| `builds_dir_is_shared` | bool | ✗ | n/a | Defines whether the environment is shared between concurrent job or not. | 187| `hostname` | string | ✗ | ✓ | The hostname to associate with job's "metadata" stored by the runner. If undefined, the hostname is not set. | 188| `driver.name` | string | ✗ | ✓ | The user-defined name for the driver. Printed with the `Using custom executor...` line. If undefined, no information about driver is printed. | 189| `driver.version` | string | ✗ | ✓ | The user-defined version for the drive. Printed with the `Using custom executor...` line. If undefined, only the name information is printed. | 190| `job_env` | object | ✗ | ✓ | Name-value pairs that will be made available through environment variables to all subsequent stages of the job. | 191 192The `STDERR` of the executable will print to the job log. 193 194The user can set 195[`config_exec_timeout`](../configuration/advanced-configuration.md#the-runnerscustom-section) 196if they want to set a deadline for how long GitLab Runner should wait to 197return the JSON string before terminating the process. 198 199If any of the 200[`config_exec_args`](../configuration/advanced-configuration.md#the-runnerscustom-section) 201are defined, these will be added in order to the executable defined in 202`config_exec`. For example we have the `config.toml` content below: 203 204```toml 205... 206[runners.custom] 207 ... 208 config_exec = "/path/to/config" 209 config_args = [ "Arg1", "Arg2" ] 210 ... 211``` 212 213GitLab Runner would execute it as `/path/to/config Arg1 Arg2`. 214 215### Prepare 216 217The Prepare stage is executed by `prepare_exec`. 218 219At this point, GitLab Runner knows everything about the job (where and 220how it's going to run). The only thing left is for the environment to be 221set up so the job can run. GitLab Runner will execute the executable 222that is specified in `prepare_exec`. 223 224This is responsible for setting up the environment (for example, 225creating the virtual machine or container, services or anything else). After 226this is done, we expect that the environment is ready to run the job. 227 228This stage is executed only once, in a job execution. 229 230The user can set 231[`prepare_exec_timeout`](../configuration/advanced-configuration.md#the-runnerscustom-section) 232if they want to set a deadline for how long GitLab Runner 233should wait to prepare the environment before terminating the process. 234 235The `STDOUT` and `STDERR` returned from this executable will print to 236the job log. 237 238If any of the 239[`prepare_exec_args`](../configuration/advanced-configuration.md#the-runnerscustom-section) 240are defined, these will be added in order to the executable defined in 241`prepare_exec`. For example we have the `config.toml` content below: 242 243```toml 244... 245[runners.custom] 246 ... 247 prepare_exec = "/path/to/bin" 248 prepare_args = [ "Arg1", "Arg2" ] 249 ... 250``` 251 252GitLab Runner would execute it as `/path/to/bin Arg1 Arg2`. 253 254### Run 255 256The Run stage is executed by `run_exec`. 257 258The `STDOUT` and `STDERR` returned from this executable will print to 259the job log. 260 261Unlike the other stages, the `run_exec` stage is executed multiple 262times, since it's split into sub stages listed below in sequential 263order: 264 2651. `prepare_script` 2661. `get_sources` 2671. `restore_cache` 2681. `download_artifacts` 2691. `step_*` 2701. `build_script` 2711. `step_*` 2721. `after_script` 2731. `archive_cache` OR `archive_cache_on_failure` 2741. `upload_artifacts_on_success` OR `upload_artifacts_on_failure` 2751. `cleanup_file_variables` 276 277NOTE: 278In GitLab Runner 14.0 and later, `build_script` will be replaced with `step_script`. For more information, see [this issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26426). 279 280For each stage mentioned above, the `run_exec` executable will be 281executed with: 282 283- The usual environment variables. 284- Two arguments: 285 - The path to the script that GitLab Runner creates for the Custom 286 executor to run. 287 - Name of the stage. 288 289For example: 290 291```shell 292/path/to/run_exec.sh /path/to/tmp/script1 prepare_executor 293/path/to/run_exec.sh /path/to/tmp/script1 prepare_script 294/path/to/run_exec.sh /path/to/tmp/script1 get_sources 295``` 296 297If you have `run_args` defined, they are the first set of arguments 298passed to the `run_exec` executable, then GitLab Runner adds others. For 299example, suppose we have the following `config.toml`: 300 301```toml 302... 303[runners.custom] 304 ... 305 run_exec = "/path/to/run_exec.sh" 306 run_args = [ "Arg1", "Arg2" ] 307 ... 308``` 309 310GitLab Runner will execute the executable with the following arguments: 311 312```shell 313/path/to/run_exec.sh Arg1 Arg2 /path/to/tmp/script1 prepare_executor 314/path/to/run_exec.sh Arg1 Arg2 /path/to/tmp/script1 prepare_script 315/path/to/run_exec.sh Arg1 Arg2 /path/to/tmp/script1 get_sources 316``` 317 318This executable should be responsible for executing the scripts that are 319specified in the first argument. They contain all the scripts any GitLab 320Runner executor would run normally to clone, download artifacts, run 321user scripts and all the other steps described below. The scripts can be 322of the following shells: 323 324- Bash 325- PowerShell Desktop 326- PowerShell Core 327- Batch (deprecated) 328 329We generate the script using the shell configured by `shell` inside of 330[`[[runners]]`](../configuration/advanced-configuration.md#the-runners-section). 331If none is provided the defaults for the OS platform are used. 332 333The table below is a detailed explanation of what each script does and 334what the main goal of that script is. 335 336| Script Name | Script Contents | 337|:-----------:|:---------------:| 338| `prepare_script` | Simple debug information which machine the Job is running on. | 339| `get_sources` | Prepares the Git configuration, and clone/fetch the repository. We suggest you keep this as is since you get all of the benefits of Git strategies that GitLab provides. | 340| `restore_cache` | Extract the cache if any are defined. This expects the `gitlab-runner` binary is available in `$PATH`. | 341| `download_artifacts` | Download artifacts, if any are defined. This expects `gitlab-runner` binary is available in `$PATH`. | 342| `step_*` | Generated by GitLab. A set of scripts to execute. It may never be sent to the custom executor. It may have multiple steps, like `step_release` and `step_accessibility`. This can be a feature from the `.gitlab-ci.yml` file. | 343| `build_script` | A combination of [`before_script`](https://docs.gitlab.com/ee/ci/yaml/#before_script-and-after_script) and [`script`](https://docs.gitlab.com/ee/ci/yaml/#script). In GitLab Runner 14.0 and later, `build_script` will be replaced with `step_script`. For more information, see [this issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26426). | 344| `after_script` | This is the [`after_script`](https://docs.gitlab.com/ee/ci/yaml/#before_script-and-after_script) defined from the job. This is always called even if any of the previous steps failed. | 345| `archive_cache` | Will create an archive of all the cache, if any are defined. Only executed when `build_script` was successful. | 346| `archive_cache_on_failure` | Will create an archive of all the cache, if any are defined. Only executed when `build_script` fails. | 347| `upload_artifacts_on_success` | Upload any artifacts that are defined. Only executed when `build_script` was successful. | 348| `upload_artifacts_on_failure` | Upload any artifacts that are defined. Only executed when `build_script` fails. | 349| `cleanup_file_variables` | Deletes all [file based](https://docs.gitlab.com/ee/ci/variables/#custom-environment-variables-of-type-file) variables from disk. | 350 351### Cleanup 352 353The Cleanup stage is executed by `cleanup_exec`. 354 355This final stage is executed even if one of the previous stages failed. 356The main goal for this stage is to clean up any of the environments that 357might have been set up. For example, turning off VMs or deleting 358containers. 359 360The result of `cleanup_exec` does not affect job statuses. For example, 361a job will be marked as successful even if the following occurs: 362 363- Both `prepare_exec` and `run_exec` are successful. 364- `cleanup_exec` fails. 365 366The user can set 367[`cleanup_exec_timeout`](../configuration/advanced-configuration.md#the-runnerscustom-section) 368if they want to set some kind of deadline of how long GitLab Runner 369should wait to clean up the environment before terminating the 370process. 371 372The `STDOUT` of this executable will be printed to GitLab Runner logs at a 373DEBUG level. The `STDERR` will be printed to the logs at a WARN level. 374 375If any of the 376[`cleanup_exec_args`](../configuration/advanced-configuration.md#the-runnerscustom-section) 377are defined, these will be added in order to the executable defined in 378`cleanup_exec`. For example we have the `config.toml` content below: 379 380```toml 381... 382[runners.custom] 383 ... 384 cleanup_exec = "/path/to/bin" 385 cleanup_args = [ "Arg1", "Arg2" ] 386 ... 387``` 388 389GitLab Runner would execute it as `/path/to/bin Arg1 Arg2`. 390 391## Terminating and killing executables 392 393GitLab Runner will try to gracefully terminate an executable under any 394of the following conditions: 395 396- `config_exec_timeout`, `prepare_exec_timeout` or `cleanup_exec_timeout` are met. 397- The job [times out](https://docs.gitlab.com/ee/ci/pipelines/settings.html#timeout). 398- The job is cancelled. 399 400When a timeout is reached, a `SIGTERM` is sent to the executable, and 401the countdown for 402[`exec_terminate_timeout`](../configuration/advanced-configuration.md#the-runnerscustom-section) 403starts. The executable should listen to this signal to make sure it 404cleans up any resources. If `exec_terminate_timeout` passes and the 405process is still running, a `SIGKILL` is sent to kill the process and 406[`exec_force_kill_timeout`](../configuration/advanced-configuration.md#the-runnerscustom-section) 407will start. If the process is still running after 408`exec_force_kill_timeout` has finished, GitLab Runner will abandon the 409process and will not try to stop/kill anymore. If both these timeouts 410are reached during `config_exec`, `prepare_exec` or `run_exec` the build 411is marked as failed. 412 413As of [GitLab 41413.1](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1743) 415any child process that is spawned by the driver will also receive the 416graceful termination process explained above on UNIX based systems. This 417is achieved by having the main process set as a [process 418group](https://man7.org/linux/man-pages/man2/setpgid.2.html) 419which all the child processes belong too. 420 421## Error handling 422 423There are two types of errors that GitLab Runner can handle differently. 424These errors are only handled when the executable inside of 425`config_exec`, `prepare_exec`, `run_exec`, and `cleanup_exec` exits with 426these codes. If the user exits with a non-zero exit code, it should be 427propagated as one of the error codes below. 428 429If the user script exits with one of these code it has to 430be propagated to the executable exit code. 431 432### Build Failure 433 434GitLab Runner provides `BUILD_FAILURE_EXIT_CODE` environment 435variable which should be used by the executable as an exit code to 436inform GitLab Runner that there is a failure on the users job. If the 437executable exits with the code from 438`BUILD_FAILURE_EXIT_CODE`, the build is marked as a failure 439appropriately in GitLab CI. 440 441If the script that the user defines inside of `.gitlab-ci.yml` file 442exits with a non-zero code, `run_exec` should exit with 443`BUILD_FAILURE_EXIT_CODE` value. 444 445NOTE: 446We strongly suggest using `BUILD_FAILURE_EXIT_CODE` to exit 447instead of a hard coded value since it can change in any release, making 448your binary/script future proof. 449 450### System Failure 451 452You can send a system failure to GitLab Runner by exiting the process with the 453error code specified in the `SYSTEM_FAILURE_EXIT_CODE`. If this error 454code is returned, on certain stages GitLab Runner will retry the stage, if none 455of the retries are successful the job will be marked as failed. 456 457Below is a table of what stages are retried, and by how many times. 458 459| Stage Name | Number of attempts | Duration to wait between each retry | 460|----------------------|-------------------------------------------------------------|-------------------------------------| 461| `prepare_exec` | 3 | 3 seconds | 462| `get_sources` | Value of `GET_SOURCES_ATTEMPTS` variable. (Default 1) | 0 seconds | 463| `restore_cache` | Value of `RESTORE_CACHE_ATTEMPTS` variable. (Default 1) | 0 seconds | 464| `download_artifacts` | Value of `ARTIFACT_DOWNLOAD_ATTEMPTS` variable. (Default 1) | 0 seconds | 465 466NOTE: 467We strongly suggest using `SYSTEM_FAILURE_EXIT_CODE` to exit 468instead of a hard coded value since it can change in any release, making 469your binary/script future proof. 470 471## Job response 472 473You can change job-level `CUSTOM_ENV_` variables as they observe the documented 474[CI/CD variable precedence](https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence). 475Though this functionality can be desirable, when the trusted job context 476is required, the full JSON job response is provided automatically. The runner 477generates a temporary file, which is referenced in the `JOB_RESPONSE_FILE` 478environment variable. This file exists in every stage and is automatically 479removed during cleanup. 480 481```shell 482$ cat ${JOB_RESPONSE_FILE} 483{"id": 123456, "token": "jobT0ken",...} 484``` 485