1--- 2stage: Configure 3group: Configure 4info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments 5--- 6 7# Customizing Auto DevOps **(FREE)** 8 9While [Auto DevOps](index.md) provides great defaults to get you started, you can customize 10almost everything to fit your needs. Auto DevOps offers everything from custom 11[buildpacks](#custom-buildpacks), to [Dockerfiles](#custom-dockerfile), and 12[Helm charts](#custom-helm-chart). You can even copy the complete 13[CI/CD configuration](#customizing-gitlab-ciyml) into your project to enable 14staging and canary deployments, 15[manage Auto DevOps with GitLab APIs](customize.md#extend-auto-devops-with-the-api), and more. 16 17## Custom buildpacks 18 19If the automatic buildpack detection fails for your project, or if you 20need more control over your build, you can customize the buildpacks 21used for the build. 22 23### Custom buildpacks with Cloud Native Buildpacks 24 25> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28165) in GitLab 12.10. 26 27Specify either: 28 29- The CI/CD variable `BUILDPACK_URL` according to [`pack`'s specifications](https://buildpacks.io/docs/app-developer-guide/specific-buildpacks/). 30- A [`project.toml` project descriptor](https://buildpacks.io/docs/app-developer-guide/using-project-descriptor/) with the buildpacks you would like to include. 31 32### Custom buildpacks with Herokuish 33 34Specify either: 35 36- The CI/CD variable `BUILDPACK_URL`. 37- A `.buildpacks` file at the root of your project, containing one buildpack URL per line. 38 39The buildpack URL can point to either a Git repository URL or a tarball URL. 40For Git repositories, you can point to a specific Git reference (such as 41commit SHA, tag name, or branch name) by appending `#<ref>` to the Git repository URL. 42For example: 43 44- The tag `v142`: `https://github.com/heroku/heroku-buildpack-ruby.git#v142`. 45- The branch `mybranch`: `https://github.com/heroku/heroku-buildpack-ruby.git#mybranch`. 46- The commit SHA `f97d8a8ab49`: `https://github.com/heroku/heroku-buildpack-ruby.git#f97d8a8ab49`. 47 48### Multiple buildpacks 49 50Using multiple buildpacks is not fully supported by Auto DevOps, because Auto Test 51can't use the `.buildpacks` file. The buildpack 52[heroku-buildpack-multi](https://github.com/heroku/heroku-buildpack-multi/), used 53in the backend to parse the `.buildpacks` file, does not provide the necessary commands 54`bin/test-compile` and `bin/test`. 55 56If your goal is to use only a single custom buildpack, you should provide the project CI/CD variable 57`BUILDPACK_URL` instead. 58 59## Custom `Dockerfile` 60 61> Support for `DOCKERFILE_PATH` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35662) in GitLab 13.2 62 63If your project has a `Dockerfile` in the root of the project repository, Auto DevOps 64builds a Docker image based on the Dockerfile, rather than using buildpacks. 65This can be much faster and result in smaller images, especially if your 66Dockerfile is based on [Alpine](https://hub.docker.com/_/alpine/). 67 68If you set the `DOCKERFILE_PATH` CI/CD variable, Auto Build looks for a Dockerfile there 69instead. 70 71## Passing arguments to `docker build` 72 73Arguments can be passed to the `docker build` command using the 74`AUTO_DEVOPS_BUILD_IMAGE_EXTRA_ARGS` project CI/CD variable. For example, to build a 75Docker image based on based on the `ruby:alpine` instead of the default `ruby:latest`: 76 771. Set `AUTO_DEVOPS_BUILD_IMAGE_EXTRA_ARGS` to `--build-arg=RUBY_VERSION=alpine`. 781. Add the following to a custom `Dockerfile`: 79 80 ```dockerfile 81 ARG RUBY_VERSION=latest 82 FROM ruby:$RUBY_VERSION 83 84 # ... put your stuff here 85 ``` 86 87Use Base64 encoding if you need to pass complex values, such as newlines and 88spaces. Left unencoded, complex values like these can cause escaping issues 89due to how Auto DevOps uses the arguments. 90 91WARNING: 92Avoid passing secrets as Docker build arguments if possible, as they may be 93persisted in your image. See 94[this discussion of best practices with secrets](https://github.com/moby/moby/issues/13490) for details. 95 96## Custom container image 97 98By default, [Auto Deploy](stages.md#auto-deploy) deploys a container image built and pushed to the GitLab registry by [Auto Build](stages.md#auto-build). 99You can override this behavior by defining specific variables: 100 101| Entry | Default | Can be overridden by | 102| ----- | ----- | ----- | 103| Image Path | `$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG` for branch pipelines. `$CI_REGISTRY_IMAGE` for tag pipelines. | `$CI_APPLICATION_REPOSITORY` | 104| Image Tag | `$CI_COMMIT_SHA` for branch pipelines. `$CI_COMMIT_TAG` for tag pipelines. | `$CI_APPLICATION_TAG` | 105 106These variables also affect Auto Build and Auto Container Scanning. If you don't want to build and push an image to 107`$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG`, consider 108including only `Jobs/Deploy.gitlab-ci.yml`, or [disabling the `build` jobs](#disable-jobs). 109 110If you use Auto Container Scanning and set a value for `$CI_APPLICATION_REPOSITORY`, then you should 111also update `$CS_DEFAULT_BRANCH_IMAGE`. See [Setting the default branch image](../../user/application_security/container_scanning/index.md#setting-the-default-branch-image) 112for more details. 113 114Here is an example setup in your `.gitlab-ci.yml`: 115 116```yaml 117variables: 118 CI_APPLICATION_REPOSITORY: <your-image-repository> 119 CI_APPLICATION_TAG: <the-tag> 120``` 121 122## Extend Auto DevOps with the API 123 124You can extend and manage your Auto DevOps configuration with GitLab APIs: 125 126- [Settings that can be accessed with API calls](../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls), 127 which include `auto_devops_enabled`, to enable Auto DevOps on projects by default. 128- [Creating a new project](../../api/projects.md#create-project). 129- [Editing groups](../../api/groups.md#update-group). 130- [Editing projects](../../api/projects.md#edit-project). 131 132## Forward CI/CD variables to the build environment 133 134> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25514) in GitLab 12.3, but available in versions 11.9 and above. 135 136CI/CD variables can be forwarded into the build environment using the 137`AUTO_DEVOPS_BUILD_IMAGE_FORWARDED_CI_VARIABLES` CI/CD variable. 138The forwarded variables should be specified by name in a comma-separated 139list. For example, to forward the variables `CI_COMMIT_SHA` and 140`CI_ENVIRONMENT_NAME`, set `AUTO_DEVOPS_BUILD_IMAGE_FORWARDED_CI_VARIABLES` 141to `CI_COMMIT_SHA,CI_ENVIRONMENT_NAME`. 142 143- When using Buildpacks, the forwarded variables are available automatically 144 as environment variables. 145- When using a `Dockerfile`, the following additional steps are required: 146 147 1. Activate the experimental `Dockerfile` syntax by adding the following code 148 to the top of the file: 149 150 ```dockerfile 151 # syntax = docker/dockerfile:experimental 152 ``` 153 154 1. To make secrets available in any `RUN $COMMAND` in the `Dockerfile`, mount 155 the secret file and source it prior to running `$COMMAND`: 156 157 ```dockerfile 158 RUN --mount=type=secret,id=auto-devops-build-secrets . /run/secrets/auto-devops-build-secrets && $COMMAND 159 ``` 160 161When `AUTO_DEVOPS_BUILD_IMAGE_FORWARDED_CI_VARIABLES` is set, Auto DevOps 162enables the experimental [Docker BuildKit](https://docs.docker.com/develop/develop-images/build_enhancements/) 163feature to use the `--secret` flag. 164 165## Custom Helm Chart 166 167Auto DevOps uses [Helm](https://helm.sh/) to deploy your application to Kubernetes. 168You can override the Helm chart used by bundling up a chart into your project 169repository or by specifying a project CI/CD variable: 170 171- **Bundled chart** - If your project has a `./chart` directory with a `Chart.yaml` 172 file in it, Auto DevOps detects the chart and uses it instead of the 173 [default chart](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/tree/master/assets/auto-deploy-app), enabling 174 you to control exactly how your application is deployed. 175- **Project variable** - Create a [project CI/CD variable](../../ci/variables/index.md) 176 `AUTO_DEVOPS_CHART` with the URL of a custom chart to use, or create two project 177 variables: `AUTO_DEVOPS_CHART_REPOSITORY` with the URL of a custom chart repository, 178 and `AUTO_DEVOPS_CHART` with the path to the chart. 179 180## Customize values for Helm Chart 181 182> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30628) in GitLab 12.6, `.gitlab/auto-deploy-values.yaml` is used by default for Helm upgrades. 183 184You can override the default values in the `values.yaml` file in the 185[default Helm chart](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/tree/master/assets/auto-deploy-app) by either: 186 187- Adding a file named `.gitlab/auto-deploy-values.yaml` to your repository, which is 188 automatically used, if found. 189- Adding a file with a different name or path to the repository, and setting the 190 `HELM_UPGRADE_VALUES_FILE` [CI/CD variable](#cicd-variables) with 191 the path and name. 192 193NOTE: 194For GitLab 12.5 and earlier, use the `HELM_UPGRADE_EXTRA_ARGS` variable 195to override the default chart values by setting `HELM_UPGRADE_EXTRA_ARGS` to `--values <my-values.yaml>`. 196 197## Customize the `helm upgrade` command 198 199You can customize the `helm upgrade` command used in the [auto-deploy-image](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image) 200by passing options to the command with the `HELM_UPGRADE_EXTRA_ARGS` CI/CD variable. 201For example, set the value of `HELM_UPGRADE_EXTRA_ARGS` to `--no-hooks` to disable 202pre-upgrade and post-upgrade hooks when the command is executed. 203 204See [the official documentation](https://helm.sh/docs/helm/helm_upgrade/) for the full 205list of options. 206 207## Custom Helm chart per environment 208 209You can specify the use of a custom Helm chart per environment by scoping the CI/CD variable 210to the desired environment. See [Limit environment scope of CI/CD variables](../../ci/variables/index.md#limit-the-environment-scope-of-a-cicd-variable). 211 212## Customizing `.gitlab-ci.yml` 213 214Auto DevOps is completely customizable because the 215[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) 216is just an implementation of a [`.gitlab-ci.yml`](../../ci/yaml/index.md) file, 217and uses only features available to any implementation of `.gitlab-ci.yml`. 218 219To modify the CI/CD pipeline used by Auto DevOps, 220[`include` the template](../../ci/yaml/index.md#includetemplate), and customize 221it as needed by adding a `.gitlab-ci.yml` file to the root of your repository 222containing the following: 223 224```yaml 225include: 226 - template: Auto-DevOps.gitlab-ci.yml 227``` 228 229Add your changes, and your additions are merged with the 230[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) 231using the behavior described for [`include`](../../ci/yaml/index.md#include). 232 233If you need to specifically remove a part of the file, you can also copy and paste the contents of the 234[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) 235into your project and edit it as needed. 236 237## Use multiple Kubernetes clusters 238 239See [Multiple Kubernetes clusters for Auto DevOps](multiple_clusters_auto_devops.md). 240 241## Customizing the Kubernetes namespace 242 243> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27630) in GitLab 12.6. 244 245For clusters not managed by GitLab, you can customize the namespace in 246`.gitlab-ci.yml` by specifying 247[`environment:kubernetes:namespace`](../../ci/environments/index.md#configure-kubernetes-deployments-deprecated). 248For example, the following configuration overrides the namespace used for 249`production` deployments: 250 251```yaml 252include: 253 - template: Auto-DevOps.gitlab-ci.yml 254 255production: 256 environment: 257 kubernetes: 258 namespace: production 259``` 260 261When deploying to a custom namespace with Auto DevOps, the service account 262provided with the cluster needs at least the `edit` role within the namespace. 263 264- If the service account can create namespaces, then the namespace can be created on-demand. 265- Otherwise, the namespace must exist prior to deployment. 266 267## Using components of Auto DevOps 268 269If you only require a subset of the features offered by Auto DevOps, you can include 270individual Auto DevOps jobs into your own `.gitlab-ci.yml`. Each component job relies 271on a stage that should be defined in the `.gitlab-ci.yml` that includes the template. 272 273For example, to make use of [Auto Build](stages.md#auto-build), you can add the following to 274your `.gitlab-ci.yml`: 275 276```yaml 277stages: 278 - build 279 280include: 281 - template: Jobs/Build.gitlab-ci.yml 282``` 283 284See the [Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) for information on available jobs. 285 286WARNING: 287Auto DevOps templates using the [`only`](../../ci/yaml/index.md#only--except) or 288[`except`](../../ci/yaml/index.md#only--except) syntax have switched 289to the [`rules`](../../ci/yaml/index.md#rules) syntax, starting in 290[GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/213336). 291If your `.gitlab-ci.yml` extends these Auto DevOps templates and override the `only` or 292`except` keywords, you must migrate your templates to use the 293[`rules`](../../ci/yaml/index.md#rules) syntax after the 294base template is migrated to use the `rules` syntax. 295For users who cannot migrate just yet, you can alternatively pin your templates to 296the [GitLab 12.10 based templates](https://gitlab.com/gitlab-org/auto-devops-v12-10). 297 298## Use images hosted in a local Docker registry 299 300You can configure many Auto DevOps jobs to run in an [offline environment](../../user/application_security/offline_deployments/index.md): 301 3021. Copy the required Auto DevOps Docker images from Docker Hub and `registry.gitlab.com` to their local GitLab container registry. 3031. After the images are hosted and available in a local registry, edit `.gitlab-ci.yml` to point to the locally-hosted images. For example: 304 305 ```yaml 306 include: 307 - template: Auto-DevOps.gitlab-ci.yml 308 309 variables: 310 REGISTRY_URL: "registry.gitlab.example" 311 312 build: 313 image: "$REGISTRY_URL/docker/auto-build-image:v0.6.0" 314 services: 315 - name: "$REGISTRY_URL/greg/docker/docker:20.10.6-dind" 316 command: ['--tls=false', '--host=tcp://0.0.0.0:2375'] 317 ``` 318 319## PostgreSQL database support 320 321To support applications requiring a database, 322[PostgreSQL](https://www.postgresql.org/) is provisioned by default. The credentials to access 323the database are preconfigured, but can be customized by setting the associated 324[CI/CD variables](#cicd-variables). You can use these credentials to define a `DATABASE_URL`: 325 326```yaml 327postgres://user:password@postgres-host:postgres-port/postgres-database 328``` 329 330### Upgrading PostgresSQL 331 332WARNING: 333The CI/CD variable `AUTO_DEVOPS_POSTGRES_CHANNEL` that controls default provisioned 334PostgreSQL was changed to `2` in [GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/210499). 335To keep using the old PostgreSQL, set the `AUTO_DEVOPS_POSTGRES_CHANNEL` variable to 336`1`. 337 338The version of the chart used to provision PostgreSQL: 339 340- Is 8.2.1 in GitLab 13.0 and later, but can be set back to 0.7.1 if needed. 341- Can be set to from 0.7.1 to 8.2.1 in GitLab 12.9 and 12.10. 342- Is 0.7.1 in GitLab 12.8 and earlier. 343 344GitLab encourages users to [migrate their database](upgrading_postgresql.md) 345to the newer PostgreSQL. 346 347### Customize values for PostgreSQL Helm Chart 348 349> [Introduced](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/issues/113) in auto-deploy-image v2, in GitLab 13.8. 350 351To set custom values, do one of the following: 352 353- Add a file named `.gitlab/auto-deploy-postgres-values.yaml` to your repository. If found, this 354 file is used automatically. This file is used by default for PostgreSQL Helm upgrades. 355- Add a file with a different name or path to the repository, and set the 356 `POSTGRES_HELM_UPGRADE_VALUES_FILE` [environment variable](#database) with the path 357 and name. 358- Set the `POSTGRES_HELM_UPGRADE_EXTRA_ARGS` [environment variable](#database). 359 360### Using external PostgreSQL database providers 361 362While Auto DevOps provides out-of-the-box support for a PostgreSQL container for 363production environments, for some use cases, it may not be sufficiently secure or 364resilient, and you may want to use an external managed provider (such as 365AWS Relational Database Service) for PostgreSQL. 366 367You must define environment-scoped CI/CD variables for `POSTGRES_ENABLED` and 368`DATABASE_URL` in your project's CI/CD settings: 369 3701. Disable the built-in PostgreSQL installation for the required environments using 371 environment-scoped [CI/CD variables](../../ci/environments/index.md#scope-environments-with-specs). 372 For this use case, it's likely that only `production` must be added to this 373 list. The built-in PostgreSQL setup for Review Apps and staging is sufficient. 374 375 ![Auto Metrics](img/disable_postgres.png) 376 3771. Define the `DATABASE_URL` variable as an environment-scoped variable that is 378 available to your application. This should be a URL in the following format: 379 380 ```yaml 381 postgres://user:password@postgres-host:postgres-port/postgres-database 382 ``` 383 384You must ensure that your Kubernetes cluster has network access to wherever 385PostgreSQL is hosted. 386 387## CI/CD variables 388 389The following variables can be used for setting up the Auto DevOps domain, 390providing a custom Helm chart, or scaling your application. PostgreSQL can 391also be customized, and you can use a [custom buildpack](#custom-buildpacks). 392 393### Build and deployment 394 395The following table lists CI/CD variables related to building and deploying 396applications. 397 398| **CI/CD Variable** | **Description** | 399|-----------------------------------------|------------------------------------| 400| `ADDITIONAL_HOSTS` | Fully qualified domain names specified as a comma-separated list that are added to the Ingress hosts. | 401| `<ENVIRONMENT>_ADDITIONAL_HOSTS` | For a specific environment, the fully qualified domain names specified as a comma-separated list that are added to the Ingress hosts. This takes precedence over `ADDITIONAL_HOSTS`. | 402| `AUTO_BUILD_IMAGE_VERSION` | Customize the image version used for the `build` job. See [list of versions](https://gitlab.com/gitlab-org/cluster-integration/auto-build-image/-/releases). | 403| `AUTO_DEPLOY_IMAGE_VERSION` | Customize the image version used for Kubernetes deployment jobs. See [list of versions](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/releases). | 404| `AUTO_DEVOPS_ATOMIC_RELEASE` | As of GitLab 13.0, Auto DevOps uses [`--atomic`](https://v2.helm.sh/docs/helm/#options-43) for Helm deployments by default. Set this variable to `false` to disable the use of `--atomic` | 405| `AUTO_DEVOPS_BUILD_IMAGE_CNB_ENABLED` | Set to `false` to use Herokuish instead of Cloud Native Buildpacks with Auto Build. [More details](stages.md#auto-build-using-cloud-native-buildpacks). | 406| `AUTO_DEVOPS_BUILD_IMAGE_CNB_BUILDER` | The builder used when building with Cloud Native Buildpacks. The default builder is `heroku/buildpacks:18`. [More details](stages.md#auto-build-using-cloud-native-buildpacks). | 407| `AUTO_DEVOPS_BUILD_IMAGE_EXTRA_ARGS` | Extra arguments to be passed to the `docker build` command. Note that using quotes doesn't prevent word splitting. [More details](#passing-arguments-to-docker-build). | 408| `AUTO_DEVOPS_BUILD_IMAGE_FORWARDED_CI_VARIABLES` | A [comma-separated list of CI/CD variable names](#forward-cicd-variables-to-the-build-environment) to be forwarded to the build environment (the buildpack builder or `docker build`). | 409| `AUTO_DEVOPS_CHART` | Helm Chart used to deploy your apps. Defaults to the one [provided by GitLab](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/tree/master/assets/auto-deploy-app). | 410| `AUTO_DEVOPS_CHART_REPOSITORY` | Helm Chart repository used to search for charts. Defaults to `https://charts.gitlab.io`. | 411| `AUTO_DEVOPS_CHART_REPOSITORY_NAME` | From GitLab 11.11, used to set the name of the Helm repository. Defaults to `gitlab`. | 412| `AUTO_DEVOPS_CHART_REPOSITORY_USERNAME` | From GitLab 11.11, used to set a username to connect to the Helm repository. Defaults to no credentials. Also set `AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD`. | 413| `AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD` | From GitLab 11.11, used to set a password to connect to the Helm repository. Defaults to no credentials. Also set `AUTO_DEVOPS_CHART_REPOSITORY_USERNAME`. | 414| `AUTO_DEVOPS_CHART_REPOSITORY_PASS_CREDENTIALS` | From GitLab 14.2, set to a non-empty value to enable forwarding of the Helm repository credentials to the chart server when the chart artifacts are on a different host than repository. | 415| `AUTO_DEVOPS_DEPLOY_DEBUG` | From GitLab 13.1, if this variable is present, Helm outputs debug logs. | 416| `AUTO_DEVOPS_ALLOW_TO_FORCE_DEPLOY_V<N>` | From [auto-deploy-image](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image) v1.0.0, if this variable is present, a new major version of chart is forcibly deployed. For more information, see [Ignore warnings and continue deploying](upgrading_auto_deploy_dependencies.md#ignore-warnings-and-continue-deploying). | 417| `BUILDPACK_URL` | Buildpack's full URL. [Must point to a URL supported by Pack or Herokuish](#custom-buildpacks). | 418| `CANARY_ENABLED` | From GitLab 11.0, used to define a [deploy policy for canary environments](#deploy-policy-for-canary-environments). | 419| `CANARY_PRODUCTION_REPLICAS` | Number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md) in the production environment. Takes precedence over `CANARY_REPLICAS`. Defaults to 1. | 420| `CANARY_REPLICAS` | Number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md). Defaults to 1. | 421| `CI_APPLICATION_REPOSITORY` | The repository of container image being built or deployed, `$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG`. For more details, read [Custom container image](#custom-container-image). | 422| `CI_APPLICATION_TAG` | The tag of the container image being built or deployed, `$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG`. For more details, read [Custom container image](#custom-container-image). | 423| `DAST_AUTO_DEPLOY_IMAGE_VERSION` | Customize the image version used for DAST deployments on the default branch. Should usually be the same as `AUTO_DEPLOY_IMAGE_VERSION`. See [list of versions](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/releases). | 424| `DOCKERFILE_PATH` | From GitLab 13.2, allows overriding the [default Dockerfile path for the build stage](#custom-dockerfile) | 425| `HELM_RELEASE_NAME` | From GitLab 12.1, allows the `helm` release name to be overridden. Can be used to assign unique release names when deploying multiple projects to a single namespace. | 426| `HELM_UPGRADE_VALUES_FILE` | From GitLab 12.6, allows the `helm upgrade` values file to be overridden. Defaults to `.gitlab/auto-deploy-values.yaml`. | 427| `HELM_UPGRADE_EXTRA_ARGS` | From GitLab 11.11, allows extra options in `helm upgrade` commands when deploying the application. Note that using quotes doesn't prevent word splitting. | 428| `INCREMENTAL_ROLLOUT_MODE` | From GitLab 11.4, if present, can be used to enable an [incremental rollout](#incremental-rollout-to-production) of your application for the production environment. Set to `manual` for manual deployment jobs or `timed` for automatic rollout deployments with a 5 minute delay each one. | 429| `K8S_SECRET_*` | From GitLab 11.7, any variable prefixed with [`K8S_SECRET_`](#application-secret-variables) is made available by Auto DevOps as environment variables to the deployed application. | 430| `KUBE_CONTEXT` | From GitLab 14.5, can be used to select which context to use from `KUBECONFIG`. When `KUBE_CONTEXT` is blank, the default context in `KUBECONFIG` (if any) will be used. A context must be selected when using the [CI/CD tunnel](../../user/clusters/agent/ci_cd_tunnel.md). | 431| `KUBE_INGRESS_BASE_DOMAIN` | From GitLab 11.8, can be used to set a domain per cluster. See [cluster domains](../../user/project/clusters/gitlab_managed_clusters.md#base-domain) for more information. | 432| `KUBE_NAMESPACE` | The namespace used for deployments. When using certificate-based clusters, [this value should not be overwritten directly](../../user/project/clusters/deploy_to_cluster.md#custom-namespace). | 433| `KUBECONFIG` | The kubeconfig to use for deployments. User-provided values take priority over GitLab-provided values. | 434| `PRODUCTION_REPLICAS` | Number of replicas to deploy in the production environment. Takes precedence over `REPLICAS` and defaults to 1. For zero downtime upgrades, set to 2 or greater. | 435| `REPLICAS` | Number of replicas to deploy. Defaults to 1. | 436| `ROLLOUT_RESOURCE_TYPE` | From GitLab 11.9, allows specification of the resource type being deployed when using a custom Helm chart. Default value is `deployment`. | 437| `ROLLOUT_STATUS_DISABLED` | From GitLab 12.0, used to disable rollout status check because it does not support all resource types, for example, `cronjob`. | 438| `STAGING_ENABLED` | From GitLab 10.8, used to define a [deploy policy for staging and production environments](#deploy-policy-for-staging-and-production-environments). | 439 440NOTE: 441After you set up your replica variables using a 442[project CI/CD variable](../../ci/variables/index.md), 443you can scale your application by redeploying it. 444 445WARNING: 446You should *not* scale your application using Kubernetes directly. This can 447cause confusion with Helm not detecting the change, and subsequent deploys with 448Auto DevOps can undo your changes. 449 450### Database 451 452The following table lists CI/CD variables related to the database. 453 454| **CI/CD Variable** | **Description** | 455|-----------------------------------------|------------------------------------| 456| `DB_INITIALIZE` | From GitLab 11.4, used to specify the command to run to initialize the application's PostgreSQL database. Runs inside the application pod. | 457| `DB_MIGRATE` | From GitLab 11.4, used to specify the command to run to migrate the application's PostgreSQL database. Runs inside the application pod. | 458| `POSTGRES_ENABLED` | Whether PostgreSQL is enabled. Defaults to `true`. Set to `false` to disable the automatic deployment of PostgreSQL. | 459| `POSTGRES_USER` | The PostgreSQL user. Defaults to `user`. Set it to use a custom username. | 460| `POSTGRES_PASSWORD` | The PostgreSQL password. Defaults to `testing-password`. Set it to use a custom password. | 461| `POSTGRES_DB` | The PostgreSQL database name. Defaults to the value of [`$CI_ENVIRONMENT_SLUG`](../../ci/variables/index.md#predefined-cicd-variables). Set it to use a custom database name. | 462| `POSTGRES_VERSION` | Tag for the [`postgres` Docker image](https://hub.docker.com/_/postgres) to use. Defaults to `9.6.16` for tests and deployments as of GitLab 13.0 (previously `9.6.2`). If `AUTO_DEVOPS_POSTGRES_CHANNEL` is set to `1`, deployments uses the default version `9.6.2`. | 463| `POSTGRES_HELM_UPGRADE_VALUES_FILE` | In GitLab 13.8 and later, and when using [auto-deploy-image v2](upgrading_auto_deploy_dependencies.md), this variable allows the `helm upgrade` values file for PostgreSQL to be overridden. Defaults to `.gitlab/auto-deploy-postgres-values.yaml`. | 464| `POSTGRES_HELM_UPGRADE_EXTRA_ARGS` | In GitLab 13.8 and later, and when using [auto-deploy-image v2](upgrading_auto_deploy_dependencies.md), this variable allows extra PostgreSQL options in `helm upgrade` commands when deploying the application. Note that using quotes doesn't prevent word splitting. | 465 466### Disable jobs 467 468The following table lists variables used to disable jobs. 469 470| **Job Name** | **CI/CDVariable** | **GitLab version** | **Description** | 471|----------------------------------------|---------------------------------|-----------------------|-----------------| 472| `.fuzz_base` | `COVFUZZ_DISABLED` | [From GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34984) | [Read more](../../user/application_security/coverage_fuzzing/) about how `.fuzz_base` provide capability for your own jobs. If the variable is present, your jobs aren't created. | 473| `apifuzzer_fuzz` | `API_FUZZING_DISABLED` | [From GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39135) | If the variable is present, the job isn't created. | 474| `build` | `BUILD_DISABLED` | | If the variable is present, the job isn't created. | 475| `build_artifact` | `BUILD_DISABLED` | | If the variable is present, the job isn't created. | 476| `bandit-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | 477| `brakeman-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | 478| `bundler-audit-dependency_scanning` | `DEPENDENCY_SCANNING_DISABLED` | | If the variable is present, the job isn't created. | 479| `canary` | `CANARY_ENABLED` | | This manual job is created if the variable is present. | 480| `code_intelligence` | `CODE_INTELLIGENCE_DISABLED` | From GitLab 13.6 | If the variable is present, the job isn't created. | 481| `codequality` | `CODE_QUALITY_DISABLED` | Until GitLab 11.0 | If the variable is present, the job isn't created. | 482| `code_quality` | `CODE_QUALITY_DISABLED` | [From GitLab 11.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5773) | If the variable is present, the job isn't created. | 483| `container_scanning` | `CONTAINER_SCANNING_DISABLED` | From GitLab 11.0 | If the variable is present, the job isn't created. | 484| `dast` | `DAST_DISABLED` | From GitLab 11.0 | If the variable is present, the job isn't created. | 485| `dast_environment_deploy` | `DAST_DISABLED_FOR_DEFAULT_BRANCH` or `DAST_DISABLED` | [From GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17789) | If either variable is present, the job isn't created. | 486| `dependency_scanning` | `DEPENDENCY_SCANNING_DISABLED` | From GitLab 11.0 | If the variable is present, the job isn't created. | 487| `eslint-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | 488| `flawfinder-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | 489| `gemnasium-dependency_scanning` | `DEPENDENCY_SCANNING_DISABLED` | | If the variable is present, the job isn't created. | 490| `gemnasium-maven-dependency_scanning` | `DEPENDENCY_SCANNING_DISABLED` | | If the variable is present, the job isn't created. | 491| `gemnasium-python-dependency_scanning` | `DEPENDENCY_SCANNING_DISABLED` | | If the variable is present, the job isn't created. | 492| `gosec-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | 493| `kubesec-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | 494| `license_management` | `LICENSE_MANAGEMENT_DISABLED` | GitLab 11.0 to 12.7 | If the variable is present, the job isn't created. Job deprecated [from GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22773) | 495| `license_scanning` | `LICENSE_MANAGEMENT_DISABLED` | [From GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22773) | If the variable is present, the job isn't created. | 496| `load_performance` | `LOAD_PERFORMANCE_DISABLED` | From GitLab 13.2 | If the variable is present, the job isn't created. | 497| `nodejs-scan-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | 498| `performance` | `PERFORMANCE_DISABLED` | GitLab 11.0 to GitLab 13.12 | Browser performance. If the variable is present, the job isn't created. Replaced by `browser_performance`. | 499| `browser_performance` | `BROWSER_PERFORMANCE_DISABLED` | From GitLab 14.0 | Browser performance. If the variable is present, the job isn't created. Replaces `performance`. | 500| `phpcs-security-audit-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | 501| `pmd-apex-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | 502| `retire-js-dependency_scanning` | `DEPENDENCY_SCANNING_DISABLED` | | If the variable is present, the job isn't created. | 503| `review` | `REVIEW_DISABLED` | From GitLab 11.0 | If the variable is present, the job isn't created. | 504| `review:stop` | `REVIEW_DISABLED` | From GitLab 11.0 | Manual job. If the variable is present, the job isn't created. | 505| `sast` | `SAST_DISABLED` | From GitLab 11.0 | If the variable is present, the job isn't created. | 506| `sast:container` | `CONTAINER_SCANNING_DISABLED` | From GitLab 11.0 | If the variable is present, the job isn't created. | 507| `secret_detection` | `SECRET_DETECTION_DISABLED` | From GitLab 13.1 | If the variable is present, the job isn't created. | 508| `secret_detection_default_branch` | `SECRET_DETECTION_DISABLED` | [From GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22773) | If the variable is present, the job isn't created. | 509| `security-code-scan-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | 510| `secrets-sast` | `SAST_DISABLED` | From GitLab 11.0 | If the variable is present, the job isn't created. | 511| `sobelaw-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | 512| `stop_dast_environment` | `DAST_DISABLED_FOR_DEFAULT_BRANCH` or `DAST_DISABLED` | [From GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17789) | If either variable is present, the job isn't created. | 513| `spotbugs-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | 514| `test` | `TEST_DISABLED` | From GitLab 11.0 | If the variable is present, the job isn't created. | 515| `staging` | `STAGING_ENABLED` | | The job is created if the variable is present. | 516| `stop_review` | `REVIEW_DISABLED` | | If the variable is present, the job isn't created. | 517 518### Application secret variables 519 520> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/49056) in GitLab 11.7. 521 522Some applications need to define secret variables that are accessible by the deployed 523application. Auto DevOps detects CI/CD variables starting with `K8S_SECRET_`, and makes 524these prefixed variables available to the deployed application as environment variables. 525 526To configure your application variables: 527 5281. On the top bar, select **Menu > Projects** and find your project. 5291. On the left sidebar, select **Settings > CI/CD**. 5301. Expand **Variables**. 5311. Create a CI/CD variable, ensuring the key is prefixed with 532 `K8S_SECRET_`. For example, you can create a variable with key 533 `K8S_SECRET_RAILS_MASTER_KEY`. 534 5351. Run an Auto DevOps pipeline, either by manually creating a new 536 pipeline or by pushing a code change to GitLab. 537 538Auto DevOps pipelines take your application secret variables to 539populate a Kubernetes secret. This secret is unique per environment. 540When deploying your application, the secret is loaded as environment 541variables in the container running the application. Following the 542example above, you can see the secret below containing the 543`RAILS_MASTER_KEY` variable. 544 545```shell 546$ kubectl get secret production-secret -n minimal-ruby-app-54 -o yaml 547 548apiVersion: v1 549data: 550 RAILS_MASTER_KEY: MTIzNC10ZXN0 551kind: Secret 552metadata: 553 creationTimestamp: 2018-12-20T01:48:26Z 554 name: production-secret 555 namespace: minimal-ruby-app-54 556 resourceVersion: "429422" 557 selfLink: /api/v1/namespaces/minimal-ruby-app-54/secrets/production-secret 558 uid: 57ac2bfd-03f9-11e9-b812-42010a9400e4 559type: Opaque 560``` 561 562Environment variables are generally considered immutable in a Kubernetes pod. 563If you update an application secret without changing any code, then manually 564create a new pipeline, any running application pods don't receive 565the updated secrets. To update the secrets, either: 566 567- Push a code update to GitLab to force the Kubernetes deployment to recreate pods. 568- Manually delete running pods to cause Kubernetes to create new pods with updated 569 secrets. 570 571Variables with multi-line values are not currently supported due to 572limitations with the current Auto DevOps scripting environment. 573 574### Advanced replica variables setup 575 576Apart from the two replica-related variables for production mentioned above, 577you can also use other variables for different environments. 578 579The Kubernetes' label named `track`, GitLab CI/CD environment names, and the 580replicas environment variable are combined into the format `TRACK_ENV_REPLICAS`, 581enabling you to define your own variables for scaling the pod's replicas: 582 583- `TRACK`: The capitalized value of the `track` 584 [Kubernetes label](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) 585 in the Helm Chart app definition. If not set, it isn't taken into account 586 to the variable name. 587- `ENV`: The capitalized environment name of the deploy job, set in 588 `.gitlab-ci.yml`. 589 590In the example below, the environment's name is `qa`, and it deploys the track 591`foo`, which results in an environment variable named `FOO_QA_REPLICAS`: 592 593```yaml 594QA testing: 595 stage: deploy 596 environment: 597 name: qa 598 script: 599 - deploy foo 600``` 601 602The track `foo` being referenced must also be defined in the application's Helm chart, like: 603 604```yaml 605replicaCount: 1 606image: 607 repository: gitlab.example.com/group/project 608 tag: stable 609 pullPolicy: Always 610 secrets: 611 - name: gitlab-registry 612application: 613 track: foo 614 tier: web 615service: 616 enabled: true 617 name: web 618 type: ClusterIP 619 url: http://my.host.com/ 620 externalPort: 5000 621 internalPort: 5000 622``` 623 624### Deploy policy for staging and production environments 625 626> [Introduced](https://gitlab.com/gitlab-org/gitlab-ci-yml/-/merge_requests/160) in GitLab 10.8. 627 628NOTE: 629You can also set this inside your [project's settings](requirements.md#auto-devops-deployment-strategy). 630 631The normal behavior of Auto DevOps is to use continuous deployment, pushing 632automatically to the `production` environment every time a new pipeline is run 633on the default branch. However, there are cases where you might want to use a 634staging environment, and deploy to production manually. For this scenario, the 635`STAGING_ENABLED` CI/CD variable was introduced. 636 637If you define `STAGING_ENABLED` with a non-empty value, then GitLab automatically deploys the application 638to a `staging` environment, and creates a `production_manual` job for 639you when you're ready to manually deploy to production. 640 641### Deploy policy for canary environments **(PREMIUM)** 642 643> [Introduced](https://gitlab.com/gitlab-org/gitlab-ci-yml/-/merge_requests/171) in GitLab 11.0. 644 645You can use a [canary environment](../../user/project/canary_deployments.md) before 646deploying any changes to production. 647 648If you define `CANARY_ENABLED` with a non-empty value, then two manual jobs are created: 649 650- `canary` - Deploys the application to the canary environment. 651- `production_manual` - Manually deploys the application to production. 652 653### Incremental rollout to production **(PREMIUM)** 654 655> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5415) in GitLab 10.8. 656 657NOTE: 658You can also set this inside your [project's settings](requirements.md#auto-devops-deployment-strategy). 659 660When you're ready to deploy a new version of your app to production, you may want 661to use an incremental rollout to replace just a few pods with the latest code to 662check how the application is behaving before manually increasing the rollout up to 100%. 663 664If `INCREMENTAL_ROLLOUT_MODE` is set to `manual` in your project, then instead 665of the standard `production` job, 4 different 666[manual jobs](../../ci/pipelines/index.md#add-manual-interaction-to-your-pipeline) 667are created: 668 6691. `rollout 10%` 6701. `rollout 25%` 6711. `rollout 50%` 6721. `rollout 100%` 673 674The percentage is based on the `REPLICAS` CI/CD variable, and defines the number of 675pods you want to have for your deployment. If the value is `10`, and you run the 676`10%` rollout job, there is `1` new pod and `9` old ones. 677 678To start a job, click the play icon (**{play}**) next to the job's name. You're not 679required to go from `10%` to `100%`, you can jump to whatever job you want. 680You can also scale down by running a lower percentage job, just before hitting 681`100%`. Once you get to `100%`, you can't scale down, and you'd have to roll 682back by redeploying the old version using the 683[rollback button](../../ci/environments/index.md#retry-or-roll-back-a-deployment) in the 684environment page. 685 686Below, you can see how the pipeline appears if the rollout or staging 687variables are defined. 688 689Without `INCREMENTAL_ROLLOUT_MODE` and without `STAGING_ENABLED`: 690 691![Staging and rollout disabled](img/rollout_staging_disabled.png) 692 693Without `INCREMENTAL_ROLLOUT_MODE` and with `STAGING_ENABLED`: 694 695![Staging enabled](img/staging_enabled.png) 696 697With `INCREMENTAL_ROLLOUT_MODE` set to `manual` and without `STAGING_ENABLED`: 698 699![Rollout enabled](img/rollout_enabled.png) 700 701With `INCREMENTAL_ROLLOUT_MODE` set to `manual` and with `STAGING_ENABLED` 702 703![Rollout and staging enabled](img/rollout_staging_enabled.png) 704 705WARNING: 706Before GitLab 11.4, the presence of the `INCREMENTAL_ROLLOUT_ENABLED` CI/CD variable 707enabled this feature. This configuration is deprecated, and is scheduled to be 708removed in the future. 709 710### Timed incremental rollout to production **(PREMIUM)** 711 712> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7545) in GitLab 11.4. 713 714NOTE: 715You can also set this inside your [project's settings](requirements.md#auto-devops-deployment-strategy). 716 717This configuration is based on 718[incremental rollout to production](#incremental-rollout-to-production). 719 720Everything behaves the same way, except: 721 722- To enable it, set the `INCREMENTAL_ROLLOUT_MODE` CI/CD variable to `timed`. 723- Instead of the standard `production` job, the following jobs are created with 724 a 5 minute delay between each: 725 726 1. `timed rollout 10%` 727 1. `timed rollout 25%` 728 1. `timed rollout 50%` 729 1. `timed rollout 100%` 730 731## Auto DevOps banner 732 733The following Auto DevOps banner displays for users with Maintainer or greater 734permissions on new projects when Auto DevOps is not enabled: 735 736![Auto DevOps banner](img/autodevops_banner_v12_6.png) 737 738The banner can be disabled for: 739 740- A user, when they dismiss it themselves. 741- A project, by explicitly [disabling Auto DevOps](index.md#enable-or-disable-auto-devops). 742- An entire GitLab instance: 743 - By an administrator running the following in a Rails console: 744 745 ```ruby 746 Feature.enable(:auto_devops_banner_disabled) 747 ``` 748 749 - Through the REST API with an administrator access token: 750 751 ```shell 752 curl --data "value=true" --header "PRIVATE-TOKEN: <personal_access_token>" "https://gitlab.example.com/api/v4/features/auto_devops_banner_disabled" 753 ``` 754