1--- 2layout: docs 3page_title: Service Mesh - Kubernetes 4description: >- 5 Connect is a feature built into to Consul that enables automatic 6 service-to-service authorization and connection encryption across your Consul 7 services. Connect can be used with Kubernetes to secure pod communication with 8 other services. 9--- 10 11# Connect Service Mesh on Kubernetes 12 13[Connect](/docs/connect) is a feature built into to Consul that enables 14automatic service-to-service authorization and connection encryption across 15your Consul services. Connect can be used with Kubernetes to secure pod 16communication with other pods and external Kubernetes services. 17 18The Connect sidecar running Envoy can be automatically injected into pods in 19your cluster, making configuration for Kubernetes automatic. 20This functionality is provided by the 21[consul-k8s project](https://github.com/hashicorp/consul-k8s) and can be 22automatically installed and configured using the 23[Consul Helm chart](/docs/k8s/installation/install). 24 25## Usage 26 27When the 28[Connect injector is installed](/docs/k8s/connect#installation-and-configuration), 29the Connect sidecar can be automatically added to all pods. This sidecar can both 30accept and establish connections using Connect, enabling the pod to communicate 31to clients and dependencies exclusively over authorized and encrypted 32connections. 33 34-> **Note:** The examples in this section are valid and use 35publicly available images. If you've installed the Connect injector, feel free 36to run the examples in this section to try Connect with Kubernetes. 37Please note the documentation below this section on how to properly install 38and configure the Connect injector. 39 40### Accepting Inbound Connections 41 42An example Deployment is shown below with Connect enabled to accept inbound 43connections. Notice that the Deployment would still be fully functional without 44Connect. Minimal to zero modifications are required to enable Connect in Kubernetes. 45Notice also that even though we're using a Deployment here, the same configuration 46would work on a Pod, a StatefulSet, or a DaemonSet. 47 48This Deployment specification starts a server that responds to any 49HTTP request with the static text "hello world". 50 51-> **Note:** As of consul-k8s `v0.26.0-beta1` and Consul Helm `v0.32.0-beta1`, having a Kubernetes 52service is **required** to run services on the Consul Service Mesh. 53 54```yaml 55apiVersion: v1 56kind: Service 57metadata: 58 # This name will be the service name in Consul. 59 name: static-server 60spec: 61 selector: 62 app: static-server 63 ports: 64 - protocol: TCP 65 port: 80 66 targetPort: 8080 67--- 68apiVersion: v1 69kind: ServiceAccount 70metadata: 71 name: static-server 72--- 73apiVersion: apps/v1 74kind: Deployment 75metadata: 76 name: static-server 77spec: 78 replicas: 1 79 selector: 80 matchLabels: 81 app: static-server 82 template: 83 metadata: 84 name: static-server 85 labels: 86 app: static-server 87 annotations: 88 'consul.hashicorp.com/connect-inject': 'true' 89 spec: 90 containers: 91 - name: static-server 92 image: hashicorp/http-echo:latest 93 args: 94 - -text="hello world" 95 - -listen=:8080 96 ports: 97 - containerPort: 8080 98 name: http 99 # If ACLs are enabled, the serviceAccountName must match the Consul service name. 100 serviceAccountName: static-server 101``` 102 103The only change for Connect is the addition of the 104`consul.hashicorp.com/connect-inject` annotation. This enables injection 105for the Pod in this Deployment. The injector can also be 106[configured](/docs/k8s/connect#installation-and-configuration) 107to automatically inject unless explicitly disabled, but the default 108installation requires opt-in using the annotation shown above. 109 110~> **A common mistake** is to set the annotation on the Deployment or 111other resource. Ensure that the injector annotations are specified on 112the _pod specification template_ as shown above. 113 114This will start a sidecar proxy that listens on port `20000` registered 115with Consul and proxies valid inbound connections to port 8080 in the pod. 116To establish a connection to the pod using Connect, a client must use another Connect 117proxy. The client Connect proxy will use Consul service discovery to find 118all available upstream proxies and their public ports. 119 120In the example above, the server is listening on `:8080`. 121By default, the Consul Service Mesh runs in [transparent proxy](/docs/connect/transparent-proxy) mode. 122This means that even though the server binds to all interfaces, 123the inbound and outbound connections will automatically go through to the sidecar proxy. 124It also allows you to use Kubernetes DNS like you normally would without the 125Consul Service Mesh. 126 127-> **Note:** As of consul `v1.10.0`, consul-k8s `v0.26.0` and Consul Helm `v0.32.0`, 128all Consul Service Mesh services will run with transparent proxy enabled by default. Running with transparent 129proxy will enforce all inbound and outbound traffic to go through the Envoy proxy. 130 131The service name registered in Consul will be set to the name of the Kubernetes service 132associated with the Pod. This can be customized with the `consul.hashicorp.com/connect-service` 133annotation. If using ACLs, this name must be the same as the Pod's `ServiceAccount` name. 134 135### Connecting to Connect-Enabled Services 136 137The example Deployment specification below configures a Deployment that is capable 138of establishing connections to our previous example "static-server" service. The 139connection to this static text service happens over an authorized and encrypted 140connection via Connect. 141 142-> **Note:** As of consul-k8s `v0.26.0` and Consul Helm `v0.32.0`, having a Kubernetes 143Service is **required** to run services on the Consul Service Mesh. 144 145```yaml 146apiVersion: v1 147kind: Service 148metadata: 149 # This name will be the service name in Consul. 150 name: static-client 151spec: 152 selector: 153 app: static-client 154 ports: 155 - port: 80 156--- 157apiVersion: v1 158kind: ServiceAccount 159metadata: 160 name: static-client 161--- 162apiVersion: apps/v1 163kind: Deployment 164metadata: 165 name: static-client 166spec: 167 replicas: 1 168 selector: 169 matchLabels: 170 app: static-client 171 template: 172 metadata: 173 name: static-client 174 labels: 175 app: static-client 176 annotations: 177 'consul.hashicorp.com/connect-inject': 'true' 178 spec: 179 containers: 180 - name: static-client 181 image: curlimages/curl:latest 182 # Just spin & wait forever, we'll use `kubectl exec` to demo 183 command: ['/bin/sh', '-c', '--'] 184 args: ['while true; do sleep 30; done;'] 185 # If ACLs are enabled, the serviceAccountName must match the Consul service name. 186 serviceAccountName: static-client 187``` 188 189By default when ACLs are enabled or when ACLs default policy is `allow`, 190Consul will automatically configure proxies with all upstreams from the same datacenter. 191When ACLs are enabled with default `deny` policy, 192you must supply an [intention](/docs/connect/intentions) to tell Consul which upstream you need to talk to. 193 194When upstreams are specified explicitly with the 195[`consul.hashicorp.com/connect-service-upstreams` annotation](/docs/k8s/connect#consul-hashicorp-com-connect-service-upstreams), 196the injector will also set environment variables `<NAME>_CONNECT_SERVICE_HOST` 197and `<NAME>_CONNECT_SERVICE_PORT` in every container in the Pod for every defined 198upstream. This is analogous to the standard Kubernetes service environment variables, but 199point instead to the correct local proxy port to establish connections via 200Connect. 201 202We can verify access to the static text server using `kubectl exec`. 203Because transparent proxy is enabled by default, 204we use Kubernetes DNS to connect to our desired upstream. 205 206```shell-session 207$ kubectl exec deploy/static-client -- curl -s http://static-server/ 208"hello world" 209``` 210 211We can control access to the server using [intentions](/docs/connect/intentions). 212If you use the Consul UI or [CLI](/commands/intention/create) to 213create a deny [intention](/docs/connect/intentions) between 214"static-client" and "static-server", connections are immediately rejected 215without updating either of the running pods. You can then remove this 216intention to allow connections again. 217 218```shell-session 219$ kubectl exec deploy/static-client -- curl -s http://static-server/ 220command terminated with exit code 52 221``` 222 223### Available Annotations 224 225Pod annotations can be used to configure the injection behavior. 226 227- `consul.hashicorp.com/connect-inject` - If this is "true" then injection 228 is enabled. If this is "false" then injection is explicitly disabled. 229 The default injector behavior requires pods to opt-in to injection by 230 specifying this value as "true". This default can be changed in the 231 injector's configuration if desired. 232 233- `consul.hashicorp.com/transparent-proxy` - If this is "true", this Pod 234 will run with transparent proxy enabled. This means you can use Kubernetes 235 DNS to access upstream services and all inbound and outbound traffic within 236 the pod is redirected to go through the proxy. 237 238- `consul.hashicorp.com/transparent-proxy-overwrite-probes` - If this is "true" 239 and transparent proxy is enabled, the Connect injector will overwrite Kubernetes 240 HTTP probes to point to the Envoy proxy. 241 242- `consul.hashicorp.com/transparent-proxy-exclude-inbound-ports` - A comma-separated 243 list of inbound ports to exclude from traffic redirection when running in transparent proxy 244 mode. 245 246- `consul.hashicorp.com/transparent-proxy-exclude-outbound-cidrs` - A comma-separated 247 list of outbound CIDRs to exclude from traffic redirection when running in transparent proxy 248 mode. 249 250- `consul.hashicorp.com/transparent-proxy-exclude-outbound-ports` - A comma-separated 251 list of outbound ports to exclude from traffic redirection when running in transparent proxy 252 mode. 253 254- `consul.hashicorp.com/transparent-proxy-exclude-uids` - A comma-separated 255 list of additional user IDs to exclude from traffic redirection when running in transparent proxy 256 mode. 257 258- `consul.hashicorp.com/connect-service` - For pods that accept inbound 259 connections, this specifies the name of the service that is being 260 served. This defaults to the name of the Kubernetes service associated with the pod. 261 262 If using ACLs, this must be the same name as the Pod's `ServiceAccount`. 263 264- `consul.hashicorp.com/connect-service-port` - For pods that accept inbound 265 connections, this specifies the port to route inbound connections to. This 266 is the port that the service is listening on. The service port defaults to 267 the first exposed port on any container in the pod. If specified, the value 268 can be the _name_ of a configured port, such as "http" or it can be a direct 269 port value such as "8080". This is the port of the _service_, the proxy 270 public listener will listen on a dynamic port. 271 272- `consul.hashicorp.com/connect-service-upstreams` - The list of upstream 273 services that this pod needs to connect to via Connect along with a static 274 local port to listen for those connections. When transparent proxy is enabled, 275 this annotation is optional. 276 277 - Services 278 279 The name of the service is the name of the service registered with Consul. You can optionally specify datacenters with this annotation. 280 281 ```yaml 282 annotations: 283 "consul.hashicorp.com/connect-service-upstreams":"[service-name]:[port]:[optional datacenter]" 284 ``` 285 286 - Consul Enterprise Namespaces 287 288 If running Consul Enterprise 1.7+, your upstream services may be running in different 289 namespaces. The upstream namespace can be specified after the service name 290 as `[service-name].[namespace]`. See [Consul Enterprise Namespaces](#consul-enterprise-namespaces) 291 below for more details on configuring the injector. 292 293 ```yaml 294 annotations: 295 "consul.hashicorp.com/connect-service-upstreams":"[service-name].[service-namespace]:[port]:[optional datacenter]" 296 ``` 297 298 -> **NOTE:** If the namespace is not specified it will default to the namespace 299 of the source service. 300 301 ~> **WARNING:** Setting a namespace when not using Consul Enterprise or using a version < 1.7 302 is not supported. It will be treated as part of the service name. 303 304 - [Prepared Query](/docs/connect/proxies#dynamic-upstreams-require-native-integration) 305 306 ```yaml 307 annotations: 308 'consul.hashicorp.com/connect-service-upstreams': 'prepared_query:[query name]:[port]' 309 ``` 310 311 - Multiple Upstreams 312 313 If you would like to specify multiple services or upstreams, delimit them with commas 314 315 ```yaml 316 annotations: 317 "consul.hashicorp.com/connect-service-upstreams":"[service-name]:[port]:[optional datacenter],[service-name]:[port]:[optional datacenter]" 318 ``` 319 320 ```yaml 321 annotations: 322 "consul.hashicorp.com/connect-service-upstreams":"[service-name]:[port]:[optional datacenter],prepared_query:[query name]:[port]" 323 ``` 324 325- `consul.hashicorp.com/envoy-extra-args` - A space-separated list of [arguments](https://www.envoyproxy.io/docs/envoy/latest/operations/cli) 326 to be passed to the injected envoy binary. 327 328 ```yaml 329 annotations: 330 consul.hashicorp.com/envoy-extra-args: '--log-level debug --disable-hot-restart' 331 ``` 332 333- `consul.hashicorp.com/service-tags` - A comma separated list of tags that will 334 be applied to the Consul service and its sidecar. 335 336 ```yaml 337 annotations: 338 consul.hashicorp.com/service-tags: foo,bar,baz 339 ``` 340 341- `consul.hashicorp.com/service-meta-<YOUR_KEY>` - Set Consul meta key/value 342 pairs that will be applied to the Consul service and its sidecar. 343 The key will be what comes after `consul.hashicorp.com/service-meta-`, e.g. 344 `consul.hashicorp.com/service-meta-foo: bar` will result in `foo: bar`. 345 346 ```yaml 347 annotations: 348 consul.hashicorp.com/service-meta-foo: baz 349 consul.hashicorp.com/service-meta-bar: baz 350 ``` 351 352- `consul.hashicorp.com/sidecar-proxy-` - Override default resource settings for 353 the sidecar proxy container. 354 The defaults are set in Helm config via the [`connectInject.sidecarProxy.resources`](/docs/k8s/helm#v-connectinject-sidecarproxy-resources) key. 355 356 - `consul.hashicorp.com/sidecar-proxy-cpu-limit` - Override the default CPU limit. 357 - `consul.hashicorp.com/sidecar-proxy-cpu-request` - Override the default CPU request. 358 - `consul.hashicorp.com/sidecar-proxy-memory-limit` - Override the default memory limit. 359 - `consul.hashicorp.com/sidecar-proxy-memory-request` - Override the default memory request. 360 361- `consul.hashicorp.com/enable-metrics` - Override the default Helm value [`connectInject.metrics.defaultEnabled`](/docs/k8s/helm#v-connectinject-metrics-defaultenabled). 362- `consul.hashicorp.com/enable-metrics-merging` - Override the default Helm value [`connectInject.metrics.defaultEnableMerging`](/docs/k8s/helm#v-connectinject-metrics-defaultenablemerging). 363- `consul.hashicorp.com/merged-metrics-port` - Override the default Helm value [`connectInject.metrics.defaultMergedMetricsPort`](/docs/k8s/helm#v-connectinject-metrics-defaultmergedmetricsport). 364- `consul.hashicorp.com/prometheus-scrape-port` - Override the default Helm value [`connectInject.metrics.defaultPrometheusScrapePort`](/docs/k8s/helm#v-connectinject-metrics-defaultprometheusscrapeport). 365- `consul.hashicorp.com/prometheus-scrape-path` - Override the default Helm value [`connectInject.metrics.defaultPrometheusScrapePath`](/docs/k8s/helm#v-connectinject-metrics-defaultprometheusscrapepath). 366- `consul.hashicorp.com/service-metrics-port` - Set the port where the Connect service exposes metrics. 367- `consul.hashicorp.com/service-metrics-path` - Set the path where the Connect service exposes metrics. 368 369## Installation and Configuration 370 371The Connect sidecar proxy is injected via a 372[mutating admission webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks) 373provided by the 374[consul-k8s project](https://github.com/hashicorp/consul-k8s). 375This enables the automatic pod mutation shown in the usage section above. 376Installation of the mutating admission webhook is automated using the 377[Helm chart](/docs/k8s/installation/install). 378 379To install the Connect injector, enable the Connect injection feature using 380[Helm values](/docs/k8s/helm#configuration-values) and 381upgrade the installation using `helm upgrade` for existing installs or 382`helm install` for a fresh install. 383 384```yaml 385connectInject: 386 enabled: true 387 388controller: 389 enabled: true 390``` 391 392This will configure the injector to inject when the 393[injection annotation](#consul-hashicorp-com-connect-inject) 394is set to `true`. Other values in the Helm chart can be used to limit the namespaces 395the injector runs in, enable injection by default, and more. 396 397### Controlling Injection Via Annotation 398 399By default, the injector will inject only when the 400[injection annotation](#consul-hashicorp-com-connect-inject) 401on the pod (not the deployment) is set to `true`: 402 403```yaml 404annotations: 405 'consul.hashicorp.com/connect-inject': 'true' 406``` 407 408### Injection Defaults 409 410If you wish for the injector to always inject, you can set the default to `true` 411in the Helm chart: 412 413```yaml 414connectInject: 415 enabled: true 416 default: true 417``` 418 419You can then exclude specific pods via annotation: 420 421```yaml 422annotations: 423 'consul.hashicorp.com/connect-inject': 'false' 424``` 425 426### Controlling Injection Via Namespace 427 428You can control which Kubernetes namespaces are allowed to be injected via 429the `k8sAllowNamespaces` and `k8sDenyNamespaces` keys: 430 431```yaml 432connectInject: 433 enabled: true 434 k8sAllowNamespaces: ['*'] 435 k8sDenyNamespaces: [] 436``` 437 438In the default configuration (shown above), services from all namespaces are allowed 439to be injected. Whether or not they're injected depends on the value of `connectInject.default` 440and the `consul.hashicorp.com/connect-inject` annotation. 441 442If you wish to only enable injection in specific namespaces, you can list only those 443namespaces in the `k8sAllowNamespaces` key. In the configuration below 444only the `my-ns-1` and `my-ns-2` namespaces will be enabled for injection. 445All other namespaces will be ignored, even if the connect inject [annotation](#consul-hashicorp-com-connect-inject) 446is set. 447 448```yaml 449connectInject: 450 enabled: true 451 k8sAllowNamespaces: ['my-ns-1', 'my-ns-2'] 452 k8sDenyNamespaces: [] 453``` 454 455If you wish to enable injection in every namespace _except_ specific namespaces, you can 456use `*` in the allow list to allow all namespaces and then specify the namespaces to exclude in the deny list: 457 458```yaml 459connectInject: 460 enabled: true 461 k8sAllowNamespaces: ['*'] 462 k8sDenyNamespaces: ['no-inject-ns-1', 'no-inject-ns-2'] 463``` 464 465-> **NOTE:** The deny list takes precedence over the allow list. If a namespace 466is listed in both lists, it will **not** be synced. 467 468~> **NOTE:** The `kube-system` and `kube-public` namespaces will never be injected. 469 470### Consul Enterprise Namespaces 471 472Consul Enterprise 1.7+ supports Consul namespaces. When Kubernetes pods are registered 473into Consul, you can control which Consul namespace they are registered into. 474 475There are three options available: 476 4771. **Single Destination Namespace** – Register all Kubernetes pods, regardless of namespace, 478 into the same Consul namespace. 479 480 This can be configured with: 481 482 ```yaml 483 global: 484 enableConsulNamespaces: true 485 486 connectInject: 487 enabled: true 488 consulNamespaces: 489 consulDestinationNamespace: 'my-consul-ns' 490 ``` 491 492 -> **NOTE:** If the destination namespace does not exist we will create it. 493 4941. **Mirror Namespaces** - Register each Kubernetes pod into a Consul namespace with the same name as its Kubernetes namespace. 495 For example, pod `foo` in Kubernetes namespace `ns-1` will be synced to the Consul namespace `ns-1`. 496 If a mirrored namespace does not exist in Consul, it will be created. 497 498 This can be configured with: 499 500 ```yaml 501 global: 502 enableConsulNamespaces: true 503 504 connectInject: 505 enabled: true 506 consulNamespaces: 507 mirroringK8S: true 508 ``` 509 5101. **Mirror Namespaces With Prefix** - Register each Kubernetes pod into a Consul namespace with the same name as its Kubernetes 511 namespace **with a prefix**. 512 For example, given a prefix `k8s-`, pod `foo` in Kubernetes namespace `ns-1` will be synced to the Consul namespace `k8s-ns-1`. 513 514 This can be configured with: 515 516 ```yaml 517 global: 518 enableConsulNamespaces: true 519 520 connectInject: 521 enabled: true 522 consulNamespaces: 523 mirroringK8S: true 524 mirroringK8SPrefix: 'k8s-' 525 ``` 526 527### Consul Enterprise Namespace Upstreams 528 529When [transparent proxy](/docs/connect/transparent-proxy) is enabled and ACLs are disabled, 530the upstreams will be configured automatically across Consul namespaces. 531When ACLs are enabled, you must configure it by specifying an [intention](/docs/connect/intentions), 532allowing services across Consul namespaces to talk to each other. 533 534If you wish to specify an upstream explicitly via the `consul.hashicorp.com/connect-service-upstreams` annotation, 535use the format `[service-name].[namespace]:[port]:[optional datacenter]`: 536 537```yaml 538annotations: 539 'consul.hashicorp.com/connect-inject': 'true' 540 'consul.hashicorp.com/connect-service-upstreams': '[service-name].[namespace]:[port]:[optional datacenter]' 541``` 542 543See [consul.hashicorp.com/connect-service-upstreams](#consul-hashicorp-com-connect-service-upstreams) for more details. 544 545-> **Note:** When you specify upstreams via an upstreams annotation, you will need to use 546`localhost:<port>` with the port from the upstreams annotation instead of KubeDNS to connect to your upstream 547application. 548 549### Verifying the Installation 550 551To verify the installation, run the 552["Accepting Inbound Connections"](/docs/k8s/connect#accepting-inbound-connections) 553example from the "Usage" section above. After running this example, run 554`kubectl get pod static-server -o yaml`. In the raw YAML output, you should 555see injected Connect containers and an annotation 556`consul.hashicorp.com/connect-inject-status` set to `injected`. This 557confirms that injection is working properly. 558 559If you do not see this, then use `kubectl logs` against the injector pod 560and note any errors. 561