1--- 2layout: docs 3page_title: Federation Between VMs and Kubernetes 4description: >- 5 Federating Kubernetes clusters and VMs. 6--- 7 8# Federation Between VMs and Kubernetes 9 10-> **1.8.0+:** This feature is available in Consul versions 1.8.0 and higher 11 12~> This topic requires familiarity with [Mesh Gateways](/docs/connect/mesh-gateway) and [WAN Federation Via Mesh Gateways](/docs/connect/gateways/wan-federation-via-mesh-gateways). 13 14Consul datacenters running on non-kubernetes platforms like VMs or bare metal can 15be federated with Kubernetes datacenters. Just like with Kubernetes, one datacenter 16must be the [primary](/docs/k8s/installation/multi-cluster/kubernetes#primary-datacenter). 17 18## Kubernetes as the Primary 19 20If your primary datacenter is running on Kubernetes, use the Helm config from the 21[Primary Datacenter](/docs/k8s/installation/multi-cluster/kubernetes#primary-datacenter) section to install Consul. 22 23Once installed on Kubernetes, and with the `ProxyDefaults` [resource created](/docs/k8s/installation/multi-cluster/kubernetes#proxydefaults), 24you'll need to export the following information from the primary Kubernetes cluster: 25 26- Certificate authority cert and key (in order to create SSL certs for VMs) 27- External addresses of Kubernetes mesh gateways 28- Replication ACL token 29- Gossip encryption key 30 31The following sections detail how to export this data. 32 33### Certificates 34 351. Retrieve the certificate authority cert: 36 37 ```sh 38 kubectl get secrets/consul-ca-cert --template='{{index .data "tls.crt" | base64decode }}' > consul-agent-ca.pem 39 ``` 40 411. And the certificate authority signing key: 42 43 ```sh 44 kubectl get secrets/consul-ca-key --template='{{index .data "tls.key" | base64decode }}' > consul-agent-ca-key.pem 45 ``` 46 471. With the `consul-agent-ca.pem` and `consul-agent-ca-key.pem` files you can 48 create certificates for your servers and clients running on VMs that share the 49 same certificate authority as your Kubernetes servers. 50 51 You can use the `consul tls` commands to generate those certificates: 52 53 ```sh 54 # NOTE: consul-agent-ca.pem and consul-agent-ca-key.pem must be in the current 55 # directory. 56 $ consul tls cert create -server -dc=vm-dc -node <node_name> 57 ==> WARNING: Server Certificates grants authority to become a 58 server and access all state in the cluster including root keys 59 and all ACL tokens. Do not distribute them to production hosts 60 that are not server nodes. Store them as securely as CA keys. 61 ==> Using consul-agent-ca.pem and consul-agent-ca-key.pem 62 ==> Saved vm-dc-server-consul-0.pem 63 ==> Saved vm-dc-server-consul-0-key.pem 64 ``` 65 66 -> Note the `-node` option in the above command. This should be same as the node name of the [Consul Agent](https://www.consul.io/docs/agent#running-an-agent). This is a [requirement](https://www.consul.io/docs/connect/gateways/mesh-gateway/wan-federation-via-mesh-gateways#tls) for Consul Federation to work. Alternatively, if you plan to use the same certificate and key pair on all your Consul server nodes, or you don't know the nodename in advance, use `-node "*"` instead. 67 Not satisfying this requirement would result in the following error in the Consul Server logs: 68 `[ERROR] agent.server.rpc: TLS handshake failed: conn=from= error="remote error: tls: bad certificate"` 69 70 See the help for output of `consul tls cert create -h` to see more options 71 for generating server certificates. 72 731. These certificates can be used in your server config file: 74 75 <CodeBlockConfig filename="server.hcl"> 76 77 ```hcl 78 cert_file = "vm-dc-server-consul-0.pem" 79 key_file = "vm-dc-server-consul-0-key.pem" 80 ca_file = "consul-agent-ca.pem" 81 ``` 82 83 </CodeBlockConfig> 84 851. For clients, you can generate TLS certs with: 86 87 ```shell-session 88 $ consul tls cert create -client 89 ==> Using consul-agent-ca.pem and consul-agent-ca-key.pem 90 ==> Saved dc1-client-consul-0.pem 91 ==> Saved dc1-client-consul-0-key.pem 92 ``` 93 94 Or use the [auto_encrypt](/docs/agent/options#auto_encrypt) feature. 95 96### Mesh Gateway Addresses 97 98Retrieve the WAN addresses of the mesh gateways: 99 100```shell-session 101$ kubectl exec statefulset/consul-server -- sh -c \ 102 'curl -sk https://localhost:8501/v1/catalog/service/mesh-gateway | jq ".[].ServiceTaggedAddresses.wan"' 103{ 104 "Address": "1.2.3.4", 105 "Port": 443 106} 107{ 108 "Address": "1.2.3.4", 109 "Port": 443 110} 111``` 112 113In this example, the addresses are the same because both mesh gateway pods are 114fronted by the same Kubernetes load balancer. 115 116These addresses will be used in the server config for the `primary_gateways` 117setting: 118 119```hcl 120primary_gateways = ["1.2.3.4:443"] 121``` 122 123### Replication ACL Token 124 125If ACLs are enabled, you'll also need the replication ACL token: 126 127```shell-session 128$ kubectl get secrets/consul-acl-replication-acl-token --template='{{.data.token}}' 129e7924dd1-dc3f-f644-da54-81a73ba0a178 130``` 131 132This token will be used in the server config for the replication token. 133 134```hcl 135acls { 136 tokens { 137 replication = "e7924dd1-dc3f-f644-da54-81a73ba0a178" 138 } 139} 140``` 141 142-> **NOTE:** You'll also need to set up additional ACL tokens as needed by the 143ACL system. See tutorial [Secure Consul with Access Control Lists (ACLs)](https://learn.hashicorp.com/tutorials/consul/access-control-setup-production#apply-individual-tokens-to-agents) 144for more information. 145 146### Gossip Encryption Key 147 148If gossip encryption is enabled, you'll need the key as well. The command 149to retrieve the key will depend on which Kubernetes secret you've stored it in. 150 151This key will be used in server and client configs for the `encrypt` setting: 152 153```hcl 154encrypt = "uF+GsbI66cuWU21kiXLze5JLEX5j4iDFlDTb0ZWNpDI=" 155``` 156 157### Final Configuration 158 159A final example server config file might look like: 160 161```hcl 162# From above 163cert_file = "vm-dc-server-consul-0.pem" 164key_file = "vm-dc-server-consul-0-key.pem" 165ca_file = "consul-agent-ca.pem" 166primary_gateways = ["1.2.3.4:443"] 167acl { 168 enabled = true 169 default_policy = "deny" 170 down_policy = "extend-cache" 171 tokens { 172 agent = "e7924dd1-dc3f-f644-da54-81a73ba0a178" 173 replication = "e7924dd1-dc3f-f644-da54-81a73ba0a178" 174 } 175} 176encrypt = "uF+GsbI66cuWU21kiXLze5JLEX5j4iDFlDTb0ZWNpDI=" 177 178# Other server settings 179server = true 180datacenter = "vm-dc" 181data_dir = "/opt/consul" 182enable_central_service_config = true 183primary_datacenter = "dc1" 184connect { 185 enabled = true 186 enable_mesh_gateway_wan_federation = true 187} 188verify_incoming_rpc = true 189verify_outgoing = true 190verify_server_hostname = true 191ports { 192 https = 8501 193 http = -1 194 grpc = 8502 195} 196``` 197 198## Kubernetes as the Secondary 199 200If you're running your primary datacenter on VMs then you'll need to manually 201construct the [Federation Secret](/docs/k8s/installation/multi-cluster/kubernetes#federation-secret) in order to federate 202Kubernetes clusters as secondaries. 203 204-> Your VM cluster must be running mesh gateways, and have mesh gateway WAN 205federation enabled. See [WAN Federation via Mesh Gateways](/docs/connect/gateways/wan-federation-via-mesh-gateways). 206 207You'll need: 208 2091. The root certificate authority cert placed in `consul-agent-ca.pem`. 2101. The root certificate authority key placed in `consul-agent-ca-key.pem`. 2111. The IP addresses of the mesh gateways running in your VM datacenter. These must 212 be routable from the Kubernetes cluster. 2131. If ACLs are enabled you must create an ACL replication token with the following rules: 214 215 ```hcl 216 acl = "write" 217 operator = "write" 218 agent_prefix "" { 219 policy = "read" 220 } 221 node_prefix "" { 222 policy = "write" 223 } 224 service_prefix "" { 225 policy = "read" 226 intentions = "read" 227 } 228 ``` 229 230 This token is used for ACL replication and for automatic ACL management in Kubernetes. 231 232 If you're running Consul Enterprise you'll need the rules: 233 234 ```hcl 235 acl = "write" 236 operator = "write" 237 agent_prefix "" { 238 policy = "read" 239 } 240 node_prefix "" { 241 policy = "write" 242 } 243 namespace_prefix "" { 244 acl = "write" 245 service_prefix "" { 246 policy = "read" 247 intentions = "read" 248 } 249 } 250 ``` 251 2521. If gossip encryption is enabled, you'll need the key. 253 254With that data ready, you can create the Kubernetes federation secret: 255 256```sh 257kubectl create secret generic consul-federation \ 258 --from-literal=caCert=$(cat consul-agent-ca.pem) \ 259 --from-literal=caKey=$(cat consul-agent-ca-key.pem) 260 # If ACLs are enabled uncomment. 261 # --from-literal=replicationToken="<your acl replication token>" \ 262 # If using gossip encryption uncomment. 263 # --from-literal=gossipEncryptionKey="<your gossip encryption key>" 264``` 265 266Then use the following Helm config file: 267 268```yaml 269global: 270 name: consul 271 datacenter: dc2 272 tls: 273 enabled: true 274 caCert: 275 secretName: consul-federation 276 secretKey: caCert 277 caKey: 278 secretName: consul-federation 279 secretKey: caKey 280 281 # Delete this acls section if ACLs are disabled. 282 acls: 283 manageSystemACLs: true 284 replicationToken: 285 secretName: consul-federation 286 secretKey: replicationToken 287 288 federation: 289 enabled: true 290 291 # Delete this gossipEncryption section if gossip encryption is disabled. 292 gossipEncryption: 293 secretName: consul-federation 294 secretKey: gossipEncryptionKey 295 296connectInject: 297 enabled: true 298controller: 299 enabled: true 300meshGateway: 301 enabled: true 302server: 303 extraConfig: | 304 { 305 "primary_datacenter": "<your VM datacenter name>", 306 "primary_gateways": ["<ip of your VM mesh gateway>", "<other ip>", ...] 307 } 308``` 309 310-> **NOTE: ** You must fill out the `server.extraConfig` section with the datacenter 311name of your primary datacenter running on VMs and with the IPs of your mesh 312gateways running on VMs. 313 314With your config file ready to go, follow our [Installation Guide](/docs/k8s/installation/install) 315to install Consul on your secondary cluster(s). 316 317After installation, if you're using consul-helm 0.30.0+, [create the 318`ProxyDefaults` resource](/docs/k8s/installation/multi-cluster/kubernetes#proxydefaults) 319to allow traffic between datacenters. 320 321## Next Steps 322 323In both cases (Kubernetes as primary or secondary), after installation, follow the [Verifying Federation](/docs/k8s/installation/multi-cluster/kubernetes#verifying-federation) 324section to verify that federation is working as expected. 325