1--- 2title: Run etcd clusters inside containers 3--- 4 5The following guide shows how to run etcd with rkt and Docker using the [static bootstrap process](clustering.md#static). 6 7## rkt 8 9### Running a single node etcd 10 11The following rkt run command will expose the etcd client API on port 2379 and expose the peer API on port 2380. 12 13Use the host IP address when configuring etcd. 14 15``` 16export NODE1=192.168.1.21 17``` 18 19Trust the CoreOS [App Signing Key](https://coreos.com/security/app-signing-key/). 20 21``` 22sudo rkt trust --prefix quay.io/coreos/etcd 23# gpg key fingerprint is: 18AD 5014 C99E F7E3 BA5F 6CE9 50BD D3E0 FC8A 365E 24``` 25 26Run the `v3.2` version of etcd or specify another release version. 27 28``` 29sudo rkt run --net=default:IP=${NODE1} quay.io/coreos/etcd:v3.2 -- -name=node1 -advertise-client-urls=http://${NODE1}:2379 -initial-advertise-peer-urls=http://${NODE1}:2380 -listen-client-urls=http://0.0.0.0:2379 -listen-peer-urls=http://${NODE1}:2380 -initial-cluster=node1=http://${NODE1}:2380 30``` 31 32List the cluster member. 33 34``` 35etcdctl --endpoints=http://192.168.1.21:2379 member list 36``` 37 38### Running a 3 node etcd cluster 39 40Setup a 3 node cluster with rkt locally, using the `-initial-cluster` flag. 41 42```sh 43export NODE1=172.16.28.21 44export NODE2=172.16.28.22 45export NODE3=172.16.28.23 46``` 47 48``` 49# node 1 50sudo rkt run --net=default:IP=${NODE1} quay.io/coreos/etcd:v3.2 -- -name=node1 -advertise-client-urls=http://${NODE1}:2379 -initial-advertise-peer-urls=http://${NODE1}:2380 -listen-client-urls=http://0.0.0.0:2379 -listen-peer-urls=http://${NODE1}:2380 -initial-cluster=node1=http://${NODE1}:2380,node2=http://${NODE2}:2380,node3=http://${NODE3}:2380 51 52# node 2 53sudo rkt run --net=default:IP=${NODE2} quay.io/coreos/etcd:v3.2 -- -name=node2 -advertise-client-urls=http://${NODE2}:2379 -initial-advertise-peer-urls=http://${NODE2}:2380 -listen-client-urls=http://0.0.0.0:2379 -listen-peer-urls=http://${NODE2}:2380 -initial-cluster=node1=http://${NODE1}:2380,node2=http://${NODE2}:2380,node3=http://${NODE3}:2380 54 55# node 3 56sudo rkt run --net=default:IP=${NODE3} quay.io/coreos/etcd:v3.2 -- -name=node3 -advertise-client-urls=http://${NODE3}:2379 -initial-advertise-peer-urls=http://${NODE3}:2380 -listen-client-urls=http://0.0.0.0:2379 -listen-peer-urls=http://${NODE3}:2380 -initial-cluster=node1=http://${NODE1}:2380,node2=http://${NODE2}:2380,node3=http://${NODE3}:2380 57``` 58 59Verify the cluster is healthy and can be reached. 60 61``` 62ETCDCTL_API=3 etcdctl --endpoints=http://172.16.28.21:2379,http://172.16.28.22:2379,http://172.16.28.23:2379 endpoint health 63``` 64 65### DNS 66 67Production clusters which refer to peers by DNS name known to the local resolver must mount the [host's DNS configuration](https://coreos.com/kubernetes/docs/latest/kubelet-wrapper.html#customizing-rkt-options). 68 69## Docker 70 71In order to expose the etcd API to clients outside of Docker host, use the host IP address of the container. Please see [`docker inspect`](https://docs.docker.com/engine/reference/commandline/inspect) for more detail on how to get the IP address. Alternatively, specify `--net=host` flag to `docker run` command to skip placing the container inside of a separate network stack. 72 73### Running a single node etcd 74 75Use the host IP address when configuring etcd: 76 77``` 78export NODE1=192.168.1.21 79``` 80 81Configure a Docker volume to store etcd data: 82 83``` 84docker volume create --name etcd-data 85export DATA_DIR="etcd-data" 86``` 87 88Run the latest version of etcd: 89 90``` 91REGISTRY=quay.io/coreos/etcd 92# available from v3.2.5 93REGISTRY=gcr.io/etcd-development/etcd 94 95docker run \ 96 -p 2379:2379 \ 97 -p 2380:2380 \ 98 --volume=${DATA_DIR}:/etcd-data \ 99 --name etcd ${REGISTRY}:latest \ 100 /usr/local/bin/etcd \ 101 --data-dir=/etcd-data --name node1 \ 102 --initial-advertise-peer-urls http://${NODE1}:2380 --listen-peer-urls http://0.0.0.0:2380 \ 103 --advertise-client-urls http://${NODE1}:2379 --listen-client-urls http://0.0.0.0:2379 \ 104 --initial-cluster node1=http://${NODE1}:2380 105``` 106 107List the cluster member: 108 109``` 110etcdctl --endpoints=http://${NODE1}:2379 member list 111``` 112 113### Running a 3 node etcd cluster 114 115``` 116REGISTRY=quay.io/coreos/etcd 117# available from v3.2.5 118REGISTRY=gcr.io/etcd-development/etcd 119 120# For each machine 121ETCD_VERSION=latest 122TOKEN=my-etcd-token 123CLUSTER_STATE=new 124NAME_1=etcd-node-0 125NAME_2=etcd-node-1 126NAME_3=etcd-node-2 127HOST_1=10.20.30.1 128HOST_2=10.20.30.2 129HOST_3=10.20.30.3 130CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380 131DATA_DIR=/var/lib/etcd 132 133# For node 1 134THIS_NAME=${NAME_1} 135THIS_IP=${HOST_1} 136docker run \ 137 -p 2379:2379 \ 138 -p 2380:2380 \ 139 --volume=${DATA_DIR}:/etcd-data \ 140 --name etcd ${REGISTRY}:${ETCD_VERSION} \ 141 /usr/local/bin/etcd \ 142 --data-dir=/etcd-data --name ${THIS_NAME} \ 143 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://0.0.0.0:2380 \ 144 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://0.0.0.0:2379 \ 145 --initial-cluster ${CLUSTER} \ 146 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN} 147 148# For node 2 149THIS_NAME=${NAME_2} 150THIS_IP=${HOST_2} 151docker run \ 152 -p 2379:2379 \ 153 -p 2380:2380 \ 154 --volume=${DATA_DIR}:/etcd-data \ 155 --name etcd ${REGISTRY}:${ETCD_VERSION} \ 156 /usr/local/bin/etcd \ 157 --data-dir=/etcd-data --name ${THIS_NAME} \ 158 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://0.0.0.0:2380 \ 159 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://0.0.0.0:2379 \ 160 --initial-cluster ${CLUSTER} \ 161 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN} 162 163# For node 3 164THIS_NAME=${NAME_3} 165THIS_IP=${HOST_3} 166docker run \ 167 -p 2379:2379 \ 168 -p 2380:2380 \ 169 --volume=${DATA_DIR}:/etcd-data \ 170 --name etcd ${REGISTRY}:${ETCD_VERSION} \ 171 /usr/local/bin/etcd \ 172 --data-dir=/etcd-data --name ${THIS_NAME} \ 173 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://0.0.0.0:2380 \ 174 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://0.0.0.0:2379 \ 175 --initial-cluster ${CLUSTER} \ 176 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN} 177``` 178 179To run `etcdctl` using API version 3: 180 181``` 182docker exec etcd /bin/sh -c "export ETCDCTL_API=3 && /usr/local/bin/etcdctl put foo bar" 183``` 184 185## Bare Metal 186 187To provision a 3 node etcd cluster on bare-metal, the examples in the [baremetal repo](https://github.com/coreos/coreos-baremetal/tree/master/examples) may be useful. 188 189## Mounting a certificate volume 190 191The etcd release container does not include default root certificates. To use HTTPS with certificates trusted by a root authority (e.g., for discovery), mount a certificate directory into the etcd container: 192 193``` 194REGISTRY=quay.io/coreos/etcd 195# available from v3.2.5 196REGISTRY=docker://gcr.io/etcd-development/etcd 197 198rkt run \ 199 --insecure-options=image \ 200 --volume etcd-ssl-certs-bundle,kind=host,source=/etc/ssl/certs/ca-certificates.crt \ 201 --mount volume=etcd-ssl-certs-bundle,target=/etc/ssl/certs/ca-certificates.crt \ 202 ${REGISTRY}:latest -- --name my-name \ 203 --initial-advertise-peer-urls http://localhost:2380 --listen-peer-urls http://localhost:2380 \ 204 --advertise-client-urls http://localhost:2379 --listen-client-urls http://localhost:2379 \ 205 --discovery https://discovery.etcd.io/c11fbcdc16972e45253491a24fcf45e1 206``` 207 208``` 209REGISTRY=quay.io/coreos/etcd 210# available from v3.2.5 211REGISTRY=gcr.io/etcd-development/etcd 212 213docker run \ 214 -p 2379:2379 \ 215 -p 2380:2380 \ 216 --volume=/etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt \ 217 ${REGISTRY}:latest \ 218 /usr/local/bin/etcd --name my-name \ 219 --initial-advertise-peer-urls http://localhost:2380 --listen-peer-urls http://localhost:2380 \ 220 --advertise-client-urls http://localhost:2379 --listen-client-urls http://localhost:2379 \ 221 --discovery https://discovery.etcd.io/86a9ff6c8cb8b4c4544c1a2f88f8b801 222``` 223