1---
2title: Run etcd clusters inside containers
3---
4
5The following guide shows how to run etcd with rkt and Docker using the [static bootstrap process](clustering.md#static).
6
7## rkt
8
9### Running a single node etcd
10
11The following rkt run command will expose the etcd client API on port 2379 and expose the peer API on port 2380.
12
13Use the host IP address when configuring etcd.
14
15```
16export NODE1=192.168.1.21
17```
18
19Trust the CoreOS [App Signing Key](https://coreos.com/security/app-signing-key/).
20
21```
22sudo rkt trust --prefix quay.io/coreos/etcd
23# gpg key fingerprint is: 18AD 5014 C99E F7E3 BA5F  6CE9 50BD D3E0 FC8A 365E
24```
25
26Run the `v3.2` version of etcd or specify another release version.
27
28```
29sudo rkt run --net=default:IP=${NODE1} quay.io/coreos/etcd:v3.2 -- -name=node1 -advertise-client-urls=http://${NODE1}:2379 -initial-advertise-peer-urls=http://${NODE1}:2380 -listen-client-urls=http://0.0.0.0:2379 -listen-peer-urls=http://${NODE1}:2380 -initial-cluster=node1=http://${NODE1}:2380
30```
31
32List the cluster member.
33
34```
35etcdctl --endpoints=http://192.168.1.21:2379 member list
36```
37
38### Running a 3 node etcd cluster
39
40Setup a 3 node cluster with rkt locally, using the `-initial-cluster` flag.
41
42```sh
43export NODE1=172.16.28.21
44export NODE2=172.16.28.22
45export NODE3=172.16.28.23
46```
47
48```
49# node 1
50sudo rkt run --net=default:IP=${NODE1} quay.io/coreos/etcd:v3.2 -- -name=node1 -advertise-client-urls=http://${NODE1}:2379 -initial-advertise-peer-urls=http://${NODE1}:2380 -listen-client-urls=http://0.0.0.0:2379 -listen-peer-urls=http://${NODE1}:2380 -initial-cluster=node1=http://${NODE1}:2380,node2=http://${NODE2}:2380,node3=http://${NODE3}:2380
51
52# node 2
53sudo rkt run --net=default:IP=${NODE2} quay.io/coreos/etcd:v3.2 -- -name=node2 -advertise-client-urls=http://${NODE2}:2379 -initial-advertise-peer-urls=http://${NODE2}:2380 -listen-client-urls=http://0.0.0.0:2379 -listen-peer-urls=http://${NODE2}:2380 -initial-cluster=node1=http://${NODE1}:2380,node2=http://${NODE2}:2380,node3=http://${NODE3}:2380
54
55# node 3
56sudo rkt run --net=default:IP=${NODE3} quay.io/coreos/etcd:v3.2 -- -name=node3 -advertise-client-urls=http://${NODE3}:2379 -initial-advertise-peer-urls=http://${NODE3}:2380 -listen-client-urls=http://0.0.0.0:2379 -listen-peer-urls=http://${NODE3}:2380 -initial-cluster=node1=http://${NODE1}:2380,node2=http://${NODE2}:2380,node3=http://${NODE3}:2380
57```
58
59Verify the cluster is healthy and can be reached.
60
61```
62ETCDCTL_API=3 etcdctl --endpoints=http://172.16.28.21:2379,http://172.16.28.22:2379,http://172.16.28.23:2379 endpoint health
63```
64
65### DNS
66
67Production clusters which refer to peers by DNS name known to the local resolver must mount the [host's DNS configuration](https://coreos.com/kubernetes/docs/latest/kubelet-wrapper.html#customizing-rkt-options).
68
69## Docker
70
71In order to expose the etcd API to clients outside of Docker host, use the host IP address of the container. Please see [`docker inspect`](https://docs.docker.com/engine/reference/commandline/inspect) for more detail on how to get the IP address. Alternatively, specify `--net=host` flag to `docker run` command to skip placing the container inside of a separate network stack.
72
73### Running a single node etcd
74
75Use the host IP address when configuring etcd:
76
77```
78export NODE1=192.168.1.21
79```
80
81Configure a Docker volume to store etcd data:
82
83```
84docker volume create --name etcd-data
85export DATA_DIR="etcd-data"
86```
87
88Run the latest version of etcd:
89
90```
91REGISTRY=quay.io/coreos/etcd
92# available from v3.2.5
93REGISTRY=gcr.io/etcd-development/etcd
94
95docker run \
96  -p 2379:2379 \
97  -p 2380:2380 \
98  --volume=${DATA_DIR}:/etcd-data \
99  --name etcd ${REGISTRY}:latest \
100  /usr/local/bin/etcd \
101  --data-dir=/etcd-data --name node1 \
102  --initial-advertise-peer-urls http://${NODE1}:2380 --listen-peer-urls http://0.0.0.0:2380 \
103  --advertise-client-urls http://${NODE1}:2379 --listen-client-urls http://0.0.0.0:2379 \
104  --initial-cluster node1=http://${NODE1}:2380
105```
106
107List the cluster member:
108
109```
110etcdctl --endpoints=http://${NODE1}:2379 member list
111```
112
113### Running a 3 node etcd cluster
114
115```
116REGISTRY=quay.io/coreos/etcd
117# available from v3.2.5
118REGISTRY=gcr.io/etcd-development/etcd
119
120# For each machine
121ETCD_VERSION=latest
122TOKEN=my-etcd-token
123CLUSTER_STATE=new
124NAME_1=etcd-node-0
125NAME_2=etcd-node-1
126NAME_3=etcd-node-2
127HOST_1=10.20.30.1
128HOST_2=10.20.30.2
129HOST_3=10.20.30.3
130CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380
131DATA_DIR=/var/lib/etcd
132
133# For node 1
134THIS_NAME=${NAME_1}
135THIS_IP=${HOST_1}
136docker run \
137  -p 2379:2379 \
138  -p 2380:2380 \
139  --volume=${DATA_DIR}:/etcd-data \
140  --name etcd ${REGISTRY}:${ETCD_VERSION} \
141  /usr/local/bin/etcd \
142  --data-dir=/etcd-data --name ${THIS_NAME} \
143  --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://0.0.0.0:2380 \
144  --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://0.0.0.0:2379 \
145  --initial-cluster ${CLUSTER} \
146  --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}
147
148# For node 2
149THIS_NAME=${NAME_2}
150THIS_IP=${HOST_2}
151docker run \
152  -p 2379:2379 \
153  -p 2380:2380 \
154  --volume=${DATA_DIR}:/etcd-data \
155  --name etcd ${REGISTRY}:${ETCD_VERSION} \
156  /usr/local/bin/etcd \
157  --data-dir=/etcd-data --name ${THIS_NAME} \
158  --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://0.0.0.0:2380 \
159  --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://0.0.0.0:2379 \
160  --initial-cluster ${CLUSTER} \
161  --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}
162
163# For node 3
164THIS_NAME=${NAME_3}
165THIS_IP=${HOST_3}
166docker run \
167  -p 2379:2379 \
168  -p 2380:2380 \
169  --volume=${DATA_DIR}:/etcd-data \
170  --name etcd ${REGISTRY}:${ETCD_VERSION} \
171  /usr/local/bin/etcd \
172  --data-dir=/etcd-data --name ${THIS_NAME} \
173  --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://0.0.0.0:2380 \
174  --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://0.0.0.0:2379 \
175  --initial-cluster ${CLUSTER} \
176  --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}
177```
178
179To run `etcdctl` using API version 3:
180
181```
182docker exec etcd /bin/sh -c "export ETCDCTL_API=3 && /usr/local/bin/etcdctl put foo bar"
183```
184
185## Bare Metal
186
187To provision a 3 node etcd cluster on bare-metal, the examples in the [baremetal repo](https://github.com/coreos/coreos-baremetal/tree/master/examples) may be useful.
188
189## Mounting a certificate volume
190
191The etcd release container does not include default root certificates. To use HTTPS with certificates trusted by a root authority (e.g., for discovery), mount a certificate directory into the etcd container:
192
193```
194REGISTRY=quay.io/coreos/etcd
195# available from v3.2.5
196REGISTRY=docker://gcr.io/etcd-development/etcd
197
198rkt run \
199  --insecure-options=image \
200  --volume etcd-ssl-certs-bundle,kind=host,source=/etc/ssl/certs/ca-certificates.crt \
201  --mount volume=etcd-ssl-certs-bundle,target=/etc/ssl/certs/ca-certificates.crt \
202  ${REGISTRY}:latest -- --name my-name \
203  --initial-advertise-peer-urls http://localhost:2380 --listen-peer-urls http://localhost:2380 \
204  --advertise-client-urls http://localhost:2379 --listen-client-urls http://localhost:2379 \
205  --discovery https://discovery.etcd.io/c11fbcdc16972e45253491a24fcf45e1
206```
207
208```
209REGISTRY=quay.io/coreos/etcd
210# available from v3.2.5
211REGISTRY=gcr.io/etcd-development/etcd
212
213docker run \
214  -p 2379:2379 \
215  -p 2380:2380 \
216  --volume=/etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt \
217  ${REGISTRY}:latest \
218  /usr/local/bin/etcd --name my-name \
219  --initial-advertise-peer-urls http://localhost:2380 --listen-peer-urls http://localhost:2380 \
220  --advertise-client-urls http://localhost:2379 --listen-client-urls http://localhost:2379 \
221  --discovery https://discovery.etcd.io/86a9ff6c8cb8b4c4544c1a2f88f8b801
222```
223