1**This is the documentation for etcd2 releases. Read [etcd3 doc][v3-docs] for etcd3 releases.**
2
3[v3-docs]: ../docs.md#documentation
4
5
6## Upgrade etcd from 2.2 to 2.3
7
8In the general case, upgrading from etcd 2.2 to 2.3 can be a zero-downtime, rolling upgrade:
9 - one by one, stop the etcd v2.2 processes and replace them with etcd v2.3 processes
10 - after running all v2.3 processes, new features in v2.3 are available to the cluster
11
12Before [starting an upgrade](#upgrade-procedure), read through the rest of this guide to prepare.
13
14### Upgrade Checklists
15
16#### Upgrade Requirements
17
18To upgrade an existing etcd deployment to 2.3, the running cluster must be 2.2 or greater. If it's before 2.2, please upgrade to [2.2](https://github.com/coreos/etcd/releases/tag/v2.2.0) before upgrading to 2.3.
19
20Also, to ensure a smooth rolling upgrade, the running cluster must be healthy. You can check the health of the cluster by using the `etcdctl cluster-health` command.
21
22#### Preparation
23
24Before upgrading etcd, always test the services relying on etcd in a staging environment before deploying the upgrade to the production environment.
25
26Before beginning,  [backup the etcd data directory](admin_guide.md#backing-up-the-datastore). Should something go wrong with the upgrade, it is possible to use this backup to[downgrade](#downgrade) back to existing etcd version.
27
28#### Mixed Versions
29
30While upgrading, an etcd cluster supports mixed versions of etcd members, and operates with the protocol of the lowest common version. The cluster is only considered upgraded once all of its members are upgraded to version 2.3. Internally, etcd members negotiate with each other to determine the overall cluster version, which controls the reported version and the supported features.
31
32#### Limitations
33
34It might take up to 2 minutes for the newly upgraded member to catch up with the existing cluster when the total data size is larger than 50MB. Check the size of a recent  snapshot to estimate  the total data size. In other words, it is safest to wait for 2 minutes between upgrading each member.
35
36For a much larger total data size, 100MB or more , this one-time process might take even more time. Administrators of very large etcd clusters of this magnitude can feel free to contact the [etcd team][etcd-contact] before upgrading, and we’ll be happy to provide advice on the procedure.
37
38#### Downgrade
39
40If all members have been upgraded to v2.3, the cluster will be upgraded to v2.3, and downgrade from this completed state is **not possible**. If any single member is still v2.2, however, the cluster and its operations remains “v2.2”, and it is possible from this mixed cluster state to return to using a v2.2 etcd binary on all members.
41
42Please [backup the data directory](admin_guide.md#backing-up-the-datastore) of all etcd members to make downgrading the cluster possible even after it has been completely upgraded.
43
44### Upgrade Procedure
45
46
47This example details the  upgrade of a three-member v2.2 ectd cluster running on a local machine.
48
49#### 1. Check upgrade requirements.
50
51Is the the cluster healthy and running v.2.2.x?
52
53```
54$ etcdctl cluster-health
55member 6e3bd23ae5f1eae0 is healthy: got healthy result from http://localhost:22379
56member 924e2e83e93f2560 is healthy: got healthy result from http://localhost:32379
57member a8266ecf031671f3 is healthy: got healthy result from http://localhost:12379
58cluster is healthy
59
60$ curl http://localhost:4001/version
61{"etcdserver":"2.2.x","etcdcluster":"2.2.0"}
62```
63
64#### 2. Stop the existing etcd process
65
66When each etcd process is stopped, expected errors will be logged by other cluster members. This is normal since a cluster member connection has been (temporarily) broken:
67
68```
692016-03-11 09:50:49.860319 E | rafthttp: failed to read 8211f1d0f64f3269 on stream Message (unexpected EOF)
702016-03-11 09:50:49.860335 I | rafthttp: the connection with 8211f1d0f64f3269 became inactive
712016-03-11 09:50:51.023804 W | etcdserver: failed to reach the peerURL(http://127.0.0.1:12380) of member 8211f1d0f64f3269 (Get http://127.0.0.1:12380/version: dial tcp 127.0.0.1:12380: getsockopt: connection refused)
722016-03-11 09:50:51.023821 W | etcdserver: cannot get the version of member 8211f1d0f64f3269 (Get http://127.0.0.1:12380/version: dial tcp 127.0.0.1:12380: getsockopt: connection refused)
73```
74
75It’s a good idea at this point to  [backup the etcd data directory](https://github.com/coreos/etcd/blob/7f7e2cc79d9c5c342a6eb1e48c386b0223cf934e/Documentation/admin_guide.md#backing-up-the-datastore) to provide a downgrade path should any problems occur:
76
77```
78$ etcdctl backup \
79      --data-dir /var/lib/etcd \
80      --backup-dir /tmp/etcd_backup
81```
82
83#### 3. Drop-in etcd v2.3 binary and start the new etcd process
84
85The new v2.3 etcd will publish its information to the cluster:
86
87```
8809:58:25.938673 I | etcdserver: published {Name:infra1 ClientURLs:[http://localhost:12379]} to cluster 524400597fb1d5f6
89```
90
91Verify that each member, and then the entire cluster, becomes healthy with the new v2.3 etcd binary:
92
93```
94$ etcdctl cluster-health
95member 6e3bd23ae5f1eae0 is healthy: got healthy result from http://localhost:22379
96member 924e2e83e93f2560 is healthy: got healthy result from http://localhost:32379
97member a8266ecf031671f3 is healthy: got healthy result from http://localhost:12379
98cluster is healthy
99```
100
101
102Upgraded members will log warnings like the following until the entire cluster is upgraded. This is expected and will cease after all etcd cluster members are upgraded to v2.3:
103
104```
1052016-03-11 09:58:26.851837 W | etcdserver: the local etcd version 2.2.0 is not up-to-date
1062016-03-11 09:58:26.851854 W | etcdserver: member c02c70ede158499f has a higher version 2.3.0
107```
108
109#### 4. Repeat step 2 to step 3 for all other members
110
111#### 5. Finish
112
113When all members are upgraded, the cluster will report  upgrading to 2.3 successfully:
114
115```
1162016-03-11 10:03:01.583392 N | etcdserver: updated the cluster version from 2.2 to 2.3
117```
118
119```
120$ curl http://127.0.0.1:4001/version
121{"etcdserver":"2.3.x","etcdcluster":"2.3.0"}
122```
123
124
125[etcd-contact]: https://coreos.com/etcd/?
126
127