1**This is the documentation for etcd2 releases. Read [etcd3 doc][v3-docs] for etcd3 releases.** 2 3[v3-docs]: ../docs.md#documentation 4 5 6## Upgrade etcd from 2.2 to 2.3 7 8In the general case, upgrading from etcd 2.2 to 2.3 can be a zero-downtime, rolling upgrade: 9 - one by one, stop the etcd v2.2 processes and replace them with etcd v2.3 processes 10 - after running all v2.3 processes, new features in v2.3 are available to the cluster 11 12Before [starting an upgrade](#upgrade-procedure), read through the rest of this guide to prepare. 13 14### Upgrade Checklists 15 16#### Upgrade Requirements 17 18To upgrade an existing etcd deployment to 2.3, the running cluster must be 2.2 or greater. If it's before 2.2, please upgrade to [2.2](https://github.com/coreos/etcd/releases/tag/v2.2.0) before upgrading to 2.3. 19 20Also, to ensure a smooth rolling upgrade, the running cluster must be healthy. You can check the health of the cluster by using the `etcdctl cluster-health` command. 21 22#### Preparation 23 24Before upgrading etcd, always test the services relying on etcd in a staging environment before deploying the upgrade to the production environment. 25 26Before beginning, [backup the etcd data directory](admin_guide.md#backing-up-the-datastore). Should something go wrong with the upgrade, it is possible to use this backup to[downgrade](#downgrade) back to existing etcd version. 27 28#### Mixed Versions 29 30While upgrading, an etcd cluster supports mixed versions of etcd members, and operates with the protocol of the lowest common version. The cluster is only considered upgraded once all of its members are upgraded to version 2.3. Internally, etcd members negotiate with each other to determine the overall cluster version, which controls the reported version and the supported features. 31 32#### Limitations 33 34It might take up to 2 minutes for the newly upgraded member to catch up with the existing cluster when the total data size is larger than 50MB. Check the size of a recent snapshot to estimate the total data size. In other words, it is safest to wait for 2 minutes between upgrading each member. 35 36For a much larger total data size, 100MB or more , this one-time process might take even more time. Administrators of very large etcd clusters of this magnitude can feel free to contact the [etcd team][etcd-contact] before upgrading, and we’ll be happy to provide advice on the procedure. 37 38#### Downgrade 39 40If all members have been upgraded to v2.3, the cluster will be upgraded to v2.3, and downgrade from this completed state is **not possible**. If any single member is still v2.2, however, the cluster and its operations remains “v2.2”, and it is possible from this mixed cluster state to return to using a v2.2 etcd binary on all members. 41 42Please [backup the data directory](admin_guide.md#backing-up-the-datastore) of all etcd members to make downgrading the cluster possible even after it has been completely upgraded. 43 44### Upgrade Procedure 45 46 47This example details the upgrade of a three-member v2.2 ectd cluster running on a local machine. 48 49#### 1. Check upgrade requirements. 50 51Is the the cluster healthy and running v.2.2.x? 52 53``` 54$ etcdctl cluster-health 55member 6e3bd23ae5f1eae0 is healthy: got healthy result from http://localhost:22379 56member 924e2e83e93f2560 is healthy: got healthy result from http://localhost:32379 57member a8266ecf031671f3 is healthy: got healthy result from http://localhost:12379 58cluster is healthy 59 60$ curl http://localhost:4001/version 61{"etcdserver":"2.2.x","etcdcluster":"2.2.0"} 62``` 63 64#### 2. Stop the existing etcd process 65 66When each etcd process is stopped, expected errors will be logged by other cluster members. This is normal since a cluster member connection has been (temporarily) broken: 67 68``` 692016-03-11 09:50:49.860319 E | rafthttp: failed to read 8211f1d0f64f3269 on stream Message (unexpected EOF) 702016-03-11 09:50:49.860335 I | rafthttp: the connection with 8211f1d0f64f3269 became inactive 712016-03-11 09:50:51.023804 W | etcdserver: failed to reach the peerURL(http://127.0.0.1:12380) of member 8211f1d0f64f3269 (Get http://127.0.0.1:12380/version: dial tcp 127.0.0.1:12380: getsockopt: connection refused) 722016-03-11 09:50:51.023821 W | etcdserver: cannot get the version of member 8211f1d0f64f3269 (Get http://127.0.0.1:12380/version: dial tcp 127.0.0.1:12380: getsockopt: connection refused) 73``` 74 75It’s a good idea at this point to [backup the etcd data directory](https://github.com/coreos/etcd/blob/7f7e2cc79d9c5c342a6eb1e48c386b0223cf934e/Documentation/admin_guide.md#backing-up-the-datastore) to provide a downgrade path should any problems occur: 76 77``` 78$ etcdctl backup \ 79 --data-dir /var/lib/etcd \ 80 --backup-dir /tmp/etcd_backup 81``` 82 83#### 3. Drop-in etcd v2.3 binary and start the new etcd process 84 85The new v2.3 etcd will publish its information to the cluster: 86 87``` 8809:58:25.938673 I | etcdserver: published {Name:infra1 ClientURLs:[http://localhost:12379]} to cluster 524400597fb1d5f6 89``` 90 91Verify that each member, and then the entire cluster, becomes healthy with the new v2.3 etcd binary: 92 93``` 94$ etcdctl cluster-health 95member 6e3bd23ae5f1eae0 is healthy: got healthy result from http://localhost:22379 96member 924e2e83e93f2560 is healthy: got healthy result from http://localhost:32379 97member a8266ecf031671f3 is healthy: got healthy result from http://localhost:12379 98cluster is healthy 99``` 100 101 102Upgraded members will log warnings like the following until the entire cluster is upgraded. This is expected and will cease after all etcd cluster members are upgraded to v2.3: 103 104``` 1052016-03-11 09:58:26.851837 W | etcdserver: the local etcd version 2.2.0 is not up-to-date 1062016-03-11 09:58:26.851854 W | etcdserver: member c02c70ede158499f has a higher version 2.3.0 107``` 108 109#### 4. Repeat step 2 to step 3 for all other members 110 111#### 5. Finish 112 113When all members are upgraded, the cluster will report upgrading to 2.3 successfully: 114 115``` 1162016-03-11 10:03:01.583392 N | etcdserver: updated the cluster version from 2.2 to 2.3 117``` 118 119``` 120$ curl http://127.0.0.1:4001/version 121{"etcdserver":"2.3.x","etcdcluster":"2.3.0"} 122``` 123 124 125[etcd-contact]: https://coreos.com/etcd/? 126 127