1--- 2title: Runtime reconfiguration 3--- 4 5etcd comes with support for incremental runtime reconfiguration, which allows users to update the membership of the cluster at run time. 6 7Reconfiguration requests can only be processed when a majority of cluster members are functioning. It is **highly recommended** to always have a cluster size greater than two in production. It is unsafe to remove a member from a two member cluster. The majority of a two member cluster is also two. If there is a failure during the removal process, the cluster might not be able to make progress and need to [restart from majority failure][majority failure]. 8 9To better understand the design behind runtime reconfiguration, please read [the runtime reconfiguration document][runtime-reconf]. 10 11## Reconfiguration use cases 12 13This section will walk through some common reasons for reconfiguring a cluster. Most of these reasons just involve combinations of adding or removing a member, which are explained below under [Cluster Reconfiguration Operations][cluster-reconf]. 14 15### Cycle or upgrade multiple machines 16 17If multiple cluster members need to move due to planned maintenance (hardware upgrades, network downtime, etc.), it is recommended to modify members one at a time. 18 19It is safe to remove the leader, however there is a brief period of downtime while the election process takes place. If the cluster holds more than 50MB of v2 data, it is recommended to [migrate the member's data directory][member migration]. 20 21### Change the cluster size 22 23Increasing the cluster size can enhance [failure tolerance][fault tolerance table] and provide better read performance. Since clients can read from any member, increasing the number of members increases the overall serialized read throughput. 24 25Decreasing the cluster size can improve the write performance of a cluster, with a trade-off of decreased resilience. Writes into the cluster are replicated to a majority of members of the cluster before considered committed. Decreasing the cluster size lowers the majority, and each write is committed more quickly. 26 27### Replace a failed machine 28 29If a machine fails due to hardware failure, data directory corruption, or some other fatal situation, it should be replaced as soon as possible. Machines that have failed but haven't been removed adversely affect the quorum and reduce the tolerance for an additional failure. 30 31To replace the machine, follow the instructions for [removing the member][remove member] from the cluster, and then [add a new member][add member] in its place. If the cluster holds more than 50MB, it is recommended to [migrate the failed member's data directory][member migration] if it is still accessible. 32 33### Restart cluster from majority failure 34 35If the majority of the cluster is lost or all of the nodes have changed IP addresses, then manual action is necessary to recover safely. The basic steps in the recovery process include [creating a new cluster using the old data][disaster recovery], forcing a single member to act as the leader, and finally using runtime configuration to [add new members][add member] to this new cluster one at a time. 36 37## Cluster reconfiguration operations 38 39With these use cases in mind, the involved operations can be described for each. 40 41Before making any change, a simple majority (quorum) of etcd members must be available. This is essentially the same requirement for any kind of write to etcd. 42 43All changes to the cluster must be done sequentially: 44 45* To update a single member peerURLs, issue an update operation 46* To replace a healthy single member, remove the old member then add a new member 47* To increase from 3 to 5 members, issue two add operations 48* To decrease from 5 to 3, issue two remove operations 49 50All of these examples use the `etcdctl` command line tool that ships with etcd. To change membership without `etcdctl`, use the [v2 HTTP members API][member-api] or the [v3 gRPC members API][member-api-grpc]. 51 52### Update a member 53 54#### Update advertise client URLs 55 56To update the advertise client URLs of a member, simply restart that member with updated client urls flag (`--advertise-client-urls`) or environment variable (`ETCD_ADVERTISE_CLIENT_URLS`). The restarted member will self publish the updated URLs. A wrongly updated client URL will not affect the health of the etcd cluster. 57 58#### Update advertise peer URLs 59 60To update the advertise peer URLs of a member, first update it explicitly via member command and then restart the member. The additional action is required since updating peer URLs changes the cluster wide configuration and can affect the health of the etcd cluster. 61 62To update the advertise peer URLs, first find the target member's ID. To list all members with `etcdctl`: 63 64```sh 65$ etcdctl member list 666e3bd23ae5f1eae0: name=node2 peerURLs=http://localhost:23802 clientURLs=http://127.0.0.1:23792 67924e2e83e93f2560: name=node3 peerURLs=http://localhost:23803 clientURLs=http://127.0.0.1:23793 68a8266ecf031671f3: name=node1 peerURLs=http://localhost:23801 clientURLs=http://127.0.0.1:23791 69``` 70 71This example will `update` a8266ecf031671f3 member ID and change its peerURLs value to `http://10.0.1.10:2380`: 72 73```sh 74$ etcdctl member update a8266ecf031671f3 --peer-urls=http://10.0.1.10:2380 75Updated member with ID a8266ecf031671f3 in cluster 76``` 77 78### Remove a member 79 80Suppose the member ID to remove is a8266ecf031671f3. Use the `remove` command to perform the removal: 81 82```sh 83$ etcdctl member remove a8266ecf031671f3 84Removed member a8266ecf031671f3 from cluster 85``` 86 87The target member will stop itself at this point and print out the removal in the log: 88 89``` 90etcd: this member has been permanently removed from the cluster. Exiting. 91``` 92 93It is safe to remove the leader, however the cluster will be inactive while a new leader is elected. This duration is normally the period of election timeout plus the voting process. 94 95### Add a new member 96 97Adding a member is a two step process: 98 99 * Add the new member to the cluster via the [HTTP members API][member-api], the [gRPC members API][member-api-grpc], or the `etcdctl member add` command. 100 * Start the new member with the new cluster configuration, including a list of the updated members (existing members + the new member). 101 102`etcdctl` adds a new member to the cluster by specifying the member's [name][conf-name] and [advertised peer URLs][conf-adv-peer]: 103 104```sh 105$ etcdctl member add infra3 --peer-urls=http://10.0.1.13:2380 106added member 9bf1b35fc7761a23 to cluster 107 108ETCD_NAME="infra3" 109ETCD_INITIAL_CLUSTER="infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380,infra3=http://10.0.1.13:2380" 110ETCD_INITIAL_CLUSTER_STATE=existing 111``` 112 113`etcdctl` has informed the cluster about the new member and printed out the environment variables needed to successfully start it. Now start the new etcd process with the relevant flags for the new member: 114 115```sh 116$ export ETCD_NAME="infra3" 117$ export ETCD_INITIAL_CLUSTER="infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380,infra3=http://10.0.1.13:2380" 118$ export ETCD_INITIAL_CLUSTER_STATE=existing 119$ etcd --listen-client-urls http://10.0.1.13:2379 --advertise-client-urls http://10.0.1.13:2379 --listen-peer-urls http://10.0.1.13:2380 --initial-advertise-peer-urls http://10.0.1.13:2380 --data-dir %data_dir% 120``` 121 122The new member will run as a part of the cluster and immediately begin catching up with the rest of the cluster. 123 124If adding multiple members the best practice is to configure a single member at a time and verify it starts correctly before adding more new members. If adding a new member to a 1-node cluster, the cluster cannot make progress before the new member starts because it needs two members as majority to agree on the consensus. This behavior only happens between the time `etcdctl member add` informs the cluster about the new member and the new member successfully establishing a connection to the existing one. 125 126#### Add a new member as learner 127 128Starting from v3.4, etcd supports adding a new member as learner / non-voting member. 129The motivation and design can be found in [design doc][design-learner]. 130In order to make the process of adding a new member safer, 131and to reduce cluster downtime when the new member is added, it is recommended that the new member is added to cluster 132as a learner until it catches up. This can be described as a three step process: 133 134 * Add the new member as learner via [gRPC members API][member-api-grpc] or the `etcdctl member add --learner` command. 135 136 * Start the new member with the new cluster configuration, including a list of the updated members (existing members + the new member). 137 This step is exactly the same as before. 138 139 * Promote the newly added learner to voting member via [gRPC members API][member-api-grpc] or the `etcdctl member promote` command. 140 etcd server validates promote request to ensure its operational safety. 141 Only after its raft log has caught up to leader’s can learner be promoted to a voting member. 142 If a learner member has not caught up to leader's raft log, member promote request will fail 143 (see [error cases when promoting a member] section for more details). 144 In this case, user should wait and retry later. 145 146In v3.4, etcd server limits the number of learners that cluster can have to one. The main consideration is to limit the 147extra workload on leader due to propagating data from leader to learner. 148 149Use `etcdctl member add` with flag `--learner` to add new member to cluster as learner. 150 151```sh 152$ etcdctl member add infra3 --peer-urls=http://10.0.1.13:2380 --learner 153Member 9bf1b35fc7761a23 added to cluster a7ef944b95711739 154 155ETCD_NAME="infra3" 156ETCD_INITIAL_CLUSTER="infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380,infra3=http://10.0.1.13:2380" 157ETCD_INITIAL_CLUSTER_STATE=existing 158``` 159 160After new etcd process is started for the newly added learner member, use `etcdctl member promote` to promote learner to voting member. 161``` 162$ etcdctl member promote 9bf1b35fc7761a23 163Member 9e29bbaa45d74461 promoted in cluster a7ef944b95711739 164``` 165 166#### Error cases when adding members 167 168In the following case a new host is not included in the list of enumerated nodes. If this is a new cluster, the node must be added to the list of initial cluster members. 169 170```sh 171$ etcd --name infra3 \ 172 --initial-cluster infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380 \ 173 --initial-cluster-state existing 174etcdserver: assign ids error: the member count is unequal 175exit 1 176``` 177 178In this case, give a different address (10.0.1.14:2380) from the one used to join the cluster (10.0.1.13:2380): 179 180```sh 181$ etcd --name infra4 \ 182 --initial-cluster infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380,infra4=http://10.0.1.14:2380 \ 183 --initial-cluster-state existing 184etcdserver: assign ids error: unmatched member while checking PeerURLs 185exit 1 186``` 187 188If etcd starts using the data directory of a removed member, etcd automatically exits if it connects to any active member in the cluster: 189 190```sh 191$ etcd 192etcd: this member has been permanently removed from the cluster. Exiting. 193exit 1 194``` 195 196#### Error cases when adding a learner member 197 198Cannot add learner to cluster if the cluster already has 1 learner (v3.4). 199``` 200$ etcdctl member add infra4 --peer-urls=http://10.0.1.14:2380 --learner 201Error: etcdserver: too many learner members in cluster 202``` 203 204#### Error cases when promoting a learner member 205 206Learner can only be promoted to voting member if it is in sync with leader. 207``` 208$ etcdctl member promote 9bf1b35fc7761a23 209Error: etcdserver: can only promote a learner member which is in sync with leader 210``` 211 212Promoting a member that is not a learner will fail. 213``` 214$ etcdctl member promote 9bf1b35fc7761a23 215Error: etcdserver: can only promote a learner member 216``` 217 218Promoting a member that does not exist in cluster will fail. 219``` 220$ etcdctl member promote 12345abcde 221Error: etcdserver: member not found 222``` 223 224 225### Strict reconfiguration check mode (`-strict-reconfig-check`) 226 227As described in the above, the best practice of adding new members is to configure a single member at a time and verify it starts correctly before adding more new members. This step by step approach is very important because if newly added members is not configured correctly (for example the peer URLs are incorrect), the cluster can lose quorum. The quorum loss happens since the newly added member are counted in the quorum even if that member is not reachable from other existing members. Also quorum loss might happen if there is a connectivity issue or there are operational issues. 228 229For avoiding this problem, etcd provides an option `-strict-reconfig-check`. If this option is passed to etcd, etcd rejects reconfiguration requests if the number of started members will be less than a quorum of the reconfigured cluster. 230 231It is enabled by default. 232 233[add member]: #add-a-new-member 234[cluster-reconf]: #cluster-reconfiguration-operations 235[conf-adv-peer]: configuration.md#-initial-advertise-peer-urls 236[conf-name]: configuration.md#-name 237[disaster recovery]: recovery.md 238[fault tolerance table]: ../v2/admin_guide.md#fault-tolerance-table 239[majority failure]: #restart-cluster-from-majority-failure 240[member-api]: ../v2/members_api.md 241[member-api-grpc]: ../dev-guide/api_reference_v3.md#service-cluster-etcdserveretcdserverpbrpcproto 242[member migration]: ../v2/admin_guide.md#member-migration 243[remove member]: #remove-a-member 244[runtime-reconf]: runtime-reconf-design.md 245[error cases when promoting a member]: #error-cases-when-promoting-a-learner-member 246[design-learner]: ../learning/design-learner.md 247