1---
2title: gRPC proxy
3---
4
5The gRPC proxy is a stateless etcd reverse proxy operating at the gRPC layer (L7). The proxy is designed to reduce the total processing load on the core etcd cluster. For horizontal scalability, it coalesces watch and lease API requests. To protect the cluster against abusive clients, it caches key range requests.
6
7The gRPC proxy supports multiple etcd server endpoints. When the proxy starts, it randomly picks one etcd server endpoint to use. This endpoint serves all requests until the proxy detects an endpoint failure. If the gRPC proxy detects an endpoint failure, it switches to a different endpoint, if available, to hide failures from its clients. Other retry policies, such as weighted round-robin, may be supported in the future.
8
9## Scalable watch API
10
11The gRPC proxy coalesces multiple client watchers (`c-watchers`) on the same key or range into a single watcher (`s-watcher`) connected to an etcd server. The proxy broadcasts all events from the `s-watcher` to its `c-watchers`.
12
13Assuming N clients watch the same key, one gRPC proxy can reduce the watch load on the etcd server from N to 1. Users can deploy multiple gRPC proxies to further distribute server load.
14
15In the following example, three clients watch on key A. The gRPC proxy coalesces the three watchers, creating a single  watcher attached to the etcd server.
16
17```
18            +-------------+
19            | etcd server |
20            +------+------+
21                   ^ watch key A (s-watcher)
22                   |
23           +-------+-----+
24           | gRPC proxy  | <-------+
25           |             |         |
26           ++-----+------+         |watch key A (c-watcher)
27watch key A ^     ^ watch key A    |
28(c-watcher) |     | (c-watcher)    |
29    +-------+-+  ++--------+  +----+----+
30    |  client |  |  client |  |  client |
31    |         |  |         |  |         |
32    +---------+  +---------+  +---------+
33```
34
35### Limitations
36
37To effectively coalesce multiple client watchers into a single watcher, the gRPC proxy coalesces new `c-watchers` into an existing `s-watcher` when possible. This coalesced `s-watcher` may be out of sync with the etcd server due to network delays or buffered undelivered events. When the watch revision is unspecified, the gRPC proxy will not guarantee the `c-watcher` will start watching from the most recent store revision. For example, if a client watches from an etcd server with revision 1000, that watcher will begin at revision 1000. If a client watches from the gRPC proxy, may begin watching from revision 990.
38
39Similar limitations apply to cancellation. When the watcher is cancelled, the etcd server’s revision may be greater than the cancellation response revision.
40
41These two limitations should not cause problems for most use cases. In the future, there may be additional options to force the watcher to bypass the gRPC proxy for more accurate revision responses.
42
43## Scalable lease API
44
45To keep its leases alive, a client must establish at least one gRPC stream to an etcd server for sending periodic heartbeats. If an etcd workload involves heavy lease activity spread over many clients, these streams may contribute to excessive CPU utilization. To reduce the total number of streams on the core cluster, the proxy supports lease stream coalescing.
46
47Assuming N clients are updating leases, a single gRPC proxy reduces the stream load on the etcd server from N to 1. Deployments may have additional gRPC proxies to further distribute streams across multiple proxies.
48
49In the following example, three clients update three independent leases (`L1`, `L2`, and `L3`). The gRPC proxy coalesces the three client lease streams (`c-streams`) into a single lease keep alive stream (`s-stream`) attached to an etcd server. The proxy forwards client-side lease heartbeats from the c-streams to the s-stream, then returns the responses to the corresponding c-streams.
50
51```
52          +-------------+
53          | etcd server |
54          +------+------+
55                 ^
56                 | heartbeat L1, L2, L3
57                 | (s-stream)
58                 v
59         +-------+-----+
60         | gRPC proxy  +<-----------+
61         +---+------+--+            | heartbeat L3
62             ^      ^               | (c-stream)
63heartbeat L1 |      | heartbeat L2  |
64(c-stream)   v      v (c-stream)    v
65      +------+-+  +-+------+  +-----+--+
66      | client |  | client |  | client |
67      +--------+  +--------+  +--------+
68```
69
70## Abusive clients protection
71
72The gRPC proxy caches responses for requests when it does not break consistency requirements. This can protect the etcd server from abusive clients in tight for loops.
73
74## Start etcd gRPC proxy
75
76Consider an etcd cluster with the following static endpoints:
77
78|Name|Address|Hostname|
79|------|---------|------------------|
80|infra0|10.0.1.10|infra0.example.com|
81|infra1|10.0.1.11|infra1.example.com|
82|infra2|10.0.1.12|infra2.example.com|
83
84Start the etcd gRPC proxy to use these static endpoints with the command:
85
86```bash
87$ etcd grpc-proxy start --endpoints=infra0.example.com,infra1.example.com,infra2.example.com --listen-addr=127.0.0.1:2379
88```
89
90The etcd gRPC proxy starts and listens on port 2379. It forwards client requests to one of the three endpoints provided above.
91
92Sending requests through the proxy:
93
94```bash
95$ ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 put foo bar
96OK
97$ ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 get foo
98foo
99bar
100```
101
102## Client endpoint synchronization and name resolution
103
104The proxy supports registering its endpoints for discovery by writing to a user-defined endpoint. This serves two purposes. First, it allows clients to synchronize their endpoints against a set of proxy endpoints for high availability. Second, it is an endpoint provider for etcd [gRPC naming](../dev-guide/grpc_naming.md).
105
106Register proxy(s) by providing a user-defined prefix:
107
108```bash
109$ etcd grpc-proxy start --endpoints=localhost:2379 \
110  --listen-addr=127.0.0.1:23790 \
111  --advertise-client-url=127.0.0.1:23790 \
112  --resolver-prefix="___grpc_proxy_endpoint" \
113  --resolver-ttl=60
114
115$ etcd grpc-proxy start --endpoints=localhost:2379 \
116  --listen-addr=127.0.0.1:23791 \
117  --advertise-client-url=127.0.0.1:23791 \
118  --resolver-prefix="___grpc_proxy_endpoint" \
119  --resolver-ttl=60
120```
121
122The proxy will list all its members for member list:
123
124```bash
125ETCDCTL_API=3 etcdctl --endpoints=http://localhost:23790 member list --write-out table
126
127+----+---------+--------------------------------+------------+-----------------+
128| ID | STATUS  |              NAME              | PEER ADDRS |  CLIENT ADDRS   |
129+----+---------+--------------------------------+------------+-----------------+
130|  0 | started | Gyu-Hos-MBP.sfo.coreos.systems |            | 127.0.0.1:23791 |
131|  0 | started | Gyu-Hos-MBP.sfo.coreos.systems |            | 127.0.0.1:23790 |
132+----+---------+--------------------------------+------------+-----------------+
133```
134
135This lets clients automatically discover proxy endpoints through Sync:
136
137```go
138cli, err := clientv3.New(clientv3.Config{
139    Endpoints: []string{"http://localhost:23790"},
140})
141if err != nil {
142    log.Fatal(err)
143}
144defer cli.Close()
145
146// fetch registered grpc-proxy endpoints
147if err := cli.Sync(context.Background()); err != nil {
148    log.Fatal(err)
149}
150```
151
152Note that if a proxy is configured without a resolver prefix,
153
154```bash
155$ etcd grpc-proxy start --endpoints=localhost:2379 \
156  --listen-addr=127.0.0.1:23792 \
157  --advertise-client-url=127.0.0.1:23792
158```
159
160The member list API to the grpc-proxy returns its own `advertise-client-url`:
161
162```bash
163ETCDCTL_API=3 etcdctl --endpoints=http://localhost:23792 member list --write-out table
164
165+----+---------+--------------------------------+------------+-----------------+
166| ID | STATUS  |              NAME              | PEER ADDRS |  CLIENT ADDRS   |
167+----+---------+--------------------------------+------------+-----------------+
168|  0 | started | Gyu-Hos-MBP.sfo.coreos.systems |            | 127.0.0.1:23792 |
169+----+---------+--------------------------------+------------+-----------------+
170```
171
172## Namespacing
173
174Suppose an application expects full control over the entire key space, but the etcd cluster is shared with other applications. To let all appications run without interfering with each other, the proxy can partition the etcd keyspace so clients appear to have access to the complete keyspace. When the proxy is given the flag `--namespace`, all client requests going into the proxy are translated to have a user-defined prefix on the keys. Accesses to the etcd cluster will be under the prefix and responses from the proxy will strip away the prefix; to the client, it appears as if there is no prefix at all.
175
176To namespace a proxy, start it with `--namespace`:
177
178```bash
179$ etcd grpc-proxy start --endpoints=localhost:2379 \
180  --listen-addr=127.0.0.1:23790 \
181  --namespace=my-prefix/
182```
183
184Accesses to the proxy are now transparently prefixed on the etcd cluster:
185
186```bash
187$ ETCDCTL_API=3 etcdctl --endpoints=localhost:23790 put my-key abc
188# OK
189$ ETCDCTL_API=3 etcdctl --endpoints=localhost:23790 get my-key
190# my-key
191# abc
192$ ETCDCTL_API=3 etcdctl --endpoints=localhost:2379 get my-prefix/my-key
193# my-prefix/my-key
194# abc
195```
196
197## TLS termination
198
199Terminate TLS from a secure etcd cluster with the gRPC proxy by serving an unencrypted local endpoint.
200
201To try it out, start a single member etcd cluster with client https:
202
203```sh
204$ etcd --listen-client-urls https://localhost:2379 --advertise-client-urls https://localhost:2379 --cert-file=peer.crt --key-file=peer.key --trusted-ca-file=ca.crt --client-cert-auth
205```
206
207Confirm the client port is serving https:
208
209```sh
210# fails
211$ ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 endpoint status
212# works
213$ ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 --cert=client.crt --key=client.key --cacert=ca.crt endpoint status
214```
215
216Next, start a gRPC proxy on `localhost:12379` by connecting to the etcd endpoint `https://localhost:2379` using the client certificates:
217
218```sh
219$ etcd grpc-proxy start --endpoints=https://localhost:2379 --listen-addr localhost:12379 --cert client.crt --key client.key --cacert=ca.crt --insecure-skip-tls-verify &
220```
221
222Finally, test the TLS termination by putting a key into the proxy over http:
223
224```sh
225$ ETCDCTL_API=3 etcdctl --endpoints=http://localhost:12379 put abc def
226# OK
227```
228
229## Metrics and Health
230
231The gRPC proxy exposes `/health` and Prometheus `/metrics` endpoints for the etcd members defined by `--endpoints`. An alternative define an additional URL that will respond to both the `/metrics` and `/health` endpoints with the `--metrics-addr` flag.
232
233```bash
234$ etcd grpc-proxy start \
235  --endpoints https://localhost:2379 \
236  --metrics-addr https://0.0.0.0:4443 \
237  --listen-addr 127.0.0.1:23790 \
238  --key client.key \
239  --key-file proxy-server.key \
240  --cert client.crt \
241  --cert-file proxy-server.crt \
242  --cacert ca.pem \
243  --trusted-ca-file proxy-ca.pem
244 ```
245
246### Known issue
247
248The main interface of the proxy serves both HTTP2 and HTTP/1.1. If proxy is setup with TLS as show in the above example, when using a client such as cURL against the listening interface will require explicitly setting the protocol to HTTP/1.1 on the request to return `/metrics` or `/health`. By using the `--metrics-addr` flag the secondary interface will not have this requirement.
249
250```bash
251 $ curl --cacert proxy-ca.pem --key proxy-client.key --cert proxy-client.crt https://127.0.0.1:23790/metrics --http1.1
252```
253