1# gRPC proxy
2
3*This is an alpha feature, we are looking for early feedback.*
4
5The gRPC proxy is a stateless etcd reverse proxy operating at the gRPC layer (L7). The proxy is designed to reduce the total processing load on the core etcd cluster. For horizontal scalability, it coalesces watch and lease API requests. To protect the cluster against abusive clients, it caches key range requests.
6
7The gRPC proxy supports multiple etcd server endpoints. When the proxy starts, it randomly picks one etcd server endpoint to use. This endpoint serves all requests until the proxy detects an endpoint failure. If the gRPC proxy detects an endpoint failure, it switches to a different endpoint, if available, to hide failures from its clients. Other retry policies, such as weighted round-robin, may be supported in the future.
8
9## Scalable watch API
10
11The gRPC proxy coalesces multiple client watchers (`c-watchers`) on the same key or range into a single watcher (`s-watcher`) connected to an etcd server. The proxy broadcasts all events from the `s-watcher` to its `c-watchers`.
12
13Assuming N clients watch the same key, one gRPC proxy can reduce the watch load on the etcd server from N to 1. Users can deploy multiple gRPC proxies to further distribute server load.
14
15In the following example, three clients watch on key A. The gRPC proxy coalesces the three watchers, creating a single  watcher attached to the etcd server.
16
17```
18            +-------------+
19            | etcd server |
20            +------+------+
21                   ^ watch key A (s-watcher)
22                   |
23           +-------+-----+
24           | gRPC proxy  | <-------+
25           |             |         |
26           ++-----+------+         |watch key A (c-watcher)
27watch key A ^     ^ watch key A    |
28(c-watcher) |     | (c-watcher)    |
29    +-------+-+  ++--------+  +----+----+
30    |  client |  |  client |  |  client |
31    |         |  |         |  |         |
32    +---------+  +---------+  +---------+
33```
34
35### Limitations
36
37To effectively coalesce multiple client watchers into a single watcher, the gRPC proxy coalesces new `c-watchers` into an existing `s-watcher` when possible. This coalesced `s-watcher` may be out of sync with the etcd server due to network delays or buffered undelivered events. When the watch revision is unspecified, the gRPC proxy will not guarantee the `c-watcher` will start watching from the most recent store revision. For example, if a client watches from an etcd server with revision 1000, that watcher will begin at revision 1000. If a client watches from the gRPC proxy, may begin watching from revision 990.
38
39Similar limitations apply to cancellation. When the watcher is cancelled, the etcd server’s revision may be greater than the cancellation response revision.
40
41These two limitations should not cause problems for most use cases. In the future, there may be additional options to force the watcher to bypass the gRPC proxy for more accurate revision responses.
42
43## Scalable lease API
44
45TODO
46
47## Abusive clients protection
48
49The gRPC proxy caches responses for requests when it does not break consistency requirements. This can protect the etcd server from abusive clients in tight for loops.
50
51## Start etcd gRPC proxy
52
53Consider an etcd cluster with the following static endpoints:
54
55|Name|Address|Hostname|
56|------|---------|------------------|
57|infra0|10.0.1.10|infra0.example.com|
58|infra1|10.0.1.11|infra1.example.com|
59|infra2|10.0.1.12|infra2.example.com|
60
61Start the etcd gRPC proxy to use these static endpoints with the command:
62
63```bash
64$ etcd grpc-proxy start --endpoints=infra0.example.com,infra1.example.com,infra2.example.com --listen-addr=127.0.0.1:2379
65```
66
67The etcd gRPC proxy starts and listens on port 8080. It forwards client requests to one of the three endpoints provided above.
68
69Sending requests through the proxy:
70
71```bash
72$ ETCDCTL_API=3 ./etcdctl --endpoints=127.0.0.1:2379 put foo bar
73OK
74$ ETCDCTL_API=3 ./etcdctl --endpoints=127.0.0.1:2379 get foo
75foo
76bar
77```
78
79