1--- 2title: Monitoring etcd 3--- 4 5Each etcd server provides local monitoring information on its client port through http endpoints. The monitoring data is useful for both system health checking and cluster debugging. 6 7## Debug endpoint 8 9If `--debug` is set, the etcd server exports debugging information on its client port under the `/debug` path. Take care when setting `--debug`, since there will be degraded performance and verbose logging. 10 11The `/debug/pprof` endpoint is the standard go runtime profiling endpoint. This can be used to profile CPU, heap, mutex, and goroutine utilization. For example, here `go tool pprof` gets the top 10 functions where etcd spends its time: 12 13```sh 14$ go tool pprof http://localhost:2379/debug/pprof/profile 15Fetching profile from http://localhost:2379/debug/pprof/profile 16Please wait... (30s) 17Saved profile in /home/etcd/pprof/pprof.etcd.localhost:2379.samples.cpu.001.pb.gz 18Entering interactive mode (type "help" for commands) 19(pprof) top10 20310ms of 480ms total (64.58%) 21Showing top 10 nodes out of 157 (cum >= 10ms) 22 flat flat% sum% cum cum% 23 130ms 27.08% 27.08% 130ms 27.08% runtime.futex 24 70ms 14.58% 41.67% 70ms 14.58% syscall.Syscall 25 20ms 4.17% 45.83% 20ms 4.17% github.com/coreos/etcd/vendor/golang.org/x/net/http2/hpack.huffmanDecode 26 20ms 4.17% 50.00% 30ms 6.25% runtime.pcvalue 27 20ms 4.17% 54.17% 50ms 10.42% runtime.schedule 28 10ms 2.08% 56.25% 10ms 2.08% github.com/coreos/etcd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).AuthInfoFromCtx 29 10ms 2.08% 58.33% 10ms 2.08% github.com/coreos/etcd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).Lead 30 10ms 2.08% 60.42% 10ms 2.08% github.com/coreos/etcd/vendor/github.com/coreos/etcd/pkg/wait.(*timeList).Trigger 31 10ms 2.08% 62.50% 10ms 2.08% github.com/coreos/etcd/vendor/github.com/prometheus/client_golang/prometheus.(*MetricVec).hashLabelValues 32 10ms 2.08% 64.58% 10ms 2.08% github.com/coreos/etcd/vendor/golang.org/x/net/http2.(*Framer).WriteHeaders 33``` 34 35The `/debug/requests` endpoint gives gRPC traces and performance statistics through a web browser. For example, here is a `Range` request for the key `abc`: 36 37``` 38When Elapsed (s) 392017/08/18 17:34:51.999317 0.000244 /etcdserverpb.KV/Range 4017:34:51.999382 . 65 ... RPC: from 127.0.0.1:47204 deadline:4.999377747s 4117:34:51.999395 . 13 ... recv: key:"abc" 4217:34:51.999499 . 104 ... OK 4317:34:51.999535 . 36 ... sent: header:<cluster_id:14841639068965178418 member_id:10276657743932975437 revision:15 raft_term:17 > kvs:<key:"abc" create_revision:6 mod_revision:14 version:9 value:"asda" > count:1 44``` 45 46## Metrics endpoint 47 48Each etcd server exports metrics under the `/metrics` path on its client port and optionally on locations given by `--listen-metrics-urls`. 49 50The metrics can be fetched with `curl`: 51 52```sh 53$ curl -L http://localhost:2379/metrics | grep -v debugging # ignore unstable debugging metrics 54 55# HELP etcd_disk_backend_commit_duration_seconds The latency distributions of commit called by backend. 56# TYPE etcd_disk_backend_commit_duration_seconds histogram 57etcd_disk_backend_commit_duration_seconds_bucket{le="0.002"} 72756 58etcd_disk_backend_commit_duration_seconds_bucket{le="0.004"} 401587 59etcd_disk_backend_commit_duration_seconds_bucket{le="0.008"} 405979 60etcd_disk_backend_commit_duration_seconds_bucket{le="0.016"} 406464 61... 62``` 63 64## Health Check 65 66Since v3.3.0, in addition to responding to the `/metrics` endpoint, any locations specified by `--listen-metrics-urls` will also respond to the `/health` endpoint. This can be useful if the standard endpoint is configured with mutual (client) TLS authentication, but a load balancer or monitoring service still needs access to the health check. 67 68## Prometheus 69 70Running a [Prometheus][prometheus] monitoring service is the easiest way to ingest and record etcd's metrics. 71 72First, install Prometheus: 73 74```sh 75PROMETHEUS_VERSION="2.0.0" 76wget https://github.com/prometheus/prometheus/releases/download/v$PROMETHEUS_VERSION/prometheus-$PROMETHEUS_VERSION.linux-amd64.tar.gz -O /tmp/prometheus-$PROMETHEUS_VERSION.linux-amd64.tar.gz 77tar -xvzf /tmp/prometheus-$PROMETHEUS_VERSION.linux-amd64.tar.gz --directory /tmp/ --strip-components=1 78/tmp/prometheus -version 79``` 80 81Set Prometheus's scraper to target the etcd cluster endpoints: 82 83```sh 84cat > /tmp/test-etcd.yaml <<EOF 85global: 86 scrape_interval: 10s 87scrape_configs: 88 - job_name: test-etcd 89 static_configs: 90 - targets: ['10.240.0.32:2379','10.240.0.33:2379','10.240.0.34:2379'] 91EOF 92cat /tmp/test-etcd.yaml 93``` 94 95Set up the Prometheus handler: 96 97```sh 98nohup /tmp/prometheus \ 99 -config.file /tmp/test-etcd.yaml \ 100 -web.listen-address ":9090" \ 101 -storage.local.path "test-etcd.data" >> /tmp/test-etcd.log 2>&1 & 102``` 103 104Now Prometheus will scrape etcd metrics every 10 seconds. 105 106 107### Alerting 108 109There is a set of default alerts for etcd v3 clusters for [Prometheus 1.x](./etcd3_alert.rules) as well as [Prometheus 2.x](./etcd3_alert.rules.yml). 110 111> Note: `job` labels may need to be adjusted to fit a particular need. The rules were written to apply to a single cluster so it is recommended to choose labels unique to a cluster. 112 113### Grafana 114 115[Grafana][grafana] has built-in Prometheus support; just add a Prometheus data source: 116 117``` 118Name: test-etcd 119Type: Prometheus 120Url: http://localhost:9090 121Access: proxy 122``` 123 124Then import the default [etcd dashboard template][template] and customize. For instance, if Prometheus data source name is `my-etcd`, the `datasource` field values in JSON also need to be `my-etcd`. 125 126Sample dashboard: 127 128![](./etcd-sample-grafana.png) 129 130 131[prometheus]: https://prometheus.io/ 132[grafana]: http://grafana.org/ 133[template]: ./grafana.json 134