1# Node exporter 2 3[![CircleCI](https://circleci.com/gh/prometheus/node_exporter/tree/master.svg?style=shield)][circleci] 4[![Buildkite status](https://badge.buildkite.com/94a0c1fb00b1f46883219c256efe9ce01d63b6505f3a942f9b.svg)](https://buildkite.com/prometheus/node-exporter) 5[![Docker Repository on Quay](https://quay.io/repository/prometheus/node-exporter/status)][quay] 6[![Docker Pulls](https://img.shields.io/docker/pulls/prom/node-exporter.svg?maxAge=604800)][hub] 7[![Go Report Card](https://goreportcard.com/badge/github.com/prometheus/node_exporter)][goreportcard] 8 9Prometheus exporter for hardware and OS metrics exposed by \*NIX kernels, written 10in Go with pluggable metric collectors. 11 12The [Windows exporter](https://github.com/prometheus-community/windows_exporter) is recommended for Windows users. 13To expose NVIDIA GPU metrics, [prometheus-dcgm 14](https://github.com/NVIDIA/gpu-monitoring-tools#dcgm-exporter) 15can be used. 16 17## Installation and Usage 18 19If you are new to Prometheus and `node_exporter` there is a [simple step-by-step guide](https://prometheus.io/docs/guides/node-exporter/). 20 21The `node_exporter` listens on HTTP port 9100 by default. See the `--help` output for more options. 22 23### Ansible 24 25For automated installs with [Ansible](https://www.ansible.com/), there is the [Cloud Alchemy role](https://github.com/cloudalchemy/ansible-node-exporter). 26 27### RHEL/CentOS/Fedora 28 29There is a [community-supplied COPR repository](https://copr.fedorainfracloud.org/coprs/ibotty/prometheus-exporters/) which closely follows upstream releases. 30 31### Docker 32 33The `node_exporter` is designed to monitor the host system. It's not recommended 34to deploy it as a Docker container because it requires access to the host system. 35 36For situations where Docker deployment is needed, some extra flags must be used to allow 37the `node_exporter` access to the host namespaces. 38 39Be aware that any non-root mount points you want to monitor will need to be bind-mounted 40into the container. 41 42If you start container for host monitoring, specify `path.rootfs` argument. 43This argument must match path in bind-mount of host root. The node\_exporter will use 44`path.rootfs` as prefix to access host filesystem. 45 46```bash 47docker run -d \ 48 --net="host" \ 49 --pid="host" \ 50 -v "/:/host:ro,rslave" \ 51 quay.io/prometheus/node-exporter:latest \ 52 --path.rootfs=/host 53``` 54 55For Docker compose, similar flag changes are needed. 56 57```yaml 58--- 59version: '3.8' 60 61services: 62 node_exporter: 63 image: quay.io/prometheus/node-exporter:latest 64 container_name: node_exporter 65 command: 66 - '--path.rootfs=/host' 67 network_mode: host 68 pid: host 69 restart: unless-stopped 70 volumes: 71 - '/:/host:ro,rslave' 72``` 73 74On some systems, the `timex` collector requires an additional Docker flag, 75`--cap-add=SYS_TIME`, in order to access the required syscalls. 76 77## Collectors 78 79There is varying support for collectors on each operating system. The tables 80below list all existing collectors and the supported systems. 81 82Collectors are enabled by providing a `--collector.<name>` flag. 83Collectors that are enabled by default can be disabled by providing a `--no-collector.<name>` flag. 84To enable only some specific collector(s), use `--collector.disable-defaults --collector.<name> ...`. 85 86### Enabled by default 87 88Name | Description | OS 89---------|-------------|---- 90arp | Exposes ARP statistics from `/proc/net/arp`. | Linux 91bcache | Exposes bcache statistics from `/sys/fs/bcache/`. | Linux 92bonding | Exposes the number of configured and active slaves of Linux bonding interfaces. | Linux 93btrfs | Exposes btrfs statistics | Linux 94boottime | Exposes system boot time derived from the `kern.boottime` sysctl. | Darwin, Dragonfly, FreeBSD, NetBSD, OpenBSD, Solaris 95conntrack | Shows conntrack statistics (does nothing if no `/proc/sys/net/netfilter/` present). | Linux 96cpu | Exposes CPU statistics | Darwin, Dragonfly, FreeBSD, Linux, Solaris, OpenBSD 97cpufreq | Exposes CPU frequency statistics | Linux, Solaris 98diskstats | Exposes disk I/O statistics. | Darwin, Linux, OpenBSD 99edac | Exposes error detection and correction statistics. | Linux 100entropy | Exposes available entropy. | Linux 101exec | Exposes execution statistics. | Dragonfly, FreeBSD 102fibrechannel | Exposes fibre channel information and statistics from `/sys/class/fc_host/`. | Linux 103filefd | Exposes file descriptor statistics from `/proc/sys/fs/file-nr`. | Linux 104filesystem | Exposes filesystem statistics, such as disk space used. | Darwin, Dragonfly, FreeBSD, Linux, OpenBSD 105hwmon | Expose hardware monitoring and sensor data from `/sys/class/hwmon/`. | Linux 106infiniband | Exposes network statistics specific to InfiniBand and Intel OmniPath configurations. | Linux 107ipvs | Exposes IPVS status from `/proc/net/ip_vs` and stats from `/proc/net/ip_vs_stats`. | Linux 108loadavg | Exposes load average. | Darwin, Dragonfly, FreeBSD, Linux, NetBSD, OpenBSD, Solaris 109mdadm | Exposes statistics about devices in `/proc/mdstat` (does nothing if no `/proc/mdstat` present). | Linux 110meminfo | Exposes memory statistics. | Darwin, Dragonfly, FreeBSD, Linux, OpenBSD 111netclass | Exposes network interface info from `/sys/class/net/` | Linux 112netdev | Exposes network interface statistics such as bytes transferred. | Darwin, Dragonfly, FreeBSD, Linux, OpenBSD 113netstat | Exposes network statistics from `/proc/net/netstat`. This is the same information as `netstat -s`. | Linux 114nfs | Exposes NFS client statistics from `/proc/net/rpc/nfs`. This is the same information as `nfsstat -c`. | Linux 115nfsd | Exposes NFS kernel server statistics from `/proc/net/rpc/nfsd`. This is the same information as `nfsstat -s`. | Linux 116nvme | Exposes NVMe info from `/sys/class/nvme/` | Linux 117powersupplyclass | Exposes Power Supply statistics from `/sys/class/power_supply` | Linux 118pressure | Exposes pressure stall statistics from `/proc/pressure/`. | Linux (kernel 4.20+ and/or [CONFIG\_PSI](https://www.kernel.org/doc/html/latest/accounting/psi.html)) 119rapl | Exposes various statistics from `/sys/class/powercap`. | Linux 120schedstat | Exposes task scheduler statistics from `/proc/schedstat`. | Linux 121sockstat | Exposes various statistics from `/proc/net/sockstat`. | Linux 122softnet | Exposes statistics from `/proc/net/softnet_stat`. | Linux 123stat | Exposes various statistics from `/proc/stat`. This includes boot time, forks and interrupts. | Linux 124tapestats | Exposes statistics from `/sys/class/scsi_tape`. | Linux 125textfile | Exposes statistics read from local disk. The `--collector.textfile.directory` flag must be set. | _any_ 126thermal\_zone | Exposes thermal zone & cooling device statistics from `/sys/class/thermal`. | Linux 127time | Exposes the current system time. | _any_ 128timex | Exposes selected adjtimex(2) system call stats. | Linux 129udp_queues | Exposes UDP total lengths of the rx_queue and tx_queue from `/proc/net/udp` and `/proc/net/udp6`. | Linux 130uname | Exposes system information as provided by the uname system call. | Darwin, FreeBSD, Linux, OpenBSD 131vmstat | Exposes statistics from `/proc/vmstat`. | Linux 132xfs | Exposes XFS runtime statistics. | Linux (kernel 4.4+) 133zfs | Exposes [ZFS](http://open-zfs.org/) performance statistics. | [Linux](http://zfsonlinux.org/), Solaris 134 135### Disabled by default 136 137`node_exporter` also implements a number of collectors that are disabled by default. Reasons for this vary by 138collector, and may include: 139* High cardinality 140* Prolonged runtime that exceeds the Prometheus `scrape_interval` or `scrape_timeout` 141* Significant resource demands on the host 142 143You can enable additional collectors as desired by adding them to your 144init system's or service supervisor's startup configuration for 145`node_exporter` but caution is advised. Enable at most one at a time, 146testing first on a non-production system, then by hand on a single 147production node. When enabling additional collectors, you should 148carefully monitor the change by observing the ` 149scrape_duration_seconds` metric to ensure that collection completes 150and does not time out. In addition, monitor the 151`scrape_samples_post_metric_relabeling` metric to see the changes in 152cardinality. 153 154The `perf` collector may not work out of the box on some Linux systems due to kernel 155configuration and security settings. To allow access, set the following `sysctl` 156parameter: 157 158``` 159sysctl -w kernel.perf_event_paranoid=X 160``` 161 162- 2 allow only user-space measurements (default since Linux 4.6). 163- 1 allow both kernel and user measurements (default before Linux 4.6). 164- 0 allow access to CPU-specific data but not raw tracepoint samples. 165- -1 no restrictions. 166 167Depending on the configured value different metrics will be available, for most 168cases `0` will provide the most complete set. For more information see [`man 2 169perf_event_open`](http://man7.org/linux/man-pages/man2/perf_event_open.2.html). 170 171By default, the `perf` collector will only collect metrics of the CPUs that 172`node_exporter` is running on (ie 173[`runtime.NumCPU`](https://golang.org/pkg/runtime/#NumCPU). If this is 174insufficient (e.g. if you run `node_exporter` with its CPU affinity set to 175specific CPUs), you can specify a list of alternate CPUs by using the 176`--collector.perf.cpus` flag. For example, to collect metrics on CPUs 2-6, you 177would specify: `--collector.perf --collector.perf.cpus=2-6`. The CPU 178configuration is zero indexed and can also take a stride value; e.g. 179`--collector.perf --collector.perf.cpus=1-10:5` would collect on CPUs 1801, 5, and 10. 181 182The `perf` collector is also able to collect 183[tracepoint](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html) 184counts when using the `--collector.perf.tracepoint` flag. Tracepoints can be 185found using [`perf list`](http://man7.org/linux/man-pages/man1/perf.1.html) or 186from debugfs. And example usage of this would be 187`--collector.perf.tracepoint="sched:sched_process_exec"`. 188 189 190Name | Description | OS 191---------|-------------|---- 192buddyinfo | Exposes statistics of memory fragments as reported by /proc/buddyinfo. | Linux 193devstat | Exposes device statistics | Dragonfly, FreeBSD 194drbd | Exposes Distributed Replicated Block Device statistics (to version 8.4) | Linux 195ethtool | Exposes network interface and network driver statistics equivalent to `ethtool -S`. | Linux 196interrupts | Exposes detailed interrupts statistics. | Linux, OpenBSD 197ksmd | Exposes kernel and system statistics from `/sys/kernel/mm/ksm`. | Linux 198logind | Exposes session counts from [logind](http://www.freedesktop.org/wiki/Software/systemd/logind/). | Linux 199meminfo\_numa | Exposes memory statistics from `/proc/meminfo_numa`. | Linux 200mountstats | Exposes filesystem statistics from `/proc/self/mountstats`. Exposes detailed NFS client statistics. | Linux 201network_route | Exposes the routing table as metrics | Linux 202ntp | Exposes local NTP daemon health to check [time](./docs/TIME.md) | _any_ 203perf | Exposes perf based metrics (Warning: Metrics are dependent on kernel configuration and settings). | Linux 204processes | Exposes aggregate process statistics from `/proc`. | Linux 205qdisc | Exposes [queuing discipline](https://en.wikipedia.org/wiki/Network_scheduler#Linux_kernel) statistics | Linux 206runit | Exposes service status from [runit](http://smarden.org/runit/). | _any_ 207supervisord | Exposes service status from [supervisord](http://supervisord.org/). | _any_ 208systemd | Exposes service and system status from [systemd](http://www.freedesktop.org/wiki/Software/systemd/). | Linux 209tcpstat | Exposes TCP connection status information from `/proc/net/tcp` and `/proc/net/tcp6`. (Warning: the current version has potential performance issues in high load situations.) | Linux 210wifi | Exposes WiFi device and station statistics. | Linux 211zoneinfo | Exposes NUMA memory zone metrics. | Linux 212 213 214### Textfile Collector 215 216The `textfile` collector is similar to the [Pushgateway](https://github.com/prometheus/pushgateway), 217in that it allows exporting of statistics from batch jobs. It can also be used 218to export static metrics, such as what role a machine has. The Pushgateway 219should be used for service-level metrics. The `textfile` module is for metrics 220that are tied to a machine. 221 222To use it, set the `--collector.textfile.directory` flag on the `node_exporter` commandline. The 223collector will parse all files in that directory matching the glob `*.prom` 224using the [text 225format](http://prometheus.io/docs/instrumenting/exposition_formats/). **Note:** Timestamps are not supported. 226 227To atomically push completion time for a cron job: 228``` 229echo my_batch_job_completion_time $(date +%s) > /path/to/directory/my_batch_job.prom.$$ 230mv /path/to/directory/my_batch_job.prom.$$ /path/to/directory/my_batch_job.prom 231``` 232 233To statically set roles for a machine using labels: 234``` 235echo 'role{role="application_server"} 1' > /path/to/directory/role.prom.$$ 236mv /path/to/directory/role.prom.$$ /path/to/directory/role.prom 237``` 238 239### Filtering enabled collectors 240 241The `node_exporter` will expose all metrics from enabled collectors by default. This is the recommended way to collect metrics to avoid errors when comparing metrics of different families. 242 243For advanced use the `node_exporter` can be passed an optional list of collectors to filter metrics. The `collect[]` parameter may be used multiple times. In Prometheus configuration you can use this syntax under the [scrape config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#<scrape_config>). 244 245``` 246 params: 247 collect[]: 248 - foo 249 - bar 250``` 251 252This can be useful for having different Prometheus servers collect specific metrics from nodes. 253 254## Development building and running 255 256Prerequisites: 257 258* [Go compiler](https://golang.org/dl/) 259* RHEL/CentOS: `glibc-static` package. 260 261Building: 262 263 git clone https://github.com/prometheus/node_exporter.git 264 cd node_exporter 265 make 266 ./node_exporter <flags> 267 268To see all available configuration flags: 269 270 ./node_exporter -h 271 272## Running tests 273 274 make test 275 276## TLS endpoint 277 278** EXPERIMENTAL ** 279 280The exporter supports TLS via a new web configuration file. 281 282```console 283./node_exporter --web.config=web-config.yml 284``` 285 286See the [exporter-toolkit https package](https://github.com/prometheus/exporter-toolkit/blob/v0.1.0/https/README.md) for more details. 287 288[travis]: https://travis-ci.org/prometheus/node_exporter 289[hub]: https://hub.docker.com/r/prom/node-exporter/ 290[circleci]: https://circleci.com/gh/prometheus/node_exporter 291[quay]: https://quay.io/repository/prometheus/node-exporter 292[goreportcard]: https://goreportcard.com/report/github.com/prometheus/node_exporter 293