Name | Date | Size | #Lines | LOC | ||
---|---|---|---|---|---|---|
.. | 09-Feb-2022 | - | ||||
fluentd/ | H | 09-Feb-2022 | - | 73 | 67 | |
grafana/provisioning/ | H | 09-Feb-2022 | - | 5,804 | 5,463 | |
prometheus/ | H | 09-Feb-2022 | - | 47 | 41 | |
.gitignore | H A D | 09-Feb-2022 | 46 | 1 | 1 | |
README.md | H A D | 09-Feb-2022 | 2.6 KiB | 138 | 93 | |
alerts.sh | H A D | 09-Feb-2022 | 3.5 KiB | 157 | 139 | |
docker-compose.yaml | H A D | 09-Feb-2022 | 2.9 KiB | 116 | 107 |
README.md
1# Grafana High Availability (HA) test setup 2 3A set of docker compose services which together creates a Grafana HA test setup with capability of easily 4scaling up/down number of Grafana instances. 5 6Included services 7 8- Grafana 9- Mysql - Grafana configuration database and session storage 10- Prometheus - Monitoring of Grafana and used as data source of provisioned alert rules 11- Nginx - Reverse proxy for Grafana and Prometheus. Enables browsing Grafana/Prometheus UI using a hostname 12 13## Prerequisites 14 15### Build grafana docker container 16 17Build a Grafana docker container from current branch and commit and tag it as grafana/grafana:dev. 18 19```bash 20$ cd <grafana repo> 21$ make build-docker-full 22``` 23 24### Virtual host names 25 26#### Alternative 1 - Use dnsmasq 27 28```bash 29$ sudo apt-get install dnsmasq 30$ echo 'address=/loc/127.0.0.1' | sudo tee /etc/dnsmasq.d/dnsmasq-loc.conf > /dev/null 31$ sudo /etc/init.d/dnsmasq restart 32$ ping whatever.loc 33PING whatever.loc (127.0.0.1) 56(84) bytes of data. 3464 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.076 ms 35--- whatever.loc ping statistics --- 361 packet transmitted, 1 received, 0% packet loss, time 1998ms 37``` 38 39#### Alternative 2 - Manually update /etc/hosts 40 41Update your `/etc/hosts` to be able to access Grafana and/or Prometheus UI using a hostname. 42 43```bash 44$ cat /etc/hosts 45127.0.0.1 grafana.loc 46127.0.0.1 prometheus.loc 47``` 48 49## Start services 50 51```bash 52$ docker-compose up -d 53``` 54 55Browse 56- http://grafana.loc/ 57- http://prometheus.loc/ 58 59Check for any errors 60 61```bash 62$ docker-compose logs | grep error 63``` 64 65### Scale Grafana instances up/down 66 67Scale number of Grafana instances to `<instances>` 68 69```bash 70$ docker-compose up --scale grafana=<instances> -d 71# for example 3 instances 72$ docker-compose up --scale grafana=3 -d 73``` 74 75## Test alerting 76 77### Create notification channels 78 79Creates default notification channels, if not already exists 80 81```bash 82$ ./alerts.sh setup 83``` 84 85### Slack notifications 86 87Disable 88 89```bash 90$ ./alerts.sh slack -d 91``` 92 93Enable and configure url 94 95```bash 96$ ./alerts.sh slack -u https://hooks.slack.com/services/... 97``` 98 99Enable, configure url and enable reminders 100 101```bash 102$ ./alerts.sh slack -u https://hooks.slack.com/services/... -r -e 10m 103``` 104 105### Provision alert dashboards with alert rules 106 107Provision 1 dashboard/alert rule (default) 108 109```bash 110$ ./alerts.sh provision 111``` 112 113Provision 10 dashboards/alert rules 114 115```bash 116$ ./alerts.sh provision -a 10 117``` 118 119Provision 10 dashboards/alert rules and change condition to `gt > 100` 120 121```bash 122$ ./alerts.sh provision -a 10 -c 100 123``` 124 125### Pause/unpause all alert rules 126 127Pause 128 129```bash 130$ ./alerts.sh pause 131``` 132 133Unpause 134 135```bash 136$ ./alerts.sh unpause 137``` 138